[GitHub] [spark] erenavsarogullari edited a comment on pull request #35536: [SPARK-38222][SQL] Expose Node Description attribute in SQL Rest API

[GitBox <gi...@apache.org>]

erenavsarogullari edited a comment on pull request #35536:
URL: https://github.com/apache/spark/pull/35536#issuecomment-1042392854


   Thanks @srowen for quick feedback. Please find some samples per operation as follows. I also attach whole sql response to jira in order to give clear idea:
   https://issues.apache.org/jira/secure/attachment/13040136/Spark_SQL_REST_Result_with-nodeDesc
   
   I think on following options:
   - We can exclude `WholeStageCodegen` and if (`nodeName == nodeDesc`) for all use-cases.
   - **Option 1:** We can expose SQLConf in order to include white-listed operators. With this option, the user can define the operator white-list. For example:
   ```
   spark.sql.rest.api.include.operators.from.desc=SortMergeJoin,HashAggregate,Project
   ```
   - **Option 2:** We can also expose another SQLConf if end user wants to see `nodeDesc` of all operators. For example:
   ```
   spark.sql.rest.api.enable.all.operators.desc=false(default)
   ```
   Sample Results:
   ```
   "nodeName" : "SerializeFromObject",
   "nodeDesc" : "SerializeFromObject [knownnotnull(assertnotnull(input[0, org.apache.spark.status.api.v1.sql.Salary, true])).personId AS personId#17, knownnotnull(assertnotnull(input[0, org.apache.spark.status.api.v1.sql.Salary, true])).salary AS salary#18]",
   
   "nodeName" : "Project",
   "nodeDesc" : "Project [personId#17 AS id#24, salary#18]",
   
   "nodeName" : "Exchange",
   "nodeDesc" : "Exchange hashpartitioning(name#5, age#6, salary#18, 5), ENSURE_REQUIREMENTS, [id=#87]",
       
   "nodeName" : "Sort",
   "nodeDesc" : "Sort [id#24 ASC NULLS FIRST], false, 0",
       
   "nodeName" : "Scan",
   "nodeDesc" : "Scan[obj#3]",
       
   "nodeName" : "SortMergeJoin",
   "nodeDesc" : "SortMergeJoin [id#4], [id#24], Inner",
       
   "nodeName" : "HashAggregate",
   "nodeDesc" : "HashAggregate(keys=[name#5, age#6, knownfloatingpointnormalized(normalizenanandzero(salary#18)) AS salary#18], functions=[partial_avg(age#6), partial_avg(salary#18)])",
           
   "nodeName" : "Filter",
   "nodeDesc" : "Filter org.apache.spark.status.api.v1.sql.SqlResourceWithActualMetricsSuite$$Lambda$1666/2062184524@72c9ebfa.apply",
   
   // Following ones need exclusion
   "nodeName" : "WholeStageCodegen (3)",
   "nodeDesc" : "WholeStageCodegen (3)",
   
   "nodeName" : "Project",
   "nodeDesc" : "Project",
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org