You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/11/25 06:16:20 UTC

[GitHub] [spark] ulysses-you commented on pull request #38736: [SPARK-41214][SQL] - SQL Metrics are missing from Spark UI when AQE for Cached DataFrame is enabled

ulysses-you commented on PR #38736:
URL: https://github.com/apache/spark/pull/38736#issuecomment-1327063095

   The reason is, the `AdaptiveSparkPlanExec` inside `InMemoryTableScanExec` does not register a new execution id in `SQLExecution`. Then when it materializes, `AdaptiveSparkPlanExec.getExecutionId` can not find the corresponding QueryExecution.
   https://github.com/apache/spark/blob/0ae82d99d13988086a297920d45a766115a70578/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala#L221-L226
   
   A simple example like:
   ```
             ...
              |
     AdaptiveSparkPlanExec (queryexecution 0, no execution id)
              |
     InMemoryTableScanExec
              |
             ...
              |
     AdaptiveSparkPlanExec (queryexecution 1, execution id 0)
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org