You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kylin.apache.org by GitBox <gi...@apache.org> on 2022/04/20 01:05:37 UTC
[GitHub] [kylin] hit-lacus commented on pull request #1859: KYLIN-5111 Refine query metrics

hit-lacus commented on PR #1859:
URL: https://github.com/apache/kylin/pull/1859#issuecomment-1103319832

   ## Background
   #### Stages of Kylin query
   ![image](https://user-images.githubusercontent.com/14030549/164125810-909b0d74-b6c4-40d5-b238-fd5e7e286b46.png)
   
   #### Detail of Spark related Job
   ![image](https://user-images.githubusercontent.com/14030549/164125989-8c41188d-ebd5-4f3d-bcb1-49b78af7ea94.png)
   
   
   ## What I do
   - Add log to show more precise metrics and statistics of stages. 
   - Refine the calculation of `WAIT_FOR_EXECUTION` and `EXECUTION`
   
   
   #### Metrics and statistics of stages
   ```java
   2022-04-18 23:42:38,687 DEBUG [Query ddd81385-3a65-b3df-fa95-c92e6d7ad60a-45] query.util.SparkJobTraceV2:58 
       Query job finished.
   2022-04-18 23:42:38,778 INFO  [Query ddd81385-3a65-b3df-fa95-c92e6d7ad60a-45] query.util.SparkJobTraceV2:54 
       Job 1 is submitted at 23:42:38.610 and completed at 23:42:38.652.It has 105 tasks, succeed 1 tasks, 0 failed tasks,0 killed tasks. Killed tasks info: EMPTY. Stages Stage:9,0, launch time of first and last task are 23:42:38.628 and 23:42:38.628,  bytesRead is [0,0,0,0,0], runningTime is [6, 6, 6, 6, 6];Stage:5,0;Stage:6,0;Stage:7,0;Stage:8,0.
   2022-04-18 23:42:38,869 INFO  [Query ddd81385-3a65-b3df-fa95-c92e6d7ad60a-45] query.util.SparkJobTraceV2:54 
       Job 0 is submitted at 23:42:33.056 and completed at 23:42:38.571.It has 105 tasks, succeed 105 tasks, 0 failed tasks,0 killed tasks. Killed tasks info: EMPTY. Stages Stage:0,0, launch time of first and last task are 23:42:33.145 and 23:42:37.141,  bytesRead is [25245,27917,29155,30857,33575], runningTime is [12, 19, 25, 43, 2125];Stage:1,0, launch time of first and last task are 23:42:37.285 and 23:42:37.285,  bytesRead is [0,0,0,0,0], runningTime is [254, 254, 254, 254, 254];Stage:2,0, launch time of first and last task are 23:42:37.166 and 23:42:38.386,  bytesRead is [25245,27917,29155,30857,33575], runningTime is [8, 9, 10, 14, 112];Stage:3,0, launch time of first and last task are 23:42:38.442 and 23:42:38.442,  bytesRead is [0,0,0,0,0], runningTime is [85, 85, 85, 85, 85];Stage:4,0, launch time of first and last task are 23:42:38.554 and 23:42:38.554,  bytesRead is [0,0,0,0,0], runningTime is [3, 3, 3, 3, 3].
   2022-04-18 23:42:38,873 INFO  [Query ddd81385-3a65-b3df-fa95-c92e6d7ad60a-45] query.util.SparkJobTraceV2:54 
       jobExecutionTime will change from 10954 to 5596 .
   2022-04-18 23:42:38,877 INFO  [Query ddd81385-3a65-b3df-fa95-c92e6d7ad60a-45] query.util.SparkJobTraceV2:54 
       Kylin submit query at 23:42:27.733, first spark job submitted at 23:42:33.056, last spark job finished at 23:42:38.652, query finished at 23:42:38.687 .
   ```
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@kylin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org