You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@quickstep.apache.org by jianqiao <gi...@git.apache.org> on 2016/08/03 17:13:58 UTC
[GitHub] incubator-quickstep pull request #76: Visualize execution plan DAGs annotate...
GitHub user jianqiao opened a pull request:
https://github.com/apache/incubator-quickstep/pull/76
Visualize execution plan DAGs annotated with profiling stats
This PR adds support for visualization of execution plan DAGs. Each relational operator in the DAG is also annotated with the execution profiling stats of its work orders. The main changes are
- Added a `getName()` method for each relational operator.
- Implemented visualization code in `utility/ExecutionDAGVisualizer.*`.
- Slightly refactored work order profiling code to record both start time and end time of each work order.
To enable this feature, set the gflag `visualize_execution_dag` to true, e.g.
```
quickstep_cli_shell -visualize_execution_dag=true
```
The execution dag will be printed via `stderr` in DOT format. Each relational operator node contains the following information:
- Relational operator ID, Relational operator name.
- The input relation's name (if it is a stored relation).
- Total execution time of all the work orders of this relational operator.
- Execution time span (start time of the first work order and finish time of the last work order) of this relational operator.
- Degree of concurrency (total time / span) of this relational operator.
Here's an example output, for SSB query 05:
```
digraph g {
rankdir=BT
node [penwidth=2]
edge [fontsize=16 fontcolor=gray penwidth=2]
0 [ label="[0] SelectOperator input stored relation [supplier] span: [0ms, 6ms] (0.74%) total: 38ms (0.13%) effective concurrency: 6.29" style=filled fillcolor="0.004717 0.004717 1.0" ]
1 [ label="[1] SelectOperator input stored relation [part] span: [0ms, 10ms] (1.25%) total: 273ms (0.90%) effective concurrency: 26.26" style=filled fillcolor="0.011673 0.011673 1.0" ]
2 [ label="[2] BuildHashOperator span: [10ms, 12ms] (0.21%) total: 19ms (0.06%) effective concurrency: 11.20" style=filled fillcolor="0.001494 0.001494 1.0" ]
3 [ label="[3] HashJoinOperator probe side stored relation [lineorder] span: [17ms, 752ms] (88.04%) total: 29059ms (95.43%) effective concurrency: 39.53" style=filled fillcolor="1.000000 1.000000 1.0" ]
5 [ label="[5] BuildHashOperator span: [6ms, 9ms] (0.39%) total: 6ms (0.02%) effective concurrency: 1.97" style=filled fillcolor="0.002248 0.002248 1.0" ]
6 [ label="[6] HashJoinOperator span: [748ms, 773ms] (3.04%) total: 438ms (1.44%) effective concurrency: 17.28" style=filled fillcolor="0.024394 0.024394 1.0" ]
8 [ label="[8] BuildHashOperator input stored relation [ddate] span: [8ms, 8ms] (0.05%) total: 0ms (0.00%) effective concurrency: 1.00" style=filled fillcolor="0.000274 0.000274 1.0" ]
9 [ label="[9] HashJoinOperator span: [767ms, 806ms] (4.65%) total: 372ms (1.22%) effective concurrency: 9.59" style=filled fillcolor="0.032002 0.032002 1.0" ]
11 [ label="[11] AggregationOperator span: [792ms, 832ms] (4.80%) total: 223ms (0.73%) effective concurrency: 5.58" style=filled fillcolor="0.030163 0.030163 1.0" ]
12 [ label="[12] FinalizeAggregationOperator span: [832ms, 833ms] (0.02%) total: 0ms (0.00%) effective concurrency: 1.00" style=filled fillcolor="0.000139 0.000139 1.0" ]
13 [ label="[13] SortRunGenerationOperator span: [834ms, 834ms] (0.01%) total: 0ms (0.00%) effective concurrency: 1.00" style=filled fillcolor="0.000030 0.000030 1.0" ]
14 [ label="[14] SortMergeRunOperator span: [834ms, 834ms] (0.01%) total: 0ms (0.00%) effective concurrency: 1.00" style=filled fillcolor="0.000043 0.000043 1.0" ]
16 [ label="[16] SelectOperator span: [834ms, 834ms] (0.00%) total: 0ms (0.00%) effective concurrency: 1.00" style=filled fillcolor="0.000011 0.000011 1.0" ]
0 -> 6 [ style=dashed ]
0 -> 5 [ ]
1 -> 3 [ style=dashed ]
1 -> 2 [ ]
2 -> 3 [ style=dashed ]
3 -> 6 [ ]
5 -> 6 [ style=dashed ]
6 -> 9 [ ]
8 -> 9 [ style=dashed ]
9 -> 11 [ ]
11 -> 12 [ style=dashed ]
12 -> 13 [ ]
13 -> 14 [ ]
14 -> 16 [ ]
}
```
To convert the DOT description into an image. A quick way is to use online visualization tools (e.g. http://www.webgraphviz.com, just copy and paste the graph description into the webpage's textbox and click _Generate Graph!_.
The output can also be redirected to a file and then converted to an image with the dot tool, e.g.
```
quickstep_cli_shell -visualize_execution_dag=true < some_query.sql 2> some_graph.dot
dot -Gsize="8,10.5" -Teps some_graph.dot -o some_graph.eps
```
Here eps can be `pdf`, `png`, etc.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/apache/incubator-quickstep execution-dag-visualizer
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-quickstep/pull/76.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #76
----
commit 4b944328afde90c961122d547c6fa4a8d0230c1c
Author: Jianqiao Zhu <ji...@cs.wisc.edu>
Date: 2016-08-02T21:57:47Z
Add visualization for execution plan DAGs combined with profiling stats
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quickstep pull request #76: Visualize execution plan DAGs annotate...
Posted by hbdeshmukh <gi...@git.apache.org>.
Github user hbdeshmukh commented on a diff in the pull request:
https://github.com/apache/incubator-quickstep/pull/76#discussion_r73384721
--- Diff: query_execution/QueryExecutionTypedefs.hpp ---
@@ -98,6 +98,12 @@ enum QueryExecutionMessageType : message_type_id {
#endif
};
+// WorkOrder profiling data structures.
+// A tuple of <Worker ID, Operator ID, Start Time, End Time>
+typedef std::tuple<std::size_t, std::size_t, std::size_t, std::size_t> WorkOrderTimeEntry;
--- End diff --
It may be helpful to make WorkOrderTimeEntry as a struct with the parameters as public variables. That way we will be less concerned about the ordering of the parameters and any potential bugs.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quickstep pull request #76: Visualize execution plan DAGs annotate...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-quickstep/pull/76
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quickstep issue #76: Visualize execution plan DAGs annotated with ...
Posted by jianqiao <gi...@git.apache.org>.
Github user jianqiao commented on the issue:
https://github.com/apache/incubator-quickstep/pull/76
Thanks Harshad!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quickstep issue #76: Visualize execution plan DAGs annotated with ...
Posted by hbdeshmukh <gi...@git.apache.org>.
Github user hbdeshmukh commented on the issue:
https://github.com/apache/incubator-quickstep/pull/76
Looks good to me. Thanks for this feature, it should be quite useful!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quickstep issue #76: Visualize execution plan DAGs annotated with ...
Posted by hbdeshmukh <gi...@git.apache.org>.
Github user hbdeshmukh commented on the issue:
https://github.com/apache/incubator-quickstep/pull/76
This branch needs rebasing with the master, I will try doing it on my own and if it doesn't work, I will let you know.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quickstep issue #76: Visualize execution plan DAGs annotated with ...
Posted by zuyu <gi...@git.apache.org>.
Github user zuyu commented on the issue:
https://github.com/apache/incubator-quickstep/pull/76
@jianqiao Please submit another PR to fix `iwyu` issues for all the files touched by this PR, including adding `<cstddef>` in `query_execution/QueryExecutionTypedefs.hpp`, and removing `<tuple>` in both `query_execution/PolicyEnforcerBase.hpp` and `query_execution/ForemanSingleNode.cpp`. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-quickstep pull request #76: Visualize execution plan DAGs annotate...
Posted by hbdeshmukh <gi...@git.apache.org>.
Github user hbdeshmukh commented on a diff in the pull request:
https://github.com/apache/incubator-quickstep/pull/76#discussion_r73384912
--- Diff: query_execution/QueryExecutionTypedefs.hpp ---
@@ -98,6 +98,12 @@ enum QueryExecutionMessageType : message_type_id {
#endif
};
+// WorkOrder profiling data structures.
+// A tuple of <Worker ID, Operator ID, Start Time, End Time>
+typedef std::tuple<std::size_t, std::size_t, std::size_t, std::size_t> WorkOrderTimeEntry;
+
+typedef std::unordered_map<std::size_t, std::vector<WorkOrderTimeEntry>> WorkOrderTimeRecorder;
--- End diff --
Can you briefly describe the key and value in this map, as a comment above this line?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---