You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Riza Suminto (Code Review)" <ge...@cloudera.org> on 2021/02/17 20:52:12 UTC
[Impala-ASF-CR] IMPALA-10509: Add tool to visualize Impala query plan from profile
Riza Suminto has uploaded this change for review. ( http://gerrit.cloudera.org:8080/17077
Change subject: IMPALA-10509: Add tool to visualize Impala query plan from profile
......................................................................
IMPALA-10509: Add tool to visualize Impala query plan from profile
Impala query plan can be huge and complex. It becomes hard to analyze
this complex query plan, its execution summary, and the runtime filter
interactions from the query profile. We need tool to help visualize this
query plan from plain text query profile.
This commit adds experimental python3 script plan-graph.py to parse the
text profile into a GraphViz DOT format. The output of this script can
then be converted into image using dot program or immediately visualized
using program such as ZGRViewer.
The script can produce graph in the following verbosity levels:
* 0: minimal graph, containing the concise query plan, time spent on
each plan node, and number of rows produced.
* 1: level 0 graph plus additional runtime filter vertices.
* 2: level 1 graph with additional description in query plan vertices
such as the plan node parameter, selective predicates, groupby
clause, runtime filter info, etc. The runtime filter vertices also
arranged to lower position than scan nodes that should finished
before the runtime filter published. This is currenty the default
verbosity level.
* 3: level 2 graph with scanner to runtime filter dependency edges
explicitly drawn.
Testing:
- Manually test the script against query profiles from TPC-DS runs and
verify that it produce a valid graph visualization.
Change-Id: Ic62e3c2b061dd2b84589336fb9e68fa1eccf76f2
---
A bin/diagnostics/experimental/plan-graph.py
1 file changed, 631 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/77/17077/1
--
To view, visit http://gerrit.cloudera.org:8080/17077
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ic62e3c2b061dd2b84589336fb9e68fa1eccf76f2
Gerrit-Change-Number: 17077
Gerrit-PatchSet: 1
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
[Impala-ASF-CR] IMPALA-10509: Add tool to visualize Impala query plan from profile
Posted by "Aman Sinha (Code Review)" <ge...@cloudera.org>.
Aman Sinha has posted comments on this change. ( http://gerrit.cloudera.org:8080/17077 )
Change subject: IMPALA-10509: Add tool to visualize Impala query plan from profile
......................................................................
Patch Set 2: Code-Review+1
(13 comments)
Thanks for creating this tooling. Some nits below, otherwise yeah it looks good for an experimental diagnostic feature.
http://gerrit.cloudera.org:8080/#/c/17077/2/bin/diagnostics/experimental/plan-graph.py
File bin/diagnostics/experimental/plan-graph.py:
http://gerrit.cloudera.org:8080/#/c/17077/2/bin/diagnostics/experimental/plan-graph.py@29
PS2, Line 29: read
nit: 'reads' instead of 'read'
http://gerrit.cloudera.org:8080/#/c/17077/2/bin/diagnostics/experimental/plan-graph.py@29
PS2, Line 29: output
nit: outputs
http://gerrit.cloudera.org:8080/#/c/17077/2/bin/diagnostics/experimental/plan-graph.py@30
PS2, Line 30: representaiton
nit: spelling
http://gerrit.cloudera.org:8080/#/c/17077/2/bin/diagnostics/experimental/plan-graph.py@36
PS2, Line 36: available
nit: 'is available'
http://gerrit.cloudera.org:8080/#/c/17077/2/bin/diagnostics/experimental/plan-graph.py@154
PS2, Line 154: parse
nit: parses
http://gerrit.cloudera.org:8080/#/c/17077/2/bin/diagnostics/experimental/plan-graph.py@221
PS2, Line 221: is
nit: if
http://gerrit.cloudera.org:8080/#/c/17077/2/bin/diagnostics/experimental/plan-graph.py@267
PS2, Line 267: represent
nit: represents
http://gerrit.cloudera.org:8080/#/c/17077/2/bin/diagnostics/experimental/plan-graph.py@286
PS2, Line 286: represent
nit: represents
http://gerrit.cloudera.org:8080/#/c/17077/2/bin/diagnostics/experimental/plan-graph.py@430
PS2, Line 430: contribute
nit: contributes
http://gerrit.cloudera.org:8080/#/c/17077/2/bin/diagnostics/experimental/plan-graph.py@431
PS2, Line 431: finished
nit: finish or 'should have finished'
http://gerrit.cloudera.org:8080/#/c/17077/2/bin/diagnostics/experimental/plan-graph.py@592
PS2, Line 592: sections
nit: section
http://gerrit.cloudera.org:8080/#/c/17077/2/bin/diagnostics/experimental/plan-graph.py@619
PS2, Line 619: output
nit: outputs
http://gerrit.cloudera.org:8080/#/c/17077/2/bin/diagnostics/experimental/plan-graph.py@619
PS2, Line 619: read
nit: reads
--
To view, visit http://gerrit.cloudera.org:8080/17077
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic62e3c2b061dd2b84589336fb9e68fa1eccf76f2
Gerrit-Change-Number: 17077
Gerrit-PatchSet: 2
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Wed, 17 Feb 2021 22:17:09 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10509: Add tool to visualize Impala query plan from profile
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/17077 )
Change subject: IMPALA-10509: Add tool to visualize Impala query plan from profile
......................................................................
Patch Set 3:
(13 comments)
Thank you for the feedback!
http://gerrit.cloudera.org:8080/#/c/17077/2/bin/diagnostics/experimental/plan-graph.py
File bin/diagnostics/experimental/plan-graph.py:
http://gerrit.cloudera.org:8080/#/c/17077/2/bin/diagnostics/experimental/plan-graph.py@29
PS2, Line 29: read
> nit: 'reads' instead of 'read'
Done
http://gerrit.cloudera.org:8080/#/c/17077/2/bin/diagnostics/experimental/plan-graph.py@29
PS2, Line 29: outpu
> nit: outputs
Done
http://gerrit.cloudera.org:8080/#/c/17077/2/bin/diagnostics/experimental/plan-graph.py@30
PS2, Line 30: representation
> nit: spelling
Done
http://gerrit.cloudera.org:8080/#/c/17077/2/bin/diagnostics/experimental/plan-graph.py@36
PS2, Line 36: is availa
> nit: 'is available'
Done
http://gerrit.cloudera.org:8080/#/c/17077/2/bin/diagnostics/experimental/plan-graph.py@154
PS2, Line 154: parse
> nit: parses
Done
http://gerrit.cloudera.org:8080/#/c/17077/2/bin/diagnostics/experimental/plan-graph.py@221
PS2, Line 221: if
> nit: if
Done
http://gerrit.cloudera.org:8080/#/c/17077/2/bin/diagnostics/experimental/plan-graph.py@267
PS2, Line 267: represent
> nit: represents
Done
http://gerrit.cloudera.org:8080/#/c/17077/2/bin/diagnostics/experimental/plan-graph.py@286
PS2, Line 286: represent
> nit: represents
Done
http://gerrit.cloudera.org:8080/#/c/17077/2/bin/diagnostics/experimental/plan-graph.py@430
PS2, Line 430: contribute
> nit: contributes
Done
http://gerrit.cloudera.org:8080/#/c/17077/2/bin/diagnostics/experimental/plan-graph.py@431
PS2, Line 431: have fin
> nit: finish or 'should have finished'
Done
http://gerrit.cloudera.org:8080/#/c/17077/2/bin/diagnostics/experimental/plan-graph.py@592
PS2, Line 592: section
> nit: section
Done
http://gerrit.cloudera.org:8080/#/c/17077/2/bin/diagnostics/experimental/plan-graph.py@619
PS2, Line 619: outpu
> nit: outputs
Done
http://gerrit.cloudera.org:8080/#/c/17077/2/bin/diagnostics/experimental/plan-graph.py@619
PS2, Line 619: read
> nit: reads
Done
--
To view, visit http://gerrit.cloudera.org:8080/17077
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic62e3c2b061dd2b84589336fb9e68fa1eccf76f2
Gerrit-Change-Number: 17077
Gerrit-PatchSet: 3
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Wed, 17 Feb 2021 23:10:20 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10509: Add tool to visualize Impala query plan from profile
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17077 )
Change subject: IMPALA-10509: Add tool to visualize Impala query plan from profile
......................................................................
Patch Set 4: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/17077
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic62e3c2b061dd2b84589336fb9e68fa1eccf76f2
Gerrit-Change-Number: 17077
Gerrit-PatchSet: 4
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Thu, 18 Feb 2021 08:49:11 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10509: Add tool to visualize Impala query plan from profile
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17077 )
Change subject: IMPALA-10509: Add tool to visualize Impala query plan from profile
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/8148/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/17077
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic62e3c2b061dd2b84589336fb9e68fa1eccf76f2
Gerrit-Change-Number: 17077
Gerrit-PatchSet: 1
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Wed, 17 Feb 2021 21:12:31 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10509: Add tool to visualize Impala query plan from profile
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17077 )
Change subject: IMPALA-10509: Add tool to visualize Impala query plan from profile
......................................................................
Patch Set 4: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/17077
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic62e3c2b061dd2b84589336fb9e68fa1eccf76f2
Gerrit-Change-Number: 17077
Gerrit-PatchSet: 4
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Thu, 18 Feb 2021 03:07:02 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10509: Add tool to visualize Impala query plan from profile
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/17077 )
Change subject: IMPALA-10509: Add tool to visualize Impala query plan from profile
......................................................................
Patch Set 2:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/17077/1/bin/diagnostics/experimental/plan-graph.py
File bin/diagnostics/experimental/plan-graph.py:
http://gerrit.cloudera.org:8080/#/c/17077/1/bin/diagnostics/experimental/plan-graph.py@50
PS1, Line 50:
> flake8: E305 expected 2 blank lines after class or function definition, fou
Done
http://gerrit.cloudera.org:8080/#/c/17077/1/bin/diagnostics/experimental/plan-graph.py@630
PS1, Line 630:
> flake8: E305 expected 2 blank lines after class or function definition, fou
Done
--
To view, visit http://gerrit.cloudera.org:8080/17077
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic62e3c2b061dd2b84589336fb9e68fa1eccf76f2
Gerrit-Change-Number: 17077
Gerrit-PatchSet: 2
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Wed, 17 Feb 2021 21:01:14 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10509: Add tool to visualize Impala query plan from profile
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17077 )
Change subject: IMPALA-10509: Add tool to visualize Impala query plan from profile
......................................................................
Patch Set 4:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6896/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/17077
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic62e3c2b061dd2b84589336fb9e68fa1eccf76f2
Gerrit-Change-Number: 17077
Gerrit-PatchSet: 4
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Thu, 18 Feb 2021 03:07:03 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10509: Add tool to visualize Impala query plan from profile
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/17077 )
Change subject: IMPALA-10509: Add tool to visualize Impala query plan from profile
......................................................................
IMPALA-10509: Add tool to visualize Impala query plan from profile
Impala query plan can be huge and complex. It becomes hard to analyze
this complex query plan, its execution summary, and the runtime filter
interactions from the query profile. We need a tool to help visualize
this query plan from a plain text query profile.
This commit adds experimental python3 script plan-graph.py to parse the
text profile into a GraphViz DOT format. The output of this script can
then be converted into an image using dot program or immediately
visualized using program such as ZGRViewer.
The script can produce a graph in the following verbosity levels:
* 0: minimal graph, containing the concise query plan, time spent on
each plan node, and the number of rows produced.
* 1: level 0 graph plus additional runtime filter vertices.
* 2: level 1 graph plus additional description in query plan vertices
such as the plan node parameter, selective predicates, group by
clause, runtime filter info, etc. The runtime filter vertices are
also arranged to a lower position than scan nodes that should
be finished before the runtime filter is published. This is
currently the default verbosity level.
* 3: level 2 graph, but with the scanner to runtime filter dependency
edges explicitly drawn.
Testing:
- Manually test the script against query profiles from TPC-DS runs and
verify that it produces a valid graph visualization.
Change-Id: Ic62e3c2b061dd2b84589336fb9e68fa1eccf76f2
Reviewed-on: http://gerrit.cloudera.org:8080/17077
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
A bin/diagnostics/experimental/plan-graph.py
1 file changed, 633 insertions(+), 0 deletions(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/17077
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Ic62e3c2b061dd2b84589336fb9e68fa1eccf76f2
Gerrit-Change-Number: 17077
Gerrit-PatchSet: 5
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
[Impala-ASF-CR] IMPALA-10509: Add tool to visualize Impala query plan from profile
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17077 )
Change subject: IMPALA-10509: Add tool to visualize Impala query plan from profile
......................................................................
Patch Set 2:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/8149/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/17077
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic62e3c2b061dd2b84589336fb9e68fa1eccf76f2
Gerrit-Change-Number: 17077
Gerrit-PatchSet: 2
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Wed, 17 Feb 2021 21:17:13 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10509: Add tool to visualize Impala query plan from profile
Posted by "Aman Sinha (Code Review)" <ge...@cloudera.org>.
Aman Sinha has posted comments on this change. ( http://gerrit.cloudera.org:8080/17077 )
Change subject: IMPALA-10509: Add tool to visualize Impala query plan from profile
......................................................................
Patch Set 3: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/17077
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic62e3c2b061dd2b84589336fb9e68fa1eccf76f2
Gerrit-Change-Number: 17077
Gerrit-PatchSet: 3
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Thu, 18 Feb 2021 03:05:38 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10509: Add tool to visualize Impala query plan from profile
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17077 )
Change subject: IMPALA-10509: Add tool to visualize Impala query plan from profile
......................................................................
Patch Set 1:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/17077/1/bin/diagnostics/experimental/plan-graph.py
File bin/diagnostics/experimental/plan-graph.py:
http://gerrit.cloudera.org:8080/#/c/17077/1/bin/diagnostics/experimental/plan-graph.py@50
PS1, Line 50: ParseTask = namedtuple('ParseTask', ['task', 'delimiter'])
flake8: E305 expected 2 blank lines after class or function definition, found 1
http://gerrit.cloudera.org:8080/#/c/17077/1/bin/diagnostics/experimental/plan-graph.py@630
PS1, Line 630: if __name__ == "__main__":
flake8: E305 expected 2 blank lines after class or function definition, found 1
--
To view, visit http://gerrit.cloudera.org:8080/17077
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic62e3c2b061dd2b84589336fb9e68fa1eccf76f2
Gerrit-Change-Number: 17077
Gerrit-PatchSet: 1
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 17 Feb 2021 20:53:06 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10509: Add tool to visualize Impala query plan from profile
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17077 )
Change subject: IMPALA-10509: Add tool to visualize Impala query plan from profile
......................................................................
Patch Set 3:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/8150/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/17077
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic62e3c2b061dd2b84589336fb9e68fa1eccf76f2
Gerrit-Change-Number: 17077
Gerrit-PatchSet: 3
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Wed, 17 Feb 2021 23:27:28 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10509: Add tool to visualize Impala query plan from profile
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Hello Aman Sinha, David Rorke, Joe McDonnell, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/17077
to look at the new patch set (#2).
Change subject: IMPALA-10509: Add tool to visualize Impala query plan from profile
......................................................................
IMPALA-10509: Add tool to visualize Impala query plan from profile
Impala query plan can be huge and complex. It becomes hard to analyze
this complex query plan, its execution summary, and the runtime filter
interactions from the query profile. We need tool to help visualize this
query plan from plain text query profile.
This commit adds experimental python3 script plan-graph.py to parse the
text profile into a GraphViz DOT format. The output of this script can
then be converted into image using dot program or immediately visualized
using program such as ZGRViewer.
The script can produce graph in the following verbosity levels:
* 0: minimal graph, containing the concise query plan, time spent on
each plan node, and number of rows produced.
* 1: level 0 graph plus additional runtime filter vertices.
* 2: level 1 graph with additional description in query plan vertices
such as the plan node parameter, selective predicates, groupby
clause, runtime filter info, etc. The runtime filter vertices also
arranged to lower position than scan nodes that should finished
before the runtime filter published. This is currenty the default
verbosity level.
* 3: level 2 graph with scanner to runtime filter dependency edges
explicitly drawn.
Testing:
- Manually test the script against query profiles from TPC-DS runs and
verify that it produce a valid graph visualization.
Change-Id: Ic62e3c2b061dd2b84589336fb9e68fa1eccf76f2
---
A bin/diagnostics/experimental/plan-graph.py
1 file changed, 633 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/77/17077/2
--
To view, visit http://gerrit.cloudera.org:8080/17077
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic62e3c2b061dd2b84589336fb9e68fa1eccf76f2
Gerrit-Change-Number: 17077
Gerrit-PatchSet: 2
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
[Impala-ASF-CR] IMPALA-10509: Add tool to visualize Impala query plan from profile
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Hello Aman Sinha, David Rorke, Joe McDonnell, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/17077
to look at the new patch set (#3).
Change subject: IMPALA-10509: Add tool to visualize Impala query plan from profile
......................................................................
IMPALA-10509: Add tool to visualize Impala query plan from profile
Impala query plan can be huge and complex. It becomes hard to analyze
this complex query plan, its execution summary, and the runtime filter
interactions from the query profile. We need a tool to help visualize
this query plan from a plain text query profile.
This commit adds experimental python3 script plan-graph.py to parse the
text profile into a GraphViz DOT format. The output of this script can
then be converted into an image using dot program or immediately
visualized using program such as ZGRViewer.
The script can produce a graph in the following verbosity levels:
* 0: minimal graph, containing the concise query plan, time spent on
each plan node, and the number of rows produced.
* 1: level 0 graph plus additional runtime filter vertices.
* 2: level 1 graph plus additional description in query plan vertices
such as the plan node parameter, selective predicates, group by
clause, runtime filter info, etc. The runtime filter vertices are
also arranged to a lower position than scan nodes that should
be finished before the runtime filter is published. This is
currently the default verbosity level.
* 3: level 2 graph, but with the scanner to runtime filter dependency
edges explicitly drawn.
Testing:
- Manually test the script against query profiles from TPC-DS runs and
verify that it produces a valid graph visualization.
Change-Id: Ic62e3c2b061dd2b84589336fb9e68fa1eccf76f2
---
A bin/diagnostics/experimental/plan-graph.py
1 file changed, 633 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/77/17077/3
--
To view, visit http://gerrit.cloudera.org:8080/17077
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic62e3c2b061dd2b84589336fb9e68fa1eccf76f2
Gerrit-Change-Number: 17077
Gerrit-PatchSet: 3
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>