You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@hive.apache.org by Karen Coppage <ka...@gmail.com> on 2017/08/15 15:37:28 UTC

Review Request 61663: WebUI query plan graphs

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61663/
-----------------------------------------------------------

Review request for hive, Peter Vary and Xuefu Zhang.


Repository: hive-git


Description
-------

I’m working on a feature of the Hive WebUI Query Plan tab that would provide the option to display the query plan as a nice graph (scroll down for screenshots). If you click on one of the graph’s stages, the plan for that stage appears as text below.
Stages are color-coded if they have a status (Success, Error, Running), and the rest are grayed out. Coloring is based on status already available in the WebUI, under the Stages tab.
There is an additional option to display stats for MapReduce tasks. This includes the job’s ID, tracking URL (where the logs are found), and mapper and reducer numbers/progress, among other info.
The library I’m using for the graph is called vis.js (http://visjs.org/). It has an Apache license, and the only necessary file to be included from this library is about 700 KB.
I tried to keep server-side changes minimal, and graph generation is taken care of by the client. Plans with more than a given number of stages (default: 25) won't be displayed in order to preserve resources.


Diffs
-----

  common/src/java/org/apache/hadoop/hive/common/LogUtils.java 0a3e0c7201 
  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 056f2d7834 
  itests/hive-unit/src/test/java/org/apache/hive/service/cli/session/TestQueryDisplay.java 155c65dd26 
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java 4e7c80f184 
  ql/src/java/org/apache/hadoop/hive/ql/MapRedStats.java 4b6051485e 
  ql/src/java/org/apache/hadoop/hive/ql/QueryDisplay.java 5bf22107dc 
  ql/src/java/org/apache/hadoop/hive/ql/exec/mr/HadoopJobExecHelper.java 3c0719717c 
  ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapRedTask.java 1bd4db7805 
  service/src/jamon/org/apache/hive/tmpl/QueryProfileTmpl.jamon 5e2d68c4a4 
  service/src/java/org/apache/hive/service/servlet/QueryProfileServlet.java 27a3b1423a 
  service/src/resources/hive-webapps/static/css/query-plan-graph.css PRE-CREATION 
  service/src/resources/hive-webapps/static/js/query-plan-graph.js PRE-CREATION 
  service/src/resources/hive-webapps/static/js/vis.min.js PRE-CREATION 


Diff: https://reviews.apache.org/r/61663/diff/1/


Testing
-------


Thanks,

Karen Coppage

Re: Review Request 61663: WebUI query plan graphs

Posted by Peter Vary <pv...@cloudera.com>.


> On Aug. 30, 2017, 4:47 a.m., Xuefu Zhang wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/QueryDisplay.java
> > Lines 127 (patched)
> > <https://reviews.apache.org/r/61663/diff/3/?file=1798755#file1798755line127>
> >
> >     What happens if this is a map-only task?

In the next patch I split the if statement


> On Aug. 30, 2017, 4:47 a.m., Xuefu Zhang wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/QueryDisplay.java
> > Lines 132 (patched)
> > <https://reviews.apache.org/r/61663/diff/3/?file=1798755#file1798755line132>
> >
> >     It might be better if the null check is put in getCountersJson() method.

I think we should keep this, as it is - it is better to omit the "counter" element, than adding an empty one? What do you think?


> On Aug. 30, 2017, 4:47 a.m., Xuefu Zhang wrote:
> > service/src/resources/hive-webapps/static/css/query-plan-graph.css
> > Lines 1 (patched)
> > <https://reviews.apache.org/r/61663/diff/3/?file=1798759#file1798759line1>
> >
> >     Apache license header if possible.

Done


> On Aug. 30, 2017, 4:47 a.m., Xuefu Zhang wrote:
> > service/src/resources/hive-webapps/static/js/query-plan-graph.js
> > Lines 1 (patched)
> > <https://reviews.apache.org/r/61663/diff/3/?file=1798760#file1798760line1>
> >
> >     I think we need apache license header.

Done


> On Aug. 30, 2017, 4:47 a.m., Xuefu Zhang wrote:
> > service/src/resources/hive-webapps/static/js/vis.min.js
> > Lines 1 (patched)
> > <https://reviews.apache.org/r/61663/diff/3/?file=1798761#file1798761line1>
> >
> >     Apache license header.

Done


- Peter


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61663/#review184121
-----------------------------------------------------------


On Aug. 16, 2017, 1:55 p.m., Karen Coppage wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61663/
> -----------------------------------------------------------
> 
> (Updated Aug. 16, 2017, 1:55 p.m.)
> 
> 
> Review request for hive, Peter Vary and Xuefu Zhang.
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> I’m working on a feature of the Hive WebUI Query Plan tab that would provide the option to display the query plan as a nice graph (scroll down for screenshots). If you click on one of the graph’s stages, the plan for that stage appears as text below.
> Stages are color-coded if they have a status (Success, Error, Running), and the rest are grayed out. Coloring is based on status already available in the WebUI, under the Stages tab.
> There is an additional option to display stats for MapReduce tasks. This includes the job’s ID, tracking URL (where the logs are found), and mapper and reducer numbers/progress, among other info.
> The library I’m using for the graph is called vis.js (http://visjs.org/). It has an Apache license, and the only necessary file to be included from this library is about 700 KB.
> I tried to keep server-side changes minimal, and graph generation is taken care of by the client. Plans with more than a given number of stages (default: 25) won't be displayed in order to preserve resources.
> 
> 
> Diffs
> -----
> 
>   common/src/java/org/apache/hadoop/hive/common/LogUtils.java 0a3e0c7201 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 3c158a6692 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 4e7c80f184 
>   ql/src/java/org/apache/hadoop/hive/ql/MapRedStats.java 4b6051485e 
>   ql/src/java/org/apache/hadoop/hive/ql/QueryDisplay.java bf6cb91745 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/mr/HadoopJobExecHelper.java 3c0719717c 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapRedTask.java 1bd4db7805 
>   service/src/jamon/org/apache/hive/tmpl/QueryProfileTmpl.jamon ff7476ee02 
>   service/src/resources/hive-webapps/static/css/query-plan-graph.css PRE-CREATION 
>   service/src/resources/hive-webapps/static/js/query-plan-graph.js PRE-CREATION 
>   service/src/resources/hive-webapps/static/js/vis.min.js PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/61663/diff/3/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Karen Coppage
> 
>

Re: Review Request 61663: WebUI query plan graphs

Posted by Xuefu Zhang <xz...@cloudera.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61663/#review184121
-----------------------------------------------------------



Patch looks good. I didn't read the js files, but hope it's okay. a few minor comments.


ql/src/java/org/apache/hadoop/hive/ql/QueryDisplay.java
Lines 127 (patched)
<https://reviews.apache.org/r/61663/#comment260180>

    What happens if this is a map-only task?



ql/src/java/org/apache/hadoop/hive/ql/QueryDisplay.java
Lines 132 (patched)
<https://reviews.apache.org/r/61663/#comment260181>

    It might be better if the null check is put in getCountersJson() method.



service/src/resources/hive-webapps/static/css/query-plan-graph.css
Lines 1 (patched)
<https://reviews.apache.org/r/61663/#comment260183>

    Apache license header if possible.



service/src/resources/hive-webapps/static/js/query-plan-graph.js
Lines 1 (patched)
<https://reviews.apache.org/r/61663/#comment260182>

    I think we need apache license header.



service/src/resources/hive-webapps/static/js/vis.min.js
Lines 1 (patched)
<https://reviews.apache.org/r/61663/#comment260184>

    Apache license header.


- Xuefu Zhang


On Aug. 16, 2017, 1:55 p.m., Karen Coppage wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61663/
> -----------------------------------------------------------
> 
> (Updated Aug. 16, 2017, 1:55 p.m.)
> 
> 
> Review request for hive, Peter Vary and Xuefu Zhang.
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> I’m working on a feature of the Hive WebUI Query Plan tab that would provide the option to display the query plan as a nice graph (scroll down for screenshots). If you click on one of the graph’s stages, the plan for that stage appears as text below.
> Stages are color-coded if they have a status (Success, Error, Running), and the rest are grayed out. Coloring is based on status already available in the WebUI, under the Stages tab.
> There is an additional option to display stats for MapReduce tasks. This includes the job’s ID, tracking URL (where the logs are found), and mapper and reducer numbers/progress, among other info.
> The library I’m using for the graph is called vis.js (http://visjs.org/). It has an Apache license, and the only necessary file to be included from this library is about 700 KB.
> I tried to keep server-side changes minimal, and graph generation is taken care of by the client. Plans with more than a given number of stages (default: 25) won't be displayed in order to preserve resources.
> 
> 
> Diffs
> -----
> 
>   common/src/java/org/apache/hadoop/hive/common/LogUtils.java 0a3e0c7201 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 3c158a6692 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 4e7c80f184 
>   ql/src/java/org/apache/hadoop/hive/ql/MapRedStats.java 4b6051485e 
>   ql/src/java/org/apache/hadoop/hive/ql/QueryDisplay.java bf6cb91745 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/mr/HadoopJobExecHelper.java 3c0719717c 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapRedTask.java 1bd4db7805 
>   service/src/jamon/org/apache/hive/tmpl/QueryProfileTmpl.jamon ff7476ee02 
>   service/src/resources/hive-webapps/static/css/query-plan-graph.css PRE-CREATION 
>   service/src/resources/hive-webapps/static/js/query-plan-graph.js PRE-CREATION 
>   service/src/resources/hive-webapps/static/js/vis.min.js PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/61663/diff/3/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Karen Coppage
> 
>

Re: Review Request 61663: WebUI query plan graphs

Posted by Peter Vary via Review Board <no...@reviews.apache.org>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61663/#review208622
-----------------------------------------------------------



Fix it then ship it!

Thanks for the revitalization of this change!


ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapRedTask.java
Line 508 (original), 509 (patched)
<https://reviews.apache.org/r/61663/#comment292721>

    Might want to use LOG.error(String, Throwable)



common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
Lines 3099 (patched)
<https://reviews.apache.org/r/61663/#comment292719>

    After committing the change, please do not forget to update the configuration Wiki. :)


- Peter Vary


On szept. 7, 2018, 3:24 du, Karen Coppage wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61663/
> -----------------------------------------------------------
> 
> (Updated szept. 7, 2018, 3:24 du)
> 
> 
> Review request for hive, Peter Vary and Xuefu Zhang.
> 
> 
> Bugs: HIVE-17300
>     https://issues.apache.org/jira/browse/HIVE-17300
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> I’m working on a feature of the Hive WebUI Query Plan tab that would provide the option to display the query plan as a nice graph (scroll down for screenshots). If you click on one of the graph’s stages, the plan for that stage appears as text below.
> Stages are color-coded if they have a status (Success, Error, Running), and the rest are grayed out. Coloring is based on status already available in the WebUI, under the Stages tab.
> There is an additional option to display stats for MapReduce tasks. This includes the job’s ID, tracking URL (where the logs are found), and mapper and reducer numbers/progress, among other info.
> The library I’m using for the graph is called vis.js (http://visjs.org/). It has an Apache license, and the only necessary file to be included from this library is about 700 KB.
> I tried to keep server-side changes minimal, and graph generation is taken care of by the client. Plans with more than a given number of stages (default: 25) won't be displayed in order to preserve resources.
> 
> 
> Diffs
> -----
> 
>   common/src/java/org/apache/hadoop/hive/common/LogUtils.java 5068eb5be7 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 40ea3ac0c5 
>   itests/hive-unit/src/test/java/org/apache/hive/service/cli/session/TestQueryDisplay.java 95b46a8149 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java dad2035362 
>   ql/src/java/org/apache/hadoop/hive/ql/MapRedStats.java ac45ec46de 
>   ql/src/java/org/apache/hadoop/hive/ql/QueryDisplay.java 9a77c2969e 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/mr/HadoopJobExecHelper.java eb6cbf71e2 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapRedTask.java a71faf8576 
>   service/src/jamon/org/apache/hive/tmpl/QueryProfileTmpl.jamon f04d655440 
>   service/src/resources/hive-webapps/static/css/query-plan-graph.css PRE-CREATION 
>   service/src/resources/hive-webapps/static/js/query-plan-graph.js PRE-CREATION 
>   service/src/resources/hive-webapps/static/js/vis.min.js PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/61663/diff/5/
> 
> 
> Testing
> -------
> 
> 
> File Attachments
> ----------------
> 
> HIVE-17300.7.patch
>   https://reviews.apache.org/media/uploaded/files/2018/09/07/e8ada965-ea33-48a2-a0b1-e56e8185c4fa__HIVE-17300.7.patch
> 
> 
> Thanks,
> 
> Karen Coppage
> 
>

Re: Review Request 61663: WebUI query plan graphs

Posted by Karen Coppage <ka...@gmail.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61663/
-----------------------------------------------------------

(Updated Sept. 7, 2018, 3:24 p.m.)


Review request for hive, Peter Vary and Xuefu Zhang.


Changes
-------

Added tests and checks, disabled zoom


Bugs: HIVE-17300
    https://issues.apache.org/jira/browse/HIVE-17300


Repository: hive-git


Description
-------

I’m working on a feature of the Hive WebUI Query Plan tab that would provide the option to display the query plan as a nice graph (scroll down for screenshots). If you click on one of the graph’s stages, the plan for that stage appears as text below.
Stages are color-coded if they have a status (Success, Error, Running), and the rest are grayed out. Coloring is based on status already available in the WebUI, under the Stages tab.
There is an additional option to display stats for MapReduce tasks. This includes the job’s ID, tracking URL (where the logs are found), and mapper and reducer numbers/progress, among other info.
The library I’m using for the graph is called vis.js (http://visjs.org/). It has an Apache license, and the only necessary file to be included from this library is about 700 KB.
I tried to keep server-side changes minimal, and graph generation is taken care of by the client. Plans with more than a given number of stages (default: 25) won't be displayed in order to preserve resources.


Diffs (updated)
-----

  common/src/java/org/apache/hadoop/hive/common/LogUtils.java 5068eb5be7 
  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 40ea3ac0c5 
  itests/hive-unit/src/test/java/org/apache/hive/service/cli/session/TestQueryDisplay.java 95b46a8149 
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java dad2035362 
  ql/src/java/org/apache/hadoop/hive/ql/MapRedStats.java ac45ec46de 
  ql/src/java/org/apache/hadoop/hive/ql/QueryDisplay.java 9a77c2969e 
  ql/src/java/org/apache/hadoop/hive/ql/exec/mr/HadoopJobExecHelper.java eb6cbf71e2 
  ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapRedTask.java a71faf8576 
  service/src/jamon/org/apache/hive/tmpl/QueryProfileTmpl.jamon f04d655440 
  service/src/resources/hive-webapps/static/css/query-plan-graph.css PRE-CREATION 
  service/src/resources/hive-webapps/static/js/query-plan-graph.js PRE-CREATION 
  service/src/resources/hive-webapps/static/js/vis.min.js PRE-CREATION 


Diff: https://reviews.apache.org/r/61663/diff/5/

Changes: https://reviews.apache.org/r/61663/diff/4-5/


Testing
-------


File Attachments (updated)
----------------

HIVE-17300.7.patch
  https://reviews.apache.org/media/uploaded/files/2018/09/07/e8ada965-ea33-48a2-a0b1-e56e8185c4fa__HIVE-17300.7.patch


Thanks,

Karen Coppage

Re: Review Request 61663: WebUI query plan graphs

Posted by Karen Coppage <ka...@gmail.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61663/
-----------------------------------------------------------

(Updated Aug. 30, 2018, 2:42 p.m.)

Review request for hive, Peter Vary and Xuefu Zhang.

Bugs: HIVE-17300
https://issues.apache.org/jira/browse/HIVE-17300

Repository: hive-git

Description
-------

I’m working on a feature of the Hive WebUI Query Plan tab that would provide the option to display the query plan as a nice graph (scroll down for screenshots). If you click on one of the graph’s stages, the plan for that stage appears as text below.
Stages are color-coded if they have a status (Success, Error, Running), and the rest are grayed out. Coloring is based on status already available in the WebUI, under the Stages tab.
There is an additional option to display stats for MapReduce tasks. This includes the job’s ID, tracking URL (where the logs are found), and mapper and reducer numbers/progress, among other info.
The library I’m using for the graph is called vis.js (http://visjs.org/). It has an Apache license, and the only necessary file to be included from this library is about 700 KB.
I tried to keep server-side changes minimal, and graph generation is taken care of by the client. Plans with more than a given number of stages (default: 25) won't be displayed in order to preserve resources.

Diffs
-----

common/src/java/org/apache/hadoop/hive/common/LogUtils.java 5068eb5be7
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 8c39de3e77
ql/src/java/org/apache/hadoop/hive/ql/Driver.java 737debd2ad
ql/src/java/org/apache/hadoop/hive/ql/MapRedStats.java ac45ec46de
ql/src/java/org/apache/hadoop/hive/ql/QueryDisplay.java 9a77c2969e
ql/src/java/org/apache/hadoop/hive/ql/exec/mr/HadoopJobExecHelper.java eb6cbf71e2
ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapRedTask.java a71faf8576
service/src/jamon/org/apache/hive/tmpl/QueryProfileTmpl.jamon f04d655440
service/src/resources/hive-webapps/static/css/query-plan-graph.css PRE-CREATION
service/src/resources/hive-webapps/static/js/query-plan-graph.js PRE-CREATION
service/src/resources/hive-webapps/static/js/vis.min.js PRE-CREATION

Diff: https://reviews.apache.org/r/61663/diff/4/

Testing
-------

Thanks,

Karen Coppage

Re: Review Request 61663: WebUI query plan graphs

Posted by Karen Coppage <ka...@gmail.com>.

(Updated Aug. 30, 2018, 1:09 p.m.)

Review request for hive, Peter Vary and Xuefu Zhang.

Changes
-------

Rebased patch (includes changes made by PVary in HIVE-17300.5.patch)

Repository: hive-git

Description
-------

Diffs (updated)
-----

Diff: https://reviews.apache.org/r/61663/diff/4/

Changes: https://reviews.apache.org/r/61663/diff/3-4/

Testing
-------

Thanks,

Karen Coppage

Re: Review Request 61663: WebUI query plan graphs

Posted by Karen Coppage <ka...@gmail.com>.

(Updated Aug. 16, 2017, 1:55 p.m.)

Review request for hive, Peter Vary and Xuefu Zhang.

Repository: hive-git

Description
-------

Diffs (updated)
-----

common/src/java/org/apache/hadoop/hive/common/LogUtils.java 0a3e0c7201
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 3c158a6692
ql/src/java/org/apache/hadoop/hive/ql/Driver.java 4e7c80f184
ql/src/java/org/apache/hadoop/hive/ql/MapRedStats.java 4b6051485e
ql/src/java/org/apache/hadoop/hive/ql/QueryDisplay.java bf6cb91745
ql/src/java/org/apache/hadoop/hive/ql/exec/mr/HadoopJobExecHelper.java 3c0719717c
ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapRedTask.java 1bd4db7805
service/src/jamon/org/apache/hive/tmpl/QueryProfileTmpl.jamon ff7476ee02
service/src/resources/hive-webapps/static/css/query-plan-graph.css PRE-CREATION
service/src/resources/hive-webapps/static/js/query-plan-graph.js PRE-CREATION
service/src/resources/hive-webapps/static/js/vis.min.js PRE-CREATION

Diff: https://reviews.apache.org/r/61663/diff/3/

Changes: https://reviews.apache.org/r/61663/diff/2-3/

Testing
-------

Thanks,

Karen Coppage

Re: Review Request 61663: WebUI query plan graphs

Posted by Karen Coppage <ka...@gmail.com>.

(Updated Aug. 16, 2017, 1:47 p.m.)

Review request for hive, Peter Vary and Xuefu Zhang.

Repository: hive-git

Description
-------

Diffs (updated)
-----

Diff: https://reviews.apache.org/r/61663/diff/2/

Changes: https://reviews.apache.org/r/61663/diff/1-2/

Testing
-------

Thanks,

Karen Coppage