You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Szehon Ho <sz...@cloudera.com> on 2016/02/01 21:05:29 UTC

Re: Review Request 43008: HIVE-12952 : Show query sub-pages on webui

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43008/
-----------------------------------------------------------

(Updated Feb. 1, 2016, 8:05 p.m.)


Review request for hive.


Changes
-------

Address Lenni review comments.

Also make a bug fix where compile-time PerfLogger was not getting displayed (it is on a different thread, so added a checkpoint at end of compile-phase to save it)


Bugs: HIVE-12952
    https://issues.apache.org/jira/browse/HIVE-12952


Repository: hive-git


Description
-------

This patch shows a query sub-page on WebUI, with detailed information of query on differnt tabs:

1.  Tabl- Base Info, ie user, query string, query id, begin time, end time, execution engine, error (if any)
2.  Tab2- Query Plan
3.  Tab3- Stages (MR jobs), their progress and info
4.  Tab4- Call trace info captured from HMSClient and PerfLogger.

Implementation notes:
The UI design choices are inspired from Impala, and HBase.  This, like HBase webui, uses Jamon, which is a superset of JSP and makes dynamic content a lot easier.  As such, brought in jamon dependency and also js bootstrap libraries to support the dynamic tabs.

On Hive side, refactored webui query logic into following classes:  SQLoperationDisplay (info captured from SQLOperation), QueryDisplay (info captured from Driver).


TODO:
1. Hard to get more MR job information for the stages including a job-tracking url, due to MR JobSubmission being a separate process, need to think about it.  Same for Spark/tez.
2. The explain plan might be a bit bulky and consume a bit of memory (though can tune with "hive.server2.webui.max.historic.queries").  Perhaps in future we can spill to local disk, and stream from there.  This might also help with (1), if we don't want to implement inter-process communciations.


Diffs (updated)
-----

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java ff376a8 
  common/src/java/org/apache/hadoop/hive/ql/log/PerfLogger.java d4194cf 
  common/src/java/org/apache/hive/http/HttpServer.java 9e23b11 
  pom.xml 802d3d4 
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java 4c89812 
  ql/src/java/org/apache/hadoop/hive/ql/QueryDisplay.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 0bab769 
  service/pom.xml b2e3a84 
  service/src/jamon/org/apache/hive/tmpl/QueryProfileTmpl.jamon PRE-CREATION 
  service/src/java/org/apache/hive/service/cli/operation/Operation.java 0c263cf 
  service/src/java/org/apache/hive/service/cli/operation/OperationManager.java f1ce6f6 
  service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 01b1d3d 
  service/src/java/org/apache/hive/service/cli/operation/SQLOperationDisplay.java PRE-CREATION 
  service/src/java/org/apache/hive/service/cli/operation/SQLOperationDisplayCache.java PRE-CREATION 
  service/src/java/org/apache/hive/service/cli/operation/SQLOperationInfo.java 179f6dd 
  service/src/java/org/apache/hive/service/server/HiveServer2.java 958458f 
  service/src/java/org/apache/hive/service/servlet/QueryProfileServlet.java PRE-CREATION 
  service/src/resources/hive-webapps/hiveserver2/hiveserver2.jsp a0b5d2e 
  service/src/resources/hive-webapps/static/js/bootstrap.js PRE-CREATION 
  service/src/resources/hive-webapps/static/js/bootstrap.min.js PRE-CREATION 
  service/src/resources/hive-webapps/static/js/jquery.min.js PRE-CREATION 
  service/src/resources/hive-webapps/static/js/tab.js PRE-CREATION 

Diff: https://reviews.apache.org/r/43008/diff/


Testing
-------

Manual testing.  Can add some unit tests in follow-up.


Thanks,

Szehon Ho


Re: Review Request 43008: HIVE-12952 : Show query sub-pages on webui

Posted by Szehon Ho <sz...@cloudera.com>.

> On Feb. 1, 2016, 11:32 p.m., Aihua Xu wrote:
> > service/src/java/org/apache/hive/service/cli/operation/OperationManager.java, line 73
> > <https://reviews.apache.org/r/43008/diff/2/?file=1228425#file1228425line73>
> >
> >     Can we just use Map<HandleIdentifier, SQLOperationDisplay> directly?

Yea, but unfortunately I'm passing in via Http request (see QueryProfileServlet) so I think I lookup via string.. I thought about this but couldn't go backward to get HandleIdentifier from String, especially as toString() only the public part perhaps for some security-obscurity reasons?


> On Feb. 1, 2016, 11:32 p.m., Aihua Xu wrote:
> > service/src/java/org/apache/hive/service/cli/operation/SQLOperationDisplayCache.java, line 35
> > <https://reviews.apache.org/r/43008/diff/2/?file=1228428#file1228428line35>
> >
> >     Add @Override to this memthod to be a little clear since it's an override.

Done


> On Feb. 1, 2016, 11:32 p.m., Aihua Xu wrote:
> > service/src/resources/hive-webapps/static/js/bootstrap.js, line 85
> > <https://reviews.apache.org/r/43008/diff/2/?file=1228433#file1228433line85>
> >
> >     Is this the style on the same line?

This is just added from Twitter bootstrap open-source js.. http://getbootstrap.com/getting-started/  I probably dont want to change it, to keep the lines consistent with what people download.


> On Feb. 1, 2016, 11:32 p.m., Aihua Xu wrote:
> > service/src/resources/hive-webapps/static/js/tab.js, line 29
> > <https://reviews.apache.org/r/43008/diff/2/?file=1228436#file1228436line29>
> >
> >     Remove the spaces here and below.

Done


- Szehon


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43008/#review117303
-----------------------------------------------------------


On Feb. 1, 2016, 8:05 p.m., Szehon Ho wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/43008/
> -----------------------------------------------------------
> 
> (Updated Feb. 1, 2016, 8:05 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-12952
>     https://issues.apache.org/jira/browse/HIVE-12952
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> This patch shows a query sub-page on WebUI, with detailed information of query on differnt tabs:
> 
> 1.  Tabl- Base Info, ie user, query string, query id, begin time, end time, execution engine, error (if any)
> 2.  Tab2- Query Plan
> 3.  Tab3- Stages (MR jobs), their progress and info
> 4.  Tab4- Call trace info captured from HMSClient and PerfLogger.
> 
> Implementation notes:
> The UI design choices are inspired from Impala, and HBase.  This, like HBase webui, uses Jamon, which is a superset of JSP and makes dynamic content a lot easier.  As such, brought in jamon dependency and also js bootstrap libraries to support the dynamic tabs.
> 
> On Hive side, refactored webui query logic into following classes:  SQLoperationDisplay (info captured from SQLOperation), QueryDisplay (info captured from Driver).
> 
> 
> TODO:
> 1. Hard to get more MR job information for the stages including a job-tracking url, due to MR JobSubmission being a separate process, need to think about it.  Same for Spark/tez.
> 2. The explain plan might be a bit bulky and consume a bit of memory (though can tune with "hive.server2.webui.max.historic.queries").  Perhaps in future we can spill to local disk, and stream from there.  This might also help with (1), if we don't want to implement inter-process communciations.
> 
> 
> Diffs
> -----
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java ff376a8 
>   common/src/java/org/apache/hadoop/hive/ql/log/PerfLogger.java d4194cf 
>   common/src/java/org/apache/hive/http/HttpServer.java 9e23b11 
>   pom.xml 802d3d4 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 4c89812 
>   ql/src/java/org/apache/hadoop/hive/ql/QueryDisplay.java PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 0bab769 
>   service/pom.xml b2e3a84 
>   service/src/jamon/org/apache/hive/tmpl/QueryProfileTmpl.jamon PRE-CREATION 
>   service/src/java/org/apache/hive/service/cli/operation/Operation.java 0c263cf 
>   service/src/java/org/apache/hive/service/cli/operation/OperationManager.java f1ce6f6 
>   service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 01b1d3d 
>   service/src/java/org/apache/hive/service/cli/operation/SQLOperationDisplay.java PRE-CREATION 
>   service/src/java/org/apache/hive/service/cli/operation/SQLOperationDisplayCache.java PRE-CREATION 
>   service/src/java/org/apache/hive/service/cli/operation/SQLOperationInfo.java 179f6dd 
>   service/src/java/org/apache/hive/service/server/HiveServer2.java 958458f 
>   service/src/java/org/apache/hive/service/servlet/QueryProfileServlet.java PRE-CREATION 
>   service/src/resources/hive-webapps/hiveserver2/hiveserver2.jsp a0b5d2e 
>   service/src/resources/hive-webapps/static/js/bootstrap.js PRE-CREATION 
>   service/src/resources/hive-webapps/static/js/bootstrap.min.js PRE-CREATION 
>   service/src/resources/hive-webapps/static/js/jquery.min.js PRE-CREATION 
>   service/src/resources/hive-webapps/static/js/tab.js PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/43008/diff/
> 
> 
> Testing
> -------
> 
> Manual testing.  Can add some unit tests in follow-up.
> 
> 
> Thanks,
> 
> Szehon Ho
> 
>


Re: Review Request 43008: HIVE-12952 : Show query sub-pages on webui

Posted by Aihua Xu <ax...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43008/#review117303
-----------------------------------------------------------




service/src/java/org/apache/hive/service/cli/operation/OperationManager.java (line 73)
<https://reviews.apache.org/r/43008/#comment178462>

    Can we just use Map<HandleIdentifier, SQLOperationDisplay> directly?



service/src/java/org/apache/hive/service/cli/operation/SQLOperationDisplayCache.java (line 35)
<https://reviews.apache.org/r/43008/#comment178463>

    Add @Override to this memthod to be a little clear since it's an override.



service/src/resources/hive-webapps/static/js/bootstrap.js (line 85)
<https://reviews.apache.org/r/43008/#comment178464>

    Is this the style on the same line?



service/src/resources/hive-webapps/static/js/tab.js (line 29)
<https://reviews.apache.org/r/43008/#comment178465>

    Remove the spaces here and below.


- Aihua Xu


On Feb. 1, 2016, 8:05 p.m., Szehon Ho wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/43008/
> -----------------------------------------------------------
> 
> (Updated Feb. 1, 2016, 8:05 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-12952
>     https://issues.apache.org/jira/browse/HIVE-12952
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> This patch shows a query sub-page on WebUI, with detailed information of query on differnt tabs:
> 
> 1.  Tabl- Base Info, ie user, query string, query id, begin time, end time, execution engine, error (if any)
> 2.  Tab2- Query Plan
> 3.  Tab3- Stages (MR jobs), their progress and info
> 4.  Tab4- Call trace info captured from HMSClient and PerfLogger.
> 
> Implementation notes:
> The UI design choices are inspired from Impala, and HBase.  This, like HBase webui, uses Jamon, which is a superset of JSP and makes dynamic content a lot easier.  As such, brought in jamon dependency and also js bootstrap libraries to support the dynamic tabs.
> 
> On Hive side, refactored webui query logic into following classes:  SQLoperationDisplay (info captured from SQLOperation), QueryDisplay (info captured from Driver).
> 
> 
> TODO:
> 1. Hard to get more MR job information for the stages including a job-tracking url, due to MR JobSubmission being a separate process, need to think about it.  Same for Spark/tez.
> 2. The explain plan might be a bit bulky and consume a bit of memory (though can tune with "hive.server2.webui.max.historic.queries").  Perhaps in future we can spill to local disk, and stream from there.  This might also help with (1), if we don't want to implement inter-process communciations.
> 
> 
> Diffs
> -----
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java ff376a8 
>   common/src/java/org/apache/hadoop/hive/ql/log/PerfLogger.java d4194cf 
>   common/src/java/org/apache/hive/http/HttpServer.java 9e23b11 
>   pom.xml 802d3d4 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 4c89812 
>   ql/src/java/org/apache/hadoop/hive/ql/QueryDisplay.java PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 0bab769 
>   service/pom.xml b2e3a84 
>   service/src/jamon/org/apache/hive/tmpl/QueryProfileTmpl.jamon PRE-CREATION 
>   service/src/java/org/apache/hive/service/cli/operation/Operation.java 0c263cf 
>   service/src/java/org/apache/hive/service/cli/operation/OperationManager.java f1ce6f6 
>   service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 01b1d3d 
>   service/src/java/org/apache/hive/service/cli/operation/SQLOperationDisplay.java PRE-CREATION 
>   service/src/java/org/apache/hive/service/cli/operation/SQLOperationDisplayCache.java PRE-CREATION 
>   service/src/java/org/apache/hive/service/cli/operation/SQLOperationInfo.java 179f6dd 
>   service/src/java/org/apache/hive/service/server/HiveServer2.java 958458f 
>   service/src/java/org/apache/hive/service/servlet/QueryProfileServlet.java PRE-CREATION 
>   service/src/resources/hive-webapps/hiveserver2/hiveserver2.jsp a0b5d2e 
>   service/src/resources/hive-webapps/static/js/bootstrap.js PRE-CREATION 
>   service/src/resources/hive-webapps/static/js/bootstrap.min.js PRE-CREATION 
>   service/src/resources/hive-webapps/static/js/jquery.min.js PRE-CREATION 
>   service/src/resources/hive-webapps/static/js/tab.js PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/43008/diff/
> 
> 
> Testing
> -------
> 
> Manual testing.  Can add some unit tests in follow-up.
> 
> 
> Thanks,
> 
> Szehon Ho
> 
>


Re: Review Request 43008: HIVE-12952 : Show query sub-pages on webui

Posted by Aihua Xu <ax...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43008/#review117405
-----------------------------------------------------------


Ship it!




Ship It!

- Aihua Xu


On Feb. 2, 2016, 12:22 a.m., Szehon Ho wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/43008/
> -----------------------------------------------------------
> 
> (Updated Feb. 2, 2016, 12:22 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-12952
>     https://issues.apache.org/jira/browse/HIVE-12952
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> This patch shows a query sub-page on WebUI, with detailed information of query on differnt tabs:
> 
> 1.  Tabl- Base Info, ie user, query string, query id, begin time, end time, execution engine, error (if any)
> 2.  Tab2- Query Plan
> 3.  Tab3- Stages (MR jobs), their progress and info
> 4.  Tab4- Call trace info captured from HMSClient and PerfLogger.
> 
> Implementation notes:
> The UI design choices are inspired from Impala, and HBase.  This, like HBase webui, uses Jamon, which is a superset of JSP and makes dynamic content a lot easier.  As such, brought in jamon dependency and also js bootstrap libraries to support the dynamic tabs.
> 
> On Hive side, refactored webui query logic into following classes:  SQLoperationDisplay (info captured from SQLOperation), QueryDisplay (info captured from Driver).
> 
> 
> TODO:
> 1. Hard to get more MR job information for the stages including a job-tracking url, due to MR JobSubmission being a separate process, need to think about it.  Same for Spark/tez.
> 2. The explain plan might be a bit bulky and consume a bit of memory (though can tune with "hive.server2.webui.max.historic.queries").  Perhaps in future we can spill to local disk, and stream from there.  This might also help with (1), if we don't want to implement inter-process communciations.
> 
> 
> Diffs
> -----
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java ff376a8 
>   common/src/java/org/apache/hadoop/hive/ql/log/PerfLogger.java d4194cf 
>   common/src/java/org/apache/hive/http/HttpServer.java 9e23b11 
>   pom.xml 802d3d4 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 4c89812 
>   ql/src/java/org/apache/hadoop/hive/ql/QueryDisplay.java PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 0bab769 
>   service/pom.xml b2e3a84 
>   service/src/jamon/org/apache/hive/tmpl/QueryProfileTmpl.jamon PRE-CREATION 
>   service/src/java/org/apache/hive/service/cli/operation/Operation.java 0c263cf 
>   service/src/java/org/apache/hive/service/cli/operation/OperationManager.java f1ce6f6 
>   service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 01b1d3d 
>   service/src/java/org/apache/hive/service/cli/operation/SQLOperationDisplay.java PRE-CREATION 
>   service/src/java/org/apache/hive/service/cli/operation/SQLOperationDisplayCache.java PRE-CREATION 
>   service/src/java/org/apache/hive/service/cli/operation/SQLOperationInfo.java 179f6dd 
>   service/src/java/org/apache/hive/service/server/HiveServer2.java 958458f 
>   service/src/java/org/apache/hive/service/servlet/QueryProfileServlet.java PRE-CREATION 
>   service/src/resources/hive-webapps/hiveserver2/hiveserver2.jsp a0b5d2e 
>   service/src/resources/hive-webapps/static/js/bootstrap.js PRE-CREATION 
>   service/src/resources/hive-webapps/static/js/bootstrap.min.js PRE-CREATION 
>   service/src/resources/hive-webapps/static/js/jquery.min.js PRE-CREATION 
>   service/src/resources/hive-webapps/static/js/tab.js PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/43008/diff/
> 
> 
> Testing
> -------
> 
> Manual testing.  Can add some unit tests in follow-up.
> 
> 
> Thanks,
> 
> Szehon Ho
> 
>


Re: Review Request 43008: HIVE-12952 : Show query sub-pages on webui

Posted by Szehon Ho <sz...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43008/
-----------------------------------------------------------

(Updated Feb. 2, 2016, 12:22 a.m.)


Review request for hive.


Bugs: HIVE-12952
    https://issues.apache.org/jira/browse/HIVE-12952


Repository: hive-git


Description
-------

This patch shows a query sub-page on WebUI, with detailed information of query on differnt tabs:

1.  Tabl- Base Info, ie user, query string, query id, begin time, end time, execution engine, error (if any)
2.  Tab2- Query Plan
3.  Tab3- Stages (MR jobs), their progress and info
4.  Tab4- Call trace info captured from HMSClient and PerfLogger.

Implementation notes:
The UI design choices are inspired from Impala, and HBase.  This, like HBase webui, uses Jamon, which is a superset of JSP and makes dynamic content a lot easier.  As such, brought in jamon dependency and also js bootstrap libraries to support the dynamic tabs.

On Hive side, refactored webui query logic into following classes:  SQLoperationDisplay (info captured from SQLOperation), QueryDisplay (info captured from Driver).


TODO:
1. Hard to get more MR job information for the stages including a job-tracking url, due to MR JobSubmission being a separate process, need to think about it.  Same for Spark/tez.
2. The explain plan might be a bit bulky and consume a bit of memory (though can tune with "hive.server2.webui.max.historic.queries").  Perhaps in future we can spill to local disk, and stream from there.  This might also help with (1), if we don't want to implement inter-process communciations.


Diffs (updated)
-----

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java ff376a8 
  common/src/java/org/apache/hadoop/hive/ql/log/PerfLogger.java d4194cf 
  common/src/java/org/apache/hive/http/HttpServer.java 9e23b11 
  pom.xml 802d3d4 
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java 4c89812 
  ql/src/java/org/apache/hadoop/hive/ql/QueryDisplay.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 0bab769 
  service/pom.xml b2e3a84 
  service/src/jamon/org/apache/hive/tmpl/QueryProfileTmpl.jamon PRE-CREATION 
  service/src/java/org/apache/hive/service/cli/operation/Operation.java 0c263cf 
  service/src/java/org/apache/hive/service/cli/operation/OperationManager.java f1ce6f6 
  service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 01b1d3d 
  service/src/java/org/apache/hive/service/cli/operation/SQLOperationDisplay.java PRE-CREATION 
  service/src/java/org/apache/hive/service/cli/operation/SQLOperationDisplayCache.java PRE-CREATION 
  service/src/java/org/apache/hive/service/cli/operation/SQLOperationInfo.java 179f6dd 
  service/src/java/org/apache/hive/service/server/HiveServer2.java 958458f 
  service/src/java/org/apache/hive/service/servlet/QueryProfileServlet.java PRE-CREATION 
  service/src/resources/hive-webapps/hiveserver2/hiveserver2.jsp a0b5d2e 
  service/src/resources/hive-webapps/static/js/bootstrap.js PRE-CREATION 
  service/src/resources/hive-webapps/static/js/bootstrap.min.js PRE-CREATION 
  service/src/resources/hive-webapps/static/js/jquery.min.js PRE-CREATION 
  service/src/resources/hive-webapps/static/js/tab.js PRE-CREATION 

Diff: https://reviews.apache.org/r/43008/diff/


Testing
-------

Manual testing.  Can add some unit tests in follow-up.


Thanks,

Szehon Ho


Re: Review Request 43008: HIVE-12952 : Show query sub-pages on webui

Posted by Szehon Ho <sz...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43008/
-----------------------------------------------------------

(Updated Feb. 2, 2016, 12:18 a.m.)


Review request for hive.


Changes
-------

Address review comments.


Bugs: HIVE-12952
    https://issues.apache.org/jira/browse/HIVE-12952


Repository: hive-git


Description
-------

This patch shows a query sub-page on WebUI, with detailed information of query on differnt tabs:

1.  Tabl- Base Info, ie user, query string, query id, begin time, end time, execution engine, error (if any)
2.  Tab2- Query Plan
3.  Tab3- Stages (MR jobs), their progress and info
4.  Tab4- Call trace info captured from HMSClient and PerfLogger.

Implementation notes:
The UI design choices are inspired from Impala, and HBase.  This, like HBase webui, uses Jamon, which is a superset of JSP and makes dynamic content a lot easier.  As such, brought in jamon dependency and also js bootstrap libraries to support the dynamic tabs.

On Hive side, refactored webui query logic into following classes:  SQLoperationDisplay (info captured from SQLOperation), QueryDisplay (info captured from Driver).


TODO:
1. Hard to get more MR job information for the stages including a job-tracking url, due to MR JobSubmission being a separate process, need to think about it.  Same for Spark/tez.
2. The explain plan might be a bit bulky and consume a bit of memory (though can tune with "hive.server2.webui.max.historic.queries").  Perhaps in future we can spill to local disk, and stream from there.  This might also help with (1), if we don't want to implement inter-process communciations.


Diffs (updated)
-----

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java ff376a8 
  common/src/java/org/apache/hadoop/hive/ql/log/PerfLogger.java d4194cf 
  common/src/java/org/apache/hive/http/HttpServer.java 9e23b11 
  pom.xml 802d3d4 
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java 4c89812 
  ql/src/java/org/apache/hadoop/hive/ql/QueryDisplay.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 0bab769 
  service/pom.xml b2e3a84 
  service/src/jamon/org/apache/hive/tmpl/QueryProfileTmpl.jamon PRE-CREATION 
  service/src/java/org/apache/hive/service/cli/operation/Operation.java 0c263cf 
  service/src/java/org/apache/hive/service/cli/operation/OperationManager.java f1ce6f6 
  service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 01b1d3d 
  service/src/java/org/apache/hive/service/cli/operation/SQLOperationDisplay.java PRE-CREATION 
  service/src/java/org/apache/hive/service/cli/operation/SQLOperationDisplayCache.java PRE-CREATION 
  service/src/java/org/apache/hive/service/cli/operation/SQLOperationInfo.java 179f6dd 
  service/src/java/org/apache/hive/service/server/HiveServer2.java 958458f 
  service/src/java/org/apache/hive/service/servlet/QueryProfileServlet.java PRE-CREATION 
  service/src/resources/hive-webapps/hiveserver2/hiveserver2.jsp a0b5d2e 
  service/src/resources/hive-webapps/static/js/bootstrap.js PRE-CREATION 
  service/src/resources/hive-webapps/static/js/bootstrap.min.js PRE-CREATION 
  service/src/resources/hive-webapps/static/js/jquery.min.js PRE-CREATION 
  service/src/resources/hive-webapps/static/js/tab.js PRE-CREATION 

Diff: https://reviews.apache.org/r/43008/diff/


Testing
-------

Manual testing.  Can add some unit tests in follow-up.


Thanks,

Szehon Ho