You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Kevin Wilfong (JIRA)" <ji...@apache.org> on 2011/08/10 02:03:27 UTC
[jira] [Created] (HIVE-2364) Make performance logging configurable.
Make performance logging configurable.
--------------------------------------
Key: HIVE-2364
URL: https://issues.apache.org/jira/browse/HIVE-2364
Project: Hive
Issue Type: Improvement
Reporter: Kevin Wilfong
Assignee: Kevin Wilfong
Currently, the way we measure the length of time spent by a piece of code is using the methods PerfLogBegin and PerfLogEnd in the Utilities class. If we made the class responsible for logging this data configurable, it would allow for more flexibility, in particular, we would not be restricted to logging to an Apache Commons Log.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2364) Make performance logging configurable.
Posted by "Kevin Wilfong (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kevin Wilfong updated HIVE-2364:
--------------------------------
Attachment: HIVE-2364.1.patch.txt
> Make performance logging configurable.
> --------------------------------------
>
> Key: HIVE-2364
> URL: https://issues.apache.org/jira/browse/HIVE-2364
> Project: Hive
> Issue Type: Improvement
> Reporter: Kevin Wilfong
> Assignee: Kevin Wilfong
> Attachments: HIVE-2364.1.patch.txt
>
>
> Currently, the way we measure the length of time spent by a piece of code is using the methods PerfLogBegin and PerfLogEnd in the Utilities class. If we made the class responsible for logging this data configurable, it would allow for more flexibility, in particular, we would not be restricted to logging to an Apache Commons Log.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2364) Make performance logging
configurable.
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13082017#comment-13082017 ]
jiraposter@reviews.apache.org commented on HIVE-2364:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1443/
-----------------------------------------------------------
(Updated 2011-08-10 00:18:50.035513)
Review request for hive and Ning Zhang.
Changes
-------
Forgot to add the licensing info to the top of PerfLogger.
Summary
-------
I created a new class PerfLogger, which wraps the old functionality of Utilities' PerfLogBegin and PerfLogBegin methods. I also added a config variable hive.exec.perf.logger. The value of this variable can be changed to point to a subclass of PerfLogger which can customize the code in PerfLogBegin and PerfLogEnd.
This addresses bug HIVE-2364.
https://issues.apache.org/jira/browse/HIVE-2364
Diffs (updated)
-----
trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1155439
trunk/conf/hive-default.xml 1155439
trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1155439
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 1155439
trunk/ql/src/java/org/apache/hadoop/hive/ql/log/PerfLogger.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java 1155439
Diff: https://reviews.apache.org/r/1443/diff
Testing
-------
I ran the code as it is and made sure it continued to log the performance messages as before.
I also created a subclass of PerfLogger and used its class name as the value of hive.exec.perf.logger, and verified the subclass's code was run.
Thanks,
Kevin
> Make performance logging configurable.
> --------------------------------------
>
> Key: HIVE-2364
> URL: https://issues.apache.org/jira/browse/HIVE-2364
> Project: Hive
> Issue Type: Improvement
> Reporter: Kevin Wilfong
> Assignee: Kevin Wilfong
> Attachments: HIVE-2364.1.patch.txt, HIVE-2364.2.patch.txt
>
>
> Currently, the way we measure the length of time spent by a piece of code is using the methods PerfLogBegin and PerfLogEnd in the Utilities class. If we made the class responsible for logging this data configurable, it would allow for more flexibility, in particular, we would not be restricted to logging to an Apache Commons Log.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2364) Make performance logging configurable.
Posted by "Kevin Wilfong (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kevin Wilfong updated HIVE-2364:
--------------------------------
Attachment: HIVE-2364.2.patch.txt
> Make performance logging configurable.
> --------------------------------------
>
> Key: HIVE-2364
> URL: https://issues.apache.org/jira/browse/HIVE-2364
> Project: Hive
> Issue Type: Improvement
> Reporter: Kevin Wilfong
> Assignee: Kevin Wilfong
> Attachments: HIVE-2364.1.patch.txt, HIVE-2364.2.patch.txt
>
>
> Currently, the way we measure the length of time spent by a piece of code is using the methods PerfLogBegin and PerfLogEnd in the Utilities class. If we made the class responsible for logging this data configurable, it would allow for more flexibility, in particular, we would not be restricted to logging to an Apache Commons Log.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2364) Make performance logging
configurable.
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13084754#comment-13084754 ]
jiraposter@reviews.apache.org commented on HIVE-2364:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1443/
-----------------------------------------------------------
(Updated 2011-08-14 00:54:16.595577)
Review request for hive and Ning Zhang.
Changes
-------
Updated patch.
Summary
-------
I created a new class PerfLogger, which wraps the old functionality of Utilities' PerfLogBegin and PerfLogBegin methods. I also added a config variable hive.exec.perf.logger. The value of this variable can be changed to point to a subclass of PerfLogger which can customize the code in PerfLogBegin and PerfLogEnd.
This addresses bug HIVE-2364.
https://issues.apache.org/jira/browse/HIVE-2364
Diffs (updated)
-----
trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1157444
trunk/conf/hive-default.xml 1157444
trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1157444
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 1157444
trunk/ql/src/java/org/apache/hadoop/hive/ql/log/PerfLogger.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java 1157444
Diff: https://reviews.apache.org/r/1443/diff
Testing
-------
I ran the code as it is and made sure it continued to log the performance messages as before.
I also created a subclass of PerfLogger and used its class name as the value of hive.exec.perf.logger, and verified the subclass's code was run.
Thanks,
Kevin
> Make performance logging configurable.
> --------------------------------------
>
> Key: HIVE-2364
> URL: https://issues.apache.org/jira/browse/HIVE-2364
> Project: Hive
> Issue Type: Improvement
> Reporter: Kevin Wilfong
> Assignee: Kevin Wilfong
> Attachments: HIVE-2364.1.patch.txt, HIVE-2364.2.patch.txt, HIVE-2364.3.patch.txt
>
>
> Currently, the way we measure the length of time spent by a piece of code is using the methods PerfLogBegin and PerfLogEnd in the Utilities class. If we made the class responsible for logging this data configurable, it would allow for more flexibility, in particular, we would not be restricted to logging to an Apache Commons Log.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2364) Make performance logging
configurable.
Posted by "Ning Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13084541#comment-13084541 ]
Ning Zhang commented on HIVE-2364:
----------------------------------
Kevin, this patch is out of date. Could you update it and repost it? Please ping me after you update it and I'll start testing right away.
> Make performance logging configurable.
> --------------------------------------
>
> Key: HIVE-2364
> URL: https://issues.apache.org/jira/browse/HIVE-2364
> Project: Hive
> Issue Type: Improvement
> Reporter: Kevin Wilfong
> Assignee: Kevin Wilfong
> Attachments: HIVE-2364.1.patch.txt, HIVE-2364.2.patch.txt
>
>
> Currently, the way we measure the length of time spent by a piece of code is using the methods PerfLogBegin and PerfLogEnd in the Utilities class. If we made the class responsible for logging this data configurable, it would allow for more flexibility, in particular, we would not be restricted to logging to an Apache Commons Log.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HIVE-2364) Make performance logging
configurable.
Posted by "Ning Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ning Zhang resolved HIVE-2364.
------------------------------
Resolution: Fixed
Fix Version/s: 0.8.0
Hadoop Flags: [Reviewed]
Committed. Thanks Kevin!
> Make performance logging configurable.
> --------------------------------------
>
> Key: HIVE-2364
> URL: https://issues.apache.org/jira/browse/HIVE-2364
> Project: Hive
> Issue Type: Improvement
> Reporter: Kevin Wilfong
> Assignee: Kevin Wilfong
> Fix For: 0.8.0
>
> Attachments: HIVE-2364.1.patch.txt, HIVE-2364.2.patch.txt, HIVE-2364.3.patch.txt
>
>
> Currently, the way we measure the length of time spent by a piece of code is using the methods PerfLogBegin and PerfLogEnd in the Utilities class. If we made the class responsible for logging this data configurable, it would allow for more flexibility, in particular, we would not be restricted to logging to an Apache Commons Log.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2364) Make performance logging
configurable.
Posted by "Kevin Wilfong (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13082012#comment-13082012 ]
Kevin Wilfong commented on HIVE-2364:
-------------------------------------
https://reviews.apache.org/r/1443/
> Make performance logging configurable.
> --------------------------------------
>
> Key: HIVE-2364
> URL: https://issues.apache.org/jira/browse/HIVE-2364
> Project: Hive
> Issue Type: Improvement
> Reporter: Kevin Wilfong
> Assignee: Kevin Wilfong
> Attachments: HIVE-2364.1.patch.txt
>
>
> Currently, the way we measure the length of time spent by a piece of code is using the methods PerfLogBegin and PerfLogEnd in the Utilities class. If we made the class responsible for logging this data configurable, it would allow for more flexibility, in particular, we would not be restricted to logging to an Apache Commons Log.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2364) Make performance logging
configurable.
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13084875#comment-13084875 ]
Hudson commented on HIVE-2364:
------------------------------
Integrated in Hive-trunk-h0.21 #894 (See [https://builds.apache.org/job/Hive-trunk-h0.21/894/])
HIVE-2364. Make performance logging configurable. (Kevin Wilfong via Ning Zhang)
nzhang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1157532
Files :
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
* /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/log/PerfLogger.java
* /hive/trunk/conf/hive-default.xml
> Make performance logging configurable.
> --------------------------------------
>
> Key: HIVE-2364
> URL: https://issues.apache.org/jira/browse/HIVE-2364
> Project: Hive
> Issue Type: Improvement
> Reporter: Kevin Wilfong
> Assignee: Kevin Wilfong
> Fix For: 0.8.0
>
> Attachments: HIVE-2364.1.patch.txt, HIVE-2364.2.patch.txt, HIVE-2364.3.patch.txt
>
>
> Currently, the way we measure the length of time spent by a piece of code is using the methods PerfLogBegin and PerfLogEnd in the Utilities class. If we made the class responsible for logging this data configurable, it would allow for more flexibility, in particular, we would not be restricted to logging to an Apache Commons Log.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2364) Make performance logging configurable.
Posted by "Kevin Wilfong (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kevin Wilfong updated HIVE-2364:
--------------------------------
Attachment: HIVE-2364.3.patch.txt
> Make performance logging configurable.
> --------------------------------------
>
> Key: HIVE-2364
> URL: https://issues.apache.org/jira/browse/HIVE-2364
> Project: Hive
> Issue Type: Improvement
> Reporter: Kevin Wilfong
> Assignee: Kevin Wilfong
> Attachments: HIVE-2364.1.patch.txt, HIVE-2364.2.patch.txt, HIVE-2364.3.patch.txt
>
>
> Currently, the way we measure the length of time spent by a piece of code is using the methods PerfLogBegin and PerfLogEnd in the Utilities class. If we made the class responsible for logging this data configurable, it would allow for more flexibility, in particular, we would not be restricted to logging to an Apache Commons Log.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2364) Make performance logging
configurable.
Posted by "Ning Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13084536#comment-13084536 ]
Ning Zhang commented on HIVE-2364:
----------------------------------
+1. Will commit if tests pass.
> Make performance logging configurable.
> --------------------------------------
>
> Key: HIVE-2364
> URL: https://issues.apache.org/jira/browse/HIVE-2364
> Project: Hive
> Issue Type: Improvement
> Reporter: Kevin Wilfong
> Assignee: Kevin Wilfong
> Attachments: HIVE-2364.1.patch.txt, HIVE-2364.2.patch.txt
>
>
> Currently, the way we measure the length of time spent by a piece of code is using the methods PerfLogBegin and PerfLogEnd in the Utilities class. If we made the class responsible for logging this data configurable, it would allow for more flexibility, in particular, we would not be restricted to logging to an Apache Commons Log.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2364) Make performance logging
configurable.
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13082014#comment-13082014 ]
jiraposter@reviews.apache.org commented on HIVE-2364:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1443/
-----------------------------------------------------------
Review request for hive and Ning Zhang.
Summary
-------
I created a new class PerfLogger, which wraps the old functionality of Utilities' PerfLogBegin and PerfLogBegin methods. I also added a config variable hive.exec.perf.logger. The value of this variable can be changed to point to a subclass of PerfLogger which can customize the code in PerfLogBegin and PerfLogEnd.
This addresses bug HIVE-2364.
https://issues.apache.org/jira/browse/HIVE-2364
Diffs
-----
trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1155439
trunk/conf/hive-default.xml 1155439
trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1155439
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 1155439
trunk/ql/src/java/org/apache/hadoop/hive/ql/log/PerfLogger.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java 1155439
Diff: https://reviews.apache.org/r/1443/diff
Testing
-------
I ran the code as it is and made sure it continued to log the performance messages as before.
I also created a subclass of PerfLogger and used its class name as the value of hive.exec.perf.logger, and verified the subclass's code was run.
Thanks,
Kevin
> Make performance logging configurable.
> --------------------------------------
>
> Key: HIVE-2364
> URL: https://issues.apache.org/jira/browse/HIVE-2364
> Project: Hive
> Issue Type: Improvement
> Reporter: Kevin Wilfong
> Assignee: Kevin Wilfong
> Attachments: HIVE-2364.1.patch.txt
>
>
> Currently, the way we measure the length of time spent by a piece of code is using the methods PerfLogBegin and PerfLogEnd in the Utilities class. If we made the class responsible for logging this data configurable, it would allow for more flexibility, in particular, we would not be restricted to logging to an Apache Commons Log.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira