You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Rajat Goel (Created) (JIRA)" <ji...@apache.org> on 2011/12/27 09:26:30 UTC
[jira] [Created] (HIVE-2682) Clean-up logs
Clean-up logs
-------------
Key: HIVE-2682
URL: https://issues.apache.org/jira/browse/HIVE-2682
Project: Hive
Issue Type: Wish
Components: Logging
Reporter: Rajat Goel
Priority: Trivial
Just wanted to cleanup some logs being printed at wrong loglevel -
1. org.apache.hadoop.hive.ql.exec.CommonJoinOperator prints "table 0 has 1000 rows for join key [...]" as WARNING. Is it really that?
2. org.apache.hadoop.hive.ql.exec.GroupByOperator prints "Hash Table completed flushed" and "Begin Hash Table flush at close: size = 21" as WARNING. It shouldn't be.
3. org.apache.hadoop.fs.FileSystem prints "Not initializing standby filesystem because the needed configuration parameters fs.default.name{0|1} are missing." as WARNING. Doesn't look like a problem since it is happening at very high rate.
4. org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher prints "Warning. Invalid statistic." which looks fishy.
5. org.apache.hadoop.mapred.Merger prints "totalBytesProcessed: ABC is larger than totalBytesExpected: XYZ" as ERROR at high rate...do we actually care about this??
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2682) Clean-up logs
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13184822#comment-13184822 ]
Hudson commented on HIVE-2682:
------------------------------
Integrated in Hive-trunk-h0.21 #1195 (See [https://builds.apache.org/job/Hive-trunk-h0.21/1195/])
HIVE-2682: Clean-up logs (Rajat Goel via Ashutosh Chauhan)
hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1230379
Files :
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/JoinOperator.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsAggregator.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsPublisher.java
* /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyStruct.java
* /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryStruct.java
> Clean-up logs
> -------------
>
> Key: HIVE-2682
> URL: https://issues.apache.org/jira/browse/HIVE-2682
> Project: Hive
> Issue Type: Wish
> Components: Logging
> Affects Versions: 0.8.1, 0.9.0
> Reporter: Rajat Goel
> Assignee: Rajat Goel
> Priority: Trivial
> Labels: logging
> Fix For: 0.9.0
>
> Attachments: HIVE-2682.D1035.1.patch, HIVE-2682.D1035.2.patch, HIVE-2682.D1035.3.patch, hive-2682.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> Just wanted to cleanup some logs being printed at wrong loglevel -
> 1. org.apache.hadoop.hive.ql.exec.CommonJoinOperator prints "table 0 has 1000 rows for join key [...]" as WARNING. Is it really that?
> 2. org.apache.hadoop.hive.ql.exec.GroupByOperator prints "Hash Table completed flushed" and "Begin Hash Table flush at close: size = 21" as WARNING. It shouldn't be.
> 3. org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher prints "Warning. Invalid statistic." which looks fishy.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2682) Clean-up logs
Posted by "Rajat Goel (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rajat Goel updated HIVE-2682:
-----------------------------
Description:
Just wanted to cleanup some logs being printed at wrong loglevel -
1. org.apache.hadoop.hive.ql.exec.CommonJoinOperator prints "table 0 has 1000 rows for join key [...]" as WARNING. Is it really that?
2. org.apache.hadoop.hive.ql.exec.GroupByOperator prints "Hash Table completed flushed" and "Begin Hash Table flush at close: size = 21" as WARNING. It shouldn't be.
4. org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher prints "Warning. Invalid statistic." which looks fishy.
was:
Just wanted to cleanup some logs being printed at wrong loglevel -
1. org.apache.hadoop.hive.ql.exec.CommonJoinOperator prints "table 0 has 1000 rows for join key [...]" as WARNING. Is it really that?
2. org.apache.hadoop.hive.ql.exec.GroupByOperator prints "Hash Table completed flushed" and "Begin Hash Table flush at close: size = 21" as WARNING. It shouldn't be.
3. org.apache.hadoop.fs.FileSystem prints "Not initializing standby filesystem because the needed configuration parameters fs.default.name{0|1} are missing." as WARNING. Doesn't look like a problem since it is happening at very high rate.
4. org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher prints "Warning. Invalid statistic." which looks fishy.
5. org.apache.hadoop.mapred.Merger prints "totalBytesProcessed: ABC is larger than totalBytesExpected: XYZ" as ERROR at high rate...do we actually care about this??
> Clean-up logs
> -------------
>
> Key: HIVE-2682
> URL: https://issues.apache.org/jira/browse/HIVE-2682
> Project: Hive
> Issue Type: Wish
> Components: Logging
> Reporter: Rajat Goel
> Priority: Trivial
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> Just wanted to cleanup some logs being printed at wrong loglevel -
> 1. org.apache.hadoop.hive.ql.exec.CommonJoinOperator prints "table 0 has 1000 rows for join key [...]" as WARNING. Is it really that?
> 2. org.apache.hadoop.hive.ql.exec.GroupByOperator prints "Hash Table completed flushed" and "Begin Hash Table flush at close: size = 21" as WARNING. It shouldn't be.
> 4. org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher prints "Warning. Invalid statistic." which looks fishy.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2682) Clean-up logs
Posted by "Rajat Goel (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rajat Goel updated HIVE-2682:
-----------------------------
Labels: logging (was: )
Affects Version/s: 0.9.0
0.8.1
Release Note: Some trivial cleanup of logs.
Hadoop Flags: Reviewed
Status: Patch Available (was: Open)
> Clean-up logs
> -------------
>
> Key: HIVE-2682
> URL: https://issues.apache.org/jira/browse/HIVE-2682
> Project: Hive
> Issue Type: Wish
> Components: Logging
> Affects Versions: 0.8.1, 0.9.0
> Reporter: Rajat Goel
> Assignee: Rajat Goel
> Priority: Trivial
> Labels: logging
> Attachments: HIVE-2682.D1035.1.patch, HIVE-2682.D1035.2.patch, HIVE-2682.D1035.3.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> Just wanted to cleanup some logs being printed at wrong loglevel -
> 1. org.apache.hadoop.hive.ql.exec.CommonJoinOperator prints "table 0 has 1000 rows for join key [...]" as WARNING. Is it really that?
> 2. org.apache.hadoop.hive.ql.exec.GroupByOperator prints "Hash Table completed flushed" and "Begin Hash Table flush at close: size = 21" as WARNING. It shouldn't be.
> 3. org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher prints "Warning. Invalid statistic." which looks fishy.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2682) Clean-up logs
Posted by "Phabricator (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HIVE-2682:
------------------------------
Attachment: HIVE-2682.D1035.2.patch
rajat updated the revision "HIVE-2682 [jira] Clean-up logs".
Reviewers: JIRA, jsichi, jonchang, heyongqiang, njain
Took care of the comments
REVISION DETAIL
https://reviews.facebook.net/D1035
AFFECTED FILES
ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java
ql/src/java/org/apache/hadoop/hive/ql/exec/JoinOperator.java
ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsAggregator.java
ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsPublisher.java
> Clean-up logs
> -------------
>
> Key: HIVE-2682
> URL: https://issues.apache.org/jira/browse/HIVE-2682
> Project: Hive
> Issue Type: Wish
> Components: Logging
> Reporter: Rajat Goel
> Priority: Trivial
> Attachments: HIVE-2682.D1035.1.patch, HIVE-2682.D1035.2.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> Just wanted to cleanup some logs being printed at wrong loglevel -
> 1. org.apache.hadoop.hive.ql.exec.CommonJoinOperator prints "table 0 has 1000 rows for join key [...]" as WARNING. Is it really that?
> 2. org.apache.hadoop.hive.ql.exec.GroupByOperator prints "Hash Table completed flushed" and "Begin Hash Table flush at close: size = 21" as WARNING. It shouldn't be.
> 3. org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher prints "Warning. Invalid statistic." which looks fishy.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2682) Clean-up logs
Posted by "Ashutosh Chauhan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13184047#comment-13184047 ]
Ashutosh Chauhan commented on HIVE-2682:
----------------------------------------
Click on More Actions button at top of this page then click on Add Files and then add your patch. Don't forget to click on radio button to grant perms.
> Clean-up logs
> -------------
>
> Key: HIVE-2682
> URL: https://issues.apache.org/jira/browse/HIVE-2682
> Project: Hive
> Issue Type: Wish
> Components: Logging
> Affects Versions: 0.8.1, 0.9.0
> Reporter: Rajat Goel
> Assignee: Rajat Goel
> Priority: Trivial
> Labels: logging
> Attachments: HIVE-2682.D1035.1.patch, HIVE-2682.D1035.2.patch, HIVE-2682.D1035.3.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> Just wanted to cleanup some logs being printed at wrong loglevel -
> 1. org.apache.hadoop.hive.ql.exec.CommonJoinOperator prints "table 0 has 1000 rows for join key [...]" as WARNING. Is it really that?
> 2. org.apache.hadoop.hive.ql.exec.GroupByOperator prints "Hash Table completed flushed" and "Begin Hash Table flush at close: size = 21" as WARNING. It shouldn't be.
> 3. org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher prints "Warning. Invalid statistic." which looks fishy.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2682) Clean-up logs
Posted by "Rajat Goel (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13184043#comment-13184043 ]
Rajat Goel commented on HIVE-2682:
----------------------------------
Definitely. How to do that? I have added you as a reviewer at
https://reviews.facebook.net/D1035
And thanks a lot of looking at the diff.
Rajat
On Wed, Jan 11, 2012 at 18:17, Ashutosh Chauhan (Commented) (JIRA)
> Clean-up logs
> -------------
>
> Key: HIVE-2682
> URL: https://issues.apache.org/jira/browse/HIVE-2682
> Project: Hive
> Issue Type: Wish
> Components: Logging
> Reporter: Rajat Goel
> Assignee: Rajat Goel
> Priority: Trivial
> Attachments: HIVE-2682.D1035.1.patch, HIVE-2682.D1035.2.patch, HIVE-2682.D1035.3.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> Just wanted to cleanup some logs being printed at wrong loglevel -
> 1. org.apache.hadoop.hive.ql.exec.CommonJoinOperator prints "table 0 has 1000 rows for join key [...]" as WARNING. Is it really that?
> 2. org.apache.hadoop.hive.ql.exec.GroupByOperator prints "Hash Table completed flushed" and "Begin Hash Table flush at close: size = 21" as WARNING. It shouldn't be.
> 3. org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher prints "Warning. Invalid statistic." which looks fishy.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2682) Clean-up logs
Posted by "Phabricator (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HIVE-2682:
------------------------------
Attachment: HIVE-2682.D1035.3.patch
rajat updated the revision "HIVE-2682 [jira] Clean-up logs".
Reviewers: JIRA, jsichi, jonchang, heyongqiang, njain
Incorporated suggestions from Dymtro (dms).
REVISION DETAIL
https://reviews.facebook.net/D1035
AFFECTED FILES
ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java
ql/src/java/org/apache/hadoop/hive/ql/exec/JoinOperator.java
ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsAggregator.java
ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsPublisher.java
serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyStruct.java
serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryStruct.java
> Clean-up logs
> -------------
>
> Key: HIVE-2682
> URL: https://issues.apache.org/jira/browse/HIVE-2682
> Project: Hive
> Issue Type: Wish
> Components: Logging
> Reporter: Rajat Goel
> Priority: Trivial
> Attachments: HIVE-2682.D1035.1.patch, HIVE-2682.D1035.2.patch, HIVE-2682.D1035.3.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> Just wanted to cleanup some logs being printed at wrong loglevel -
> 1. org.apache.hadoop.hive.ql.exec.CommonJoinOperator prints "table 0 has 1000 rows for join key [...]" as WARNING. Is it really that?
> 2. org.apache.hadoop.hive.ql.exec.GroupByOperator prints "Hash Table completed flushed" and "Begin Hash Table flush at close: size = 21" as WARNING. It shouldn't be.
> 3. org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher prints "Warning. Invalid statistic." which looks fishy.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2682) Clean-up logs
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13184042#comment-13184042 ]
Phabricator commented on HIVE-2682:
-----------------------------------
rajat has commented on the revision "HIVE-2682 [jira] Clean-up logs".
REVISION DETAIL
https://reviews.facebook.net/D1035
> Clean-up logs
> -------------
>
> Key: HIVE-2682
> URL: https://issues.apache.org/jira/browse/HIVE-2682
> Project: Hive
> Issue Type: Wish
> Components: Logging
> Reporter: Rajat Goel
> Assignee: Rajat Goel
> Priority: Trivial
> Attachments: HIVE-2682.D1035.1.patch, HIVE-2682.D1035.2.patch, HIVE-2682.D1035.3.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> Just wanted to cleanup some logs being printed at wrong loglevel -
> 1. org.apache.hadoop.hive.ql.exec.CommonJoinOperator prints "table 0 has 1000 rows for join key [...]" as WARNING. Is it really that?
> 2. org.apache.hadoop.hive.ql.exec.GroupByOperator prints "Hash Table completed flushed" and "Begin Hash Table flush at close: size = 21" as WARNING. It shouldn't be.
> 3. org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher prints "Warning. Invalid statistic." which looks fishy.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2682) Clean-up logs
Posted by "Phabricator (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HIVE-2682:
------------------------------
Attachment: HIVE-2682.D1035.1.patch
rajat requested code review of "HIVE-2682 [jira] Clean-up logs".
Reviewers: JIRA
Clean-up logs
Just wanted to cleanup some logs being printed at wrong loglevel -
1. org.apache.hadoop.hive.ql.exec.CommonJoinOperator prints "table 0 has 1000 rows for join key [...]" as WARNING. Is it really that?
2. org.apache.hadoop.hive.ql.exec.GroupByOperator prints "Hash Table completed flushed" and "Begin Hash Table flush at close: size = 21" as WARNING. It shouldn't be.
4. org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher prints "Warning. Invalid statistic." which looks fishy.
TEST PLAN
EMPTY
REVISION DETAIL
https://reviews.facebook.net/D1035
AFFECTED FILES
ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java
ql/src/java/org/apache/hadoop/hive/ql/exec/JoinOperator.java
ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsPublisher.java
MANAGE HERALD DIFFERENTIAL RULES
https://reviews.facebook.net/herald/view/differential/
WHY DID I GET THIS EMAIL?
https://reviews.facebook.net/herald/transcript/2169/
Tip: use the X-Herald-Rules header to filter Herald messages in your client.
> Clean-up logs
> -------------
>
> Key: HIVE-2682
> URL: https://issues.apache.org/jira/browse/HIVE-2682
> Project: Hive
> Issue Type: Wish
> Components: Logging
> Reporter: Rajat Goel
> Priority: Trivial
> Attachments: HIVE-2682.D1035.1.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> Just wanted to cleanup some logs being printed at wrong loglevel -
> 1. org.apache.hadoop.hive.ql.exec.CommonJoinOperator prints "table 0 has 1000 rows for join key [...]" as WARNING. Is it really that?
> 2. org.apache.hadoop.hive.ql.exec.GroupByOperator prints "Hash Table completed flushed" and "Begin Hash Table flush at close: size = 21" as WARNING. It shouldn't be.
> 4. org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher prints "Warning. Invalid statistic." which looks fishy.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2682) Clean-up logs
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13176269#comment-13176269 ]
Phabricator commented on HIVE-2682:
-----------------------------------
raghotham has commented on the revision "HIVE-2682 [jira] Clean-up logs".
INLINE COMMENTS
ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsPublisher.java:128 you dont really need to say 'Warning' - it will show up in the log anyway.
ql/src/java/org/apache/hadoop/hive/ql/exec/JoinOperator.java:112-114 what does this comment even mean? :)
REVISION DETAIL
https://reviews.facebook.net/D1035
> Clean-up logs
> -------------
>
> Key: HIVE-2682
> URL: https://issues.apache.org/jira/browse/HIVE-2682
> Project: Hive
> Issue Type: Wish
> Components: Logging
> Reporter: Rajat Goel
> Priority: Trivial
> Attachments: HIVE-2682.D1035.1.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> Just wanted to cleanup some logs being printed at wrong loglevel -
> 1. org.apache.hadoop.hive.ql.exec.CommonJoinOperator prints "table 0 has 1000 rows for join key [...]" as WARNING. Is it really that?
> 2. org.apache.hadoop.hive.ql.exec.GroupByOperator prints "Hash Table completed flushed" and "Begin Hash Table flush at close: size = 21" as WARNING. It shouldn't be.
> 3. org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher prints "Warning. Invalid statistic." which looks fishy.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HIVE-2682) Clean-up logs
Posted by "Ashutosh Chauhan (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ashutosh Chauhan reassigned HIVE-2682:
--------------------------------------
Assignee: Rajat Goel
> Clean-up logs
> -------------
>
> Key: HIVE-2682
> URL: https://issues.apache.org/jira/browse/HIVE-2682
> Project: Hive
> Issue Type: Wish
> Components: Logging
> Reporter: Rajat Goel
> Assignee: Rajat Goel
> Priority: Trivial
> Attachments: HIVE-2682.D1035.1.patch, HIVE-2682.D1035.2.patch, HIVE-2682.D1035.3.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> Just wanted to cleanup some logs being printed at wrong loglevel -
> 1. org.apache.hadoop.hive.ql.exec.CommonJoinOperator prints "table 0 has 1000 rows for join key [...]" as WARNING. Is it really that?
> 2. org.apache.hadoop.hive.ql.exec.GroupByOperator prints "Hash Table completed flushed" and "Begin Hash Table flush at close: size = 21" as WARNING. It shouldn't be.
> 3. org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher prints "Warning. Invalid statistic." which looks fishy.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2682) Clean-up logs
Posted by "Ashutosh Chauhan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13184030#comment-13184030 ]
Ashutosh Chauhan commented on HIVE-2682:
----------------------------------------
Can you upload the patch on jira directly and grant perms?
> Clean-up logs
> -------------
>
> Key: HIVE-2682
> URL: https://issues.apache.org/jira/browse/HIVE-2682
> Project: Hive
> Issue Type: Wish
> Components: Logging
> Reporter: Rajat Goel
> Assignee: Rajat Goel
> Priority: Trivial
> Attachments: HIVE-2682.D1035.1.patch, HIVE-2682.D1035.2.patch, HIVE-2682.D1035.3.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> Just wanted to cleanup some logs being printed at wrong loglevel -
> 1. org.apache.hadoop.hive.ql.exec.CommonJoinOperator prints "table 0 has 1000 rows for join key [...]" as WARNING. Is it really that?
> 2. org.apache.hadoop.hive.ql.exec.GroupByOperator prints "Hash Table completed flushed" and "Begin Hash Table flush at close: size = 21" as WARNING. It shouldn't be.
> 3. org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher prints "Warning. Invalid statistic." which looks fishy.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2682) Clean-up logs
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13184506#comment-13184506 ]
Phabricator commented on HIVE-2682:
-----------------------------------
njain has accepted the revision "HIVE-2682 [jira] Clean-up logs".
REVISION DETAIL
https://reviews.facebook.net/D1035
> Clean-up logs
> -------------
>
> Key: HIVE-2682
> URL: https://issues.apache.org/jira/browse/HIVE-2682
> Project: Hive
> Issue Type: Wish
> Components: Logging
> Affects Versions: 0.8.1, 0.9.0
> Reporter: Rajat Goel
> Assignee: Rajat Goel
> Priority: Trivial
> Labels: logging
> Attachments: HIVE-2682.D1035.1.patch, HIVE-2682.D1035.2.patch, HIVE-2682.D1035.3.patch, hive-2682.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> Just wanted to cleanup some logs being printed at wrong loglevel -
> 1. org.apache.hadoop.hive.ql.exec.CommonJoinOperator prints "table 0 has 1000 rows for join key [...]" as WARNING. Is it really that?
> 2. org.apache.hadoop.hive.ql.exec.GroupByOperator prints "Hash Table completed flushed" and "Begin Hash Table flush at close: size = 21" as WARNING. It shouldn't be.
> 3. org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher prints "Warning. Invalid statistic." which looks fishy.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2682) Clean-up logs
Posted by "Rajat Goel (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rajat Goel updated HIVE-2682:
-----------------------------
Attachment: hive-2682.patch
Generated via
"git format-patch origin --stdout > hive-2862.patch"
> Clean-up logs
> -------------
>
> Key: HIVE-2682
> URL: https://issues.apache.org/jira/browse/HIVE-2682
> Project: Hive
> Issue Type: Wish
> Components: Logging
> Affects Versions: 0.8.1, 0.9.0
> Reporter: Rajat Goel
> Assignee: Rajat Goel
> Priority: Trivial
> Labels: logging
> Attachments: HIVE-2682.D1035.1.patch, HIVE-2682.D1035.2.patch, HIVE-2682.D1035.3.patch, hive-2682.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> Just wanted to cleanup some logs being printed at wrong loglevel -
> 1. org.apache.hadoop.hive.ql.exec.CommonJoinOperator prints "table 0 has 1000 rows for join key [...]" as WARNING. Is it really that?
> 2. org.apache.hadoop.hive.ql.exec.GroupByOperator prints "Hash Table completed flushed" and "Begin Hash Table flush at close: size = 21" as WARNING. It shouldn't be.
> 3. org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher prints "Warning. Invalid statistic." which looks fishy.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2682) Clean-up logs
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13187020#comment-13187020 ]
Hudson commented on HIVE-2682:
------------------------------
Integrated in Hive-trunk-h0.21-dbg #5 (See [https://builds.apache.org/job/Hive-trunk-h0.21-dbg/5/])
HIVE-2682: Clean-up logs (Rajat Goel via Ashutosh Chauhan)
hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1230379
Files :
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/JoinOperator.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsAggregator.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsPublisher.java
* /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyStruct.java
* /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryStruct.java
> Clean-up logs
> -------------
>
> Key: HIVE-2682
> URL: https://issues.apache.org/jira/browse/HIVE-2682
> Project: Hive
> Issue Type: Wish
> Components: Logging
> Affects Versions: 0.8.1, 0.9.0
> Reporter: Rajat Goel
> Assignee: Rajat Goel
> Priority: Trivial
> Labels: logging
> Fix For: 0.9.0
>
> Attachments: HIVE-2682.D1035.1.patch, HIVE-2682.D1035.2.patch, HIVE-2682.D1035.3.patch, hive-2682.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> Just wanted to cleanup some logs being printed at wrong loglevel -
> 1. org.apache.hadoop.hive.ql.exec.CommonJoinOperator prints "table 0 has 1000 rows for join key [...]" as WARNING. Is it really that?
> 2. org.apache.hadoop.hive.ql.exec.GroupByOperator prints "Hash Table completed flushed" and "Begin Hash Table flush at close: size = 21" as WARNING. It shouldn't be.
> 3. org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher prints "Warning. Invalid statistic." which looks fishy.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2682) Clean-up logs
Posted by "Namit Jain (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13184505#comment-13184505 ]
Namit Jain commented on HIVE-2682:
----------------------------------
Sorry Ashutosh, didn't notice that you are looking at it.
> Clean-up logs
> -------------
>
> Key: HIVE-2682
> URL: https://issues.apache.org/jira/browse/HIVE-2682
> Project: Hive
> Issue Type: Wish
> Components: Logging
> Affects Versions: 0.8.1, 0.9.0
> Reporter: Rajat Goel
> Assignee: Rajat Goel
> Priority: Trivial
> Labels: logging
> Attachments: HIVE-2682.D1035.1.patch, HIVE-2682.D1035.2.patch, HIVE-2682.D1035.3.patch, hive-2682.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> Just wanted to cleanup some logs being printed at wrong loglevel -
> 1. org.apache.hadoop.hive.ql.exec.CommonJoinOperator prints "table 0 has 1000 rows for join key [...]" as WARNING. Is it really that?
> 2. org.apache.hadoop.hive.ql.exec.GroupByOperator prints "Hash Table completed flushed" and "Begin Hash Table flush at close: size = 21" as WARNING. It shouldn't be.
> 3. org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher prints "Warning. Invalid statistic." which looks fishy.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2682) Clean-up logs
Posted by "Rajat Goel (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rajat Goel updated HIVE-2682:
-----------------------------
Description:
Just wanted to cleanup some logs being printed at wrong loglevel -
1. org.apache.hadoop.hive.ql.exec.CommonJoinOperator prints "table 0 has 1000 rows for join key [...]" as WARNING. Is it really that?
2. org.apache.hadoop.hive.ql.exec.GroupByOperator prints "Hash Table completed flushed" and "Begin Hash Table flush at close: size = 21" as WARNING. It shouldn't be.
3. org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher prints "Warning. Invalid statistic." which looks fishy.
was:
Just wanted to cleanup some logs being printed at wrong loglevel -
1. org.apache.hadoop.hive.ql.exec.CommonJoinOperator prints "table 0 has 1000 rows for join key [...]" as WARNING. Is it really that?
2. org.apache.hadoop.hive.ql.exec.GroupByOperator prints "Hash Table completed flushed" and "Begin Hash Table flush at close: size = 21" as WARNING. It shouldn't be.
4. org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher prints "Warning. Invalid statistic." which looks fishy.
> Clean-up logs
> -------------
>
> Key: HIVE-2682
> URL: https://issues.apache.org/jira/browse/HIVE-2682
> Project: Hive
> Issue Type: Wish
> Components: Logging
> Reporter: Rajat Goel
> Priority: Trivial
> Attachments: HIVE-2682.D1035.1.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> Just wanted to cleanup some logs being printed at wrong loglevel -
> 1. org.apache.hadoop.hive.ql.exec.CommonJoinOperator prints "table 0 has 1000 rows for join key [...]" as WARNING. Is it really that?
> 2. org.apache.hadoop.hive.ql.exec.GroupByOperator prints "Hash Table completed flushed" and "Begin Hash Table flush at close: size = 21" as WARNING. It shouldn't be.
> 3. org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher prints "Warning. Invalid statistic." which looks fishy.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2682) Clean-up logs
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13186777#comment-13186777 ]
Hudson commented on HIVE-2682:
------------------------------
Integrated in Hive-trunk-h0.21-dbg #3 (See [https://builds.apache.org/job/Hive-trunk-h0.21-dbg/3/])
HIVE-2682: Clean-up logs (Rajat Goel via Ashutosh Chauhan)
hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1230379
Files :
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/JoinOperator.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsAggregator.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsPublisher.java
* /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyStruct.java
* /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryStruct.java
> Clean-up logs
> -------------
>
> Key: HIVE-2682
> URL: https://issues.apache.org/jira/browse/HIVE-2682
> Project: Hive
> Issue Type: Wish
> Components: Logging
> Affects Versions: 0.8.1, 0.9.0
> Reporter: Rajat Goel
> Assignee: Rajat Goel
> Priority: Trivial
> Labels: logging
> Fix For: 0.9.0
>
> Attachments: HIVE-2682.D1035.1.patch, HIVE-2682.D1035.2.patch, HIVE-2682.D1035.3.patch, hive-2682.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> Just wanted to cleanup some logs being printed at wrong loglevel -
> 1. org.apache.hadoop.hive.ql.exec.CommonJoinOperator prints "table 0 has 1000 rows for join key [...]" as WARNING. Is it really that?
> 2. org.apache.hadoop.hive.ql.exec.GroupByOperator prints "Hash Table completed flushed" and "Begin Hash Table flush at close: size = 21" as WARNING. It shouldn't be.
> 3. org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher prints "Warning. Invalid statistic." which looks fishy.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2682) Clean-up logs
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13176429#comment-13176429 ]
Phabricator commented on HIVE-2682:
-----------------------------------
rajat has commented on the revision "HIVE-2682 [jira] Clean-up logs".
INLINE COMMENTS
ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsPublisher.java:128 :P
Then I need to change this in multiple places.
ql/src/java/org/apache/hadoop/hive/ql/exec/JoinOperator.java:112-114 I think this is related to skew joins. If we touch 1000 rows for a single key, there is a chance of data being skewed for that key and it needs to be handled specially. (I am just guessing, though).
REVISION DETAIL
https://reviews.facebook.net/D1035
> Clean-up logs
> -------------
>
> Key: HIVE-2682
> URL: https://issues.apache.org/jira/browse/HIVE-2682
> Project: Hive
> Issue Type: Wish
> Components: Logging
> Reporter: Rajat Goel
> Priority: Trivial
> Attachments: HIVE-2682.D1035.1.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> Just wanted to cleanup some logs being printed at wrong loglevel -
> 1. org.apache.hadoop.hive.ql.exec.CommonJoinOperator prints "table 0 has 1000 rows for join key [...]" as WARNING. Is it really that?
> 2. org.apache.hadoop.hive.ql.exec.GroupByOperator prints "Hash Table completed flushed" and "Begin Hash Table flush at close: size = 21" as WARNING. It shouldn't be.
> 3. org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher prints "Warning. Invalid statistic." which looks fishy.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2682) Clean-up logs
Posted by "Ashutosh Chauhan (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ashutosh Chauhan updated HIVE-2682:
-----------------------------------
Resolution: Fixed
Fix Version/s: 0.9.0
Status: Resolved (was: Patch Available)
Committed to trunk. Thanks, Rajat!
> Clean-up logs
> -------------
>
> Key: HIVE-2682
> URL: https://issues.apache.org/jira/browse/HIVE-2682
> Project: Hive
> Issue Type: Wish
> Components: Logging
> Affects Versions: 0.8.1, 0.9.0
> Reporter: Rajat Goel
> Assignee: Rajat Goel
> Priority: Trivial
> Labels: logging
> Fix For: 0.9.0
>
> Attachments: HIVE-2682.D1035.1.patch, HIVE-2682.D1035.2.patch, HIVE-2682.D1035.3.patch, hive-2682.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> Just wanted to cleanup some logs being printed at wrong loglevel -
> 1. org.apache.hadoop.hive.ql.exec.CommonJoinOperator prints "table 0 has 1000 rows for join key [...]" as WARNING. Is it really that?
> 2. org.apache.hadoop.hive.ql.exec.GroupByOperator prints "Hash Table completed flushed" and "Begin Hash Table flush at close: size = 21" as WARNING. It shouldn't be.
> 3. org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher prints "Warning. Invalid statistic." which looks fishy.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2682) Clean-up logs
Posted by "Ashutosh Chauhan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13183941#comment-13183941 ]
Ashutosh Chauhan commented on HIVE-2682:
----------------------------------------
+1 Looks good. Will commit once tests pass.
> Clean-up logs
> -------------
>
> Key: HIVE-2682
> URL: https://issues.apache.org/jira/browse/HIVE-2682
> Project: Hive
> Issue Type: Wish
> Components: Logging
> Reporter: Rajat Goel
> Priority: Trivial
> Attachments: HIVE-2682.D1035.1.patch, HIVE-2682.D1035.2.patch, HIVE-2682.D1035.3.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> Just wanted to cleanup some logs being printed at wrong loglevel -
> 1. org.apache.hadoop.hive.ql.exec.CommonJoinOperator prints "table 0 has 1000 rows for join key [...]" as WARNING. Is it really that?
> 2. org.apache.hadoop.hive.ql.exec.GroupByOperator prints "Hash Table completed flushed" and "Begin Hash Table flush at close: size = 21" as WARNING. It shouldn't be.
> 3. org.apache.hadoop.hive.ql.stats.jdbc.JDBCStatsPublisher prints "Warning. Invalid statistic." which looks fishy.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira