You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by GitBox <gi...@apache.org> on 2020/07/23 01:17:15 UTC

[GitHub] [hive] hsnusonic opened a new pull request #1301: HIVE-23901: Overhead of Logger in ColumnStatsMerger damage the perfor…

hsnusonic opened a new pull request #1301:
URL: https://github.com/apache/hive/pull/1301


   …mance


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org

[GitHub] [hive] nrg4878 commented on pull request #1301: HIVE-23901: Overhead of Logger in ColumnStatsMerger damage the perfor…

Posted by GitBox <gi...@apache.org>.
nrg4878 commented on pull request #1301:
URL: https://github.com/apache/hive/pull/1301#issuecomment-671419902


   fix has been committed to master. Please close out this PR.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org

[GitHub] [hive] hsnusonic commented on pull request #1301: HIVE-23901: Overhead of Logger in ColumnStatsMerger damage the perfor…

Posted by GitBox <gi...@apache.org>.
hsnusonic commented on pull request #1301:
URL: https://github.com/apache/hive/pull/1301#issuecomment-662803187


   @belugabehr Thank you for pointing out that.
   
   The problem happens in [here](https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/columnstats/aggr/LongColumnStatsAggregator.java#L110) and other aggregators. When a table has many partitions, the merger is being instantiated repeatedly.
   
   Yeah, I think each subclass has its own Logger is a good idea.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org

[GitHub] [hive] hsnusonic commented on pull request #1301: HIVE-23901: Overhead of Logger in ColumnStatsMerger damage the perfor…

Posted by GitBox <gi...@apache.org>.
hsnusonic commented on pull request #1301:
URL: https://github.com/apache/hive/pull/1301#issuecomment-664685908


   @nrg4878 the method doMerge is declared as protected originally and is used only in class *ColumnStatsMerger, so it seems fine to be reduced to only one method merge.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org

[GitHub] [hive] hsnusonic closed pull request #1301: HIVE-23901: Overhead of Logger in ColumnStatsMerger damage the perfor…

Posted by GitBox <gi...@apache.org>.
hsnusonic closed pull request #1301:
URL: https://github.com/apache/hive/pull/1301


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org

[GitHub] [hive] belugabehr commented on pull request #1301: HIVE-23901: Overhead of Logger in ColumnStatsMerger damage the perfor…

Posted by GitBox <gi...@apache.org>.
belugabehr commented on pull request #1301:
URL: https://github.com/apache/hive/pull/1301#issuecomment-662777247


   The current implementation is purposeful.  This is an abstract class.  The idea is that every subclass gets it's own logger.  With the implementation proposed in this PR, every subclass will have the same logger.  This makes logging very difficult to tease out.
   
   Can you please elaborate on the performance implications here?  If it's absolutely required, it is better that each subclass have their own Logger than to inherit a logger in this fashion.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org

[GitHub] [hive] nrg4878 commented on pull request #1301: HIVE-23901: Overhead of Logger in ColumnStatsMerger damage the perfor…

Posted by GitBox <gi...@apache.org>.
nrg4878 commented on pull request #1301:
URL: https://github.com/apache/hive/pull/1301#issuecomment-664659969


   IsColumnStatsMerger a class internal to hive? I think it is but I am not certain. Otherwise changing the method name might be an issue.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org

[GitHub] [hive] belugabehr commented on pull request #1301: HIVE-23901: Overhead of Logger in ColumnStatsMerger damage the perfor…

Posted by GitBox <gi...@apache.org>.
belugabehr commented on pull request #1301:
URL: https://github.com/apache/hive/pull/1301#issuecomment-662778176


   This only happens if these objects are being instantiated repeatedly. May need to look at that.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org