You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Rajesh Balamohan <ra...@gmail.com> on 2016/08/04 11:07:46 UTC
Review Request 50788: S3: Fetching partition sizes from FS can be
expensive when stats are not available in metastore
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50788/
-----------------------------------------------------------
Review request for hive and Ashutosh Chauhan.
Bugs: HIVE-14423
https://issues.apache.org/jira/browse/HIVE-14423
Repository: hive-git
Description
-------
When partition stats are not available in metastore, it tries to get the file sizes from FS.
e.g
at org.apache.hadoop.fs.FileSystem.getContentSummary(FileSystem.java:1487)
at org.apache.hadoop.hive.ql.stats.StatsUtils.getFileSizeForPartitions(StatsUtils.java:598)
at org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:235)
at org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:144)
at org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:132)
at org.apache.hadoop.hive.ql.optimizer.stats.annotation.StatsRulesProcFactory$TableScanStatsRule.process(StatsRulesProcFactory.java:126)
at org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
This can be quite expensive in some FS like S3. Especially when table is partitioned (e.g TPC-DS store_sales which has 1000s of partitions), query can spend 1000s of seconds just waiting for these information to be pulled in.
Also, it would be good to remove FS.getContentSummary usage to find out file sizes.
Diffs
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 9f5f619
ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java 2a9dc11
Diff: https://reviews.apache.org/r/50788/diff/
Testing
-------
Thanks,
Rajesh Balamohan
Re: Review Request 50788: S3: Fetching partition sizes from FS can be
expensive when stats are not available in metastore
Posted by Rajesh Balamohan <ra...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50788/
-----------------------------------------------------------
(Updated Aug. 4, 2016, 11:08 a.m.)
Review request for hive and Ashutosh Chauhan.
Bugs: HIVE-14423
https://issues.apache.org/jira/browse/HIVE-14423
Repository: hive-git
Description (updated)
-------
When partition stats are not available in metastore, it tries to get the file sizes from FS.
e.g
at org.apache.hadoop.fs.FileSystem.getContentSummary(FileSystem.java:1487)
at org.apache.hadoop.hive.ql.stats.StatsUtils.getFileSizeForPartitions(StatsUtils.java:598)
at org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:235)
at org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:144)
at org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:132)
at org.apache.hadoop.hive.ql.optimizer.stats.annotation.StatsRulesProcFactory$TableScanStatsRule.process(StatsRulesProcFactory.java:126)
at org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
This can be quite expensive in some FS like S3. Especially when table is partitioned (e.g TPC-DS store_sales which has 1000s of partitions), query can spend 1000s of seconds just waiting for these information to be pulled in.
Also, it would be good to remove FS.getContentSummary usage to find out file sizes.
With patch, timing comes down to ~80 seconds for file size gathering instead of 1000s of seconds.
Diffs
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 9f5f619
ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java 2a9dc11
Diff: https://reviews.apache.org/r/50788/diff/
Testing
-------
Thanks,
Rajesh Balamohan