You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Xin Hao (JIRA)" <ji...@apache.org> on 2015/02/03 08:41:34 UTC
[jira] [Commented] (HIVE-9560) When
hive.stats.collect.rawdatasize=true, 'rawDataSize' for an ORC table will
result in value '0' after running 'analyze table TABLE_NAME compute
statistics;'
[ https://issues.apache.org/jira/browse/HIVE-9560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14302891#comment-14302891 ]
Xin Hao commented on HIVE-9560:
-------------------------------
For example, we have an ORC table named 'item'.
(a) Before running 'analyze table item compute statistics;',
the 'rawDataSize' was '884720592'.
The result of 'describe extended item':
Detailed Table Information Table(tableName:item, dbName:bigbenchorc, owner:root, createTime:1421984899, lastAccessTime:0, retention:0, sd:StorageDescriptor(cols:[FieldSchema(name:i_item_sk, type:bigint, comment:null), FieldSchema(name:i_item_id, type:string, comment:null), FieldSchema(name:i_rec_start_date, type:string, comment:null), FieldSchema(name:i_rec_end_date, type:string, comment:null), FieldSchema(name:i_item_desc, type:string, comment:null), FieldSchema(name:i_current_price, type:double, comment:null), FieldSchema(name:i_wholesale_cost, type:double, comment:null), FieldSchema(name:i_brand_id, type:int, comment:null), FieldSchema(name:i_brand, type:string, comment:null), FieldSchema(name:i_class_id, type:int, comment:null), FieldSchema(name:i_class, type:string, comment:null), FieldSchema(name:i_category_id, type:int, comment:null), FieldSchema(name:i_category, type:string, comment:null), FieldSchema(name:i_manufact_id, type:int, comment:null), FieldSchema(name:i_manufact, type:string, comment:null), FieldSchema(name:i_size, type:string, comment:null), FieldSchema(name:i_formulation, type:string, comment:null), FieldSchema(name:i_color, type:string, comment:null), FieldSchema(name:i_units, type:string, comment:null), FieldSchema(name:i_container, type:string, comment:null), FieldSchema(name:i_manager_id, type:int, comment:null), FieldSchema(name:i_product_name, type:string, comment:null)], location:hdfs://bhx1:8020/user/hive/warehouse/bigbenchorc.db/item, inputFormat:org.apache.hadoop.hive.ql.io.orc.OrcInputFormat, outputFormat:org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat, compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null, serializationLib:org.apache.hadoop.hive.ql.io.orc.OrcSerde, parameters:{serialization.format=1}), bucketCols:[], sortCols:[], parameters:{}, skewedInfo:SkewedInfo(skewedColNames:[], skewedColValues:[], skewedColValueLocationMaps:{}), storedAsSubDirectories:false), partitionKeys:[], parameters:{numFiles=4, transient_lastDdlTime=1421984899, COLUMN_STATS_ACCURATE=true, totalSize=83267548, numRows=563518, rawDataSize=884720592}, viewOriginalText:null, viewExpandedText:null, tableType:MANAGED_TABLE)
Time taken: 0.527 seconds, Fetched: 24 row(s)
(b)After running 'analyze table TABLE_NAME compute statistics;'
the 'rawDataSize' will be changed to '0',
The result of 'describe extended item':
Detailed Table Information Table(tableName:item, dbName:bigbenchorc, owner:root, createTime:1421984899, lastAccessTime:0, retention:0, sd:StorageDescriptor(cols:[FieldSchema(name:i_item_sk, type:bigint, comment:null), FieldSchema(name:i_item_id, type:string, comment:null), FieldSchema(name:i_rec_start_date, type:string, comment:null), FieldSchema(name:i_rec_end_date, type:string, comment:null), FieldSchema(name:i_item_desc, type:string, comment:null), FieldSchema(name:i_current_price, type:double, comment:null), FieldSchema(name:i_wholesale_cost, type:double, comment:null), FieldSchema(name:i_brand_id, type:int, comment:null), FieldSchema(name:i_brand, type:string, comment:null), FieldSchema(name:i_class_id, type:int, comment:null), FieldSchema(name:i_class, type:string, comment:null), FieldSchema(name:i_category_id, type:int, comment:null), FieldSchema(name:i_category, type:string, comment:null), FieldSchema(name:i_manufact_id, type:int, comment:null), FieldSchema(name:i_manufact, type:string, comment:null), FieldSchema(name:i_size, type:string, comment:null), FieldSchema(name:i_formulation, type:string, comment:null), FieldSchema(name:i_color, type:string, comment:null), FieldSchema(name:i_units, type:string, comment:null), FieldSchema(name:i_container, type:string, comment:null), FieldSchema(name:i_manager_id, type:int, comment:null), FieldSchema(name:i_product_name, type:string, comment:null)], location:hdfs://bhx1:8020/user/hive/warehouse/bigbenchorc.db/item, inputFormat:org.apache.hadoop.hive.ql.io.orc.OrcInputFormat, outputFormat:org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat, compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null, serializationLib:org.apache.hadoop.hive.ql.io.orc.OrcSerde, parameters:{serialization.format=1}), bucketCols:[], sortCols:[], parameters:{}, skewedInfo:SkewedInfo(skewedColNames:[], skewedColValues:[], skewedColValueLocationMaps:{}), storedAsSubDirectories:false), partitionKeys:[], parameters:{numFiles=4, transient_lastDdlTime=1421984899, COLUMN_STATS_ACCURATE=true, totalSize=83267548, numRows=563518, rawDataSize=884720592}, viewOriginalText:null, viewExpandedText:null, tableType:MANAGED_TABLE)
Time taken: 0.527 seconds, Fetched: 24 row(s)
> When hive.stats.collect.rawdatasize=true, 'rawDataSize' for an ORC table will result in value '0' after running 'analyze table TABLE_NAME compute statistics;'
> --------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-9560
> URL: https://issues.apache.org/jira/browse/HIVE-9560
> Project: Hive
> Issue Type: Bug
> Reporter: Xin Hao
>
> When hive.stats.collect.rawdatasize=true, 'rawDataSize' for an ORC table will result in value '0' after running 'analyze table TABLE_NAME compute statistics;'
> Reproduce step:
> (1) set hive.stats.collect.rawdatasize=true；
> (2) Generate an ORC table in hive, and the value of its 'rawDataSize' is NOT zero.
> You can find the value of 'rawDataSize' (NOT zero) by executing 'describe extended TABLE_NAME;'
> (4) Execute 'analyze table TABLE_NAME compute statistics;'
> (5) Execute 'describe extended TABLE_NAME;' again, and you will find that the value of 'rawDataSize' will be changed to '0'.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)