You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Daniel Dai (JIRA)" <ji...@apache.org> on 2017/08/31 21:05:00 UTC
[jira] [Updated] (HIVE-17421) Clear incorrect stats after
replication
[ https://issues.apache.org/jira/browse/HIVE-17421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated HIVE-17421:
------------------------------
Description:
After replication, some stats summary are incorrect. If hive.compute.query.using.stats set to true, we will get wrong result on the destination side.
This will not happen with bootstrap replication. This is because stats summary are in table properties and will be replicated to the destination. However, in incremental replication, this won't work. When creating table, the stats summary are empty (eg, numRows=0). Later when we insert data, stats summary are updated with update_table_column_statistics/update_partition_column_statistics, however, both events are not captured in incremental replication. Thus on the destination side, we will get count\(*\)=0. The simple solution is to remove COLUMN_STATS_ACCURATE property after incremental replication.
was:
After replication, some stats summary are incorrect. If hive.compute.query.using.stats set to true, we will get wrong result on the destination side.
This will not happen with bootstrap replication. This is because stats summary are in table properties and will be replicated to the destination. However, in incremental replication, this won't work. When creating table, the stats summary are empty (eg, numRows=0). Later when we insert data, stats summary are updated with update_table_column_statistics/update_partition_column_statistics, however, both events are not captured in incremental replication. Thus on the destination side, we will get count(*)=0. The simple solution is to remove COLUMN_STATS_ACCURATE property after incremental replication.
> Clear incorrect stats after replication
> ---------------------------------------
>
> Key: HIVE-17421
> URL: https://issues.apache.org/jira/browse/HIVE-17421
> Project: Hive
> Issue Type: Bug
> Components: repl
> Reporter: Daniel Dai
> Assignee: Daniel Dai
>
> After replication, some stats summary are incorrect. If hive.compute.query.using.stats set to true, we will get wrong result on the destination side.
> This will not happen with bootstrap replication. This is because stats summary are in table properties and will be replicated to the destination. However, in incremental replication, this won't work. When creating table, the stats summary are empty (eg, numRows=0). Later when we insert data, stats summary are updated with update_table_column_statistics/update_partition_column_statistics, however, both events are not captured in incremental replication. Thus on the destination side, we will get count\(*\)=0. The simple solution is to remove COLUMN_STATS_ACCURATE property after incremental replication.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)