You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "George Pachitariu (JIRA)" <ji...@apache.org> on 2018/09/27 19:05:00 UTC
[jira] [Comment Edited] (HIVE-20523) Improve table statistics for
Parquet format
[ https://issues.apache.org/jira/browse/HIVE-20523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16630911#comment-16630911 ]
George Pachitariu edited comment on HIVE-20523 at 9/27/18 7:04 PM:
-------------------------------------------------------------------
Hi [~kgyrtkirk] , can you please have a look at this patch?
I have looked at the failed tests. They all failed because the expected raw data size has changed, which is the expected behaviour of this patch.
was (Author: george.pachitariu):
Hi Zoltan Haindrich, can you please have a look at this patch?
I have looked at the failed tests. They all failed because the expected raw data size has changed, which is the expected behaviour of this patch.
> Improve table statistics for Parquet format
> -------------------------------------------
>
> Key: HIVE-20523
> URL: https://issues.apache.org/jira/browse/HIVE-20523
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
> Reporter: George Pachitariu
> Assignee: George Pachitariu
> Priority: Minor
> Attachments: HIVE-20523.1.patch, HIVE-20523.2.patch, HIVE-20523.3.patch, HIVE-20523.4.patch, HIVE-20523.5.patch, HIVE-20523.6.patch, HIVE-20523.patch
>
>
> Right now, in the table basic statistics, the *raw data size* for a row with any data type in the Parquet format is 1. This is an underestimated value when columns are complex data structures, like arrays.
> Having tables with underestimated raw data size makes Hive assign less containers (mappers/reducers) to it, making the overall query slower.
> Heavy underestimation also makes Hive choose MapJoin instead of the ShuffleJoin that can fail with OOM errors.
> In this patch, I compute the columns data size better, taking into account complex structures. I followed the Writer implementation for the ORC format.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)