You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Gopal V (JIRA)" <ji...@apache.org> on 2017/07/03 05:11:00 UTC
[jira] [Updated] (HIVE-14797) reducer number estimating may lead to
data skew
[ https://issues.apache.org/jira/browse/HIVE-14797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gopal V updated HIVE-14797:
---------------------------
Labels: breaking_change (was: )
> reducer number estimating may lead to data skew
> -----------------------------------------------
>
> Key: HIVE-14797
> URL: https://issues.apache.org/jira/browse/HIVE-14797
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
> Reporter: roncenzhao
> Assignee: roncenzhao
> Labels: breaking_change
> Attachments: HIVE-14797.2.patch, HIVE-14797.3.patch, HIVE-14797.4.patch, HIVE-14797.patch
>
>
> HiveKey's hash code is generated by multipling by 31 key by key which is implemented in method `ObjectInspectorUtils.getBucketHashCode()`:
> for (int i = 0; i < bucketFields.length; i++) {
> int fieldHash = ObjectInspectorUtils.hashCode(bucketFields[i], bucketFieldInspectors[i]);
> hashCode = 31 * hashCode + fieldHash;
> }
> The follow example will lead to data skew:
> I hava two table called tbl1 and tbl2 and they have the same column: a int, b string. The values of column 'a' in both two tables are not skew, but values of column 'b' in both two tables are skew.
> When my sql is "select * from tbl1 join tbl2 on tbl1.a=tbl2.a and tbl1.b=tbl2.b" and the estimated reducer number is 31, it will lead to data skew.
> As we know, the HiveKey's hash code is generated by `hash(a)*31 + hash(b)`. When reducer number is 31 the reducer No. of each row is `hash(b)%31`. In the result, the job will be skew.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)