[jira] [Commented] (HIVE-19671) Distribute by rand() can lead to data inconsistency

["Rui Li (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-19671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16517846#comment-16517846 ] 

Rui Li commented on HIVE-19671:
-------------------------------

[~xuefuz], thanks for your input. I think rand(seed) may not work if the mapper's input is not in deterministic order. As an example, suppose a mapper needs to process key {{1, 2, 3, 4, 5}}. The partition in 1st attempt is as below:
{noformat}
key   rand(seed)
 1  ->   1
 2  ->   2
 3  ->   3
 4  ->   4
 5  ->   5
{noformat}
So there'll be 5 reducers to fetch data from this mapper. Suppose the first 4 reducers have finished. And when the 5th reducer starts, the node hosting the mapper's output is lost, so the mapper is rerun. And the 2nd attempt has the following partition:
{noformat}
key   rand(seed)
 1  ->   1
 3  ->   2
 5  ->   3
 2  ->   4
 4  ->   5
{noformat}
Then the 5th reducer is rerun and fetches key 4, which means key 4 is duplicated and key 5 is lost.

To avoid the issue, we need to make sure record reader can guarantee an order when reading data from HDFS, and we don't use shuffling that doesn't order the keys, e.g. groupByKey of Spark. What do you think?

> Distribute by rand() can lead to data inconsistency
> ---------------------------------------------------
>
>                 Key: HIVE-19671
>                 URL: https://issues.apache.org/jira/browse/HIVE-19671
>             Project: Hive
>          Issue Type: Bug
>            Reporter: Rui Li
>            Assignee: Rui Li
>            Priority: Major
>
> Noticed the following queries can give different results:
> {code}
> select count(*) from tbl;
> select count(*) from (select * from tbl distribute by rand()) a;
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)