[jira] [Commented] (HIVE-21167) Bucketing: Bucketing version 1 is incorrectly partitioning data

["JithendhiraKumar (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-21167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17042189#comment-17042189 ] 

JithendhiraKumar commented on HIVE-21167:
-----------------------------------------

[~djaiswal] I tried backporting this patch to branch-3 & branch-3.1, it didn't apply cleanly on them.

Can someone please help in backporting this patch to branch-3 & branch-3.1?

[~vgarg] [~jdere]

 

> Bucketing: Bucketing version 1 is incorrectly partitioning data
> ---------------------------------------------------------------
>
>                 Key: HIVE-21167
>                 URL: https://issues.apache.org/jira/browse/HIVE-21167
>             Project: Hive
>          Issue Type: Bug
>    Affects Versions: 3.1.1
>            Reporter: Vaibhav Gumashta
>            Assignee: Deepak Jaiswal
>            Priority: Major
>             Fix For: 4.0.0
>
>         Attachments: HIVE-21167.1.patch, HIVE-21167.2.patch, HIVE-21167.3.patch, HIVE-21167.4.patch
>
>
> Using murmur hash for bucketing columns was introduced in HIVE-18910, following which {{'bucketing_version'='1'}} stands for the old behaviour (where for example integer columns were partitioned based on mod values). Looks like we have a bug in the old bucketing scheme now. I could repro it when modified the existing schema using an alter table add column and adding new data. Repro:
> {code}
> 0: jdbc:hive2://localhost:10010> create transactional table acid_ptn_bucket1 (a int, b int) partitioned by(ds string) clustered by (a) into 2 buckets stored as ORC TBLPROPERTIES('bucketing_version'='1', 'transactional'='true', 'transactional_properties'='default');
> No rows affected (0.418 seconds)
> 0: jdbc:hive2://localhost:10010> insert into acid_ptn_bucket1 partition (ds) values(1,2,'today'),(1,3,'today'),(1,4,'yesterday'),(2,2,'yesterday'),(2,3,'today'),(2,4,'today');
> 6 rows affected (3.695 seconds)
> {code}
> Data from ORC file (data as expected):
> {code}
> /apps/hive/warehouse/acid_ptn_bucket1/ds=today/delta_0000001_0000001_0000/bucket_00000
> {"operation": 0, "originalTransaction": 1, "bucket": 536870912, "rowId": 0, "currentTransaction": 1, "row": {"a": 2, "b": 4}}
> {"operation": 0, "originalTransaction": 1, "bucket": 536870912, "rowId": 1, "currentTransaction": 1, "row": {"a": 2, "b": 3}}
> /apps/hive/warehouse/acid_ptn_bucket1/ds=today/delta_0000001_0000001_0000/bucket_00001
> {"operation": 0, "originalTransaction": 1, "bucket": 536936448, "rowId": 0, "currentTransaction": 1, "row": {"a": 1, "b": 3}}
> {"operation": 0, "originalTransaction": 1, "bucket": 536936448, "rowId": 1, "currentTransaction": 1, "row": {"a": 1, "b": 2}}
> {code}
> Modifying table schema and inserting new data:
> {code}
> 0: jdbc:hive2://localhost:10010> alter table acid_ptn_bucket1 add columns(c int);
> No rows affected (0.541 seconds)
> 0: jdbc:hive2://localhost:10010> insert into acid_ptn_bucket1 partition (ds) values(3,2,1000,'yesterday'),(3,3,1001,'today'),(3,4,1002,'yesterday'),(4,2,1003,'today'), (4,3,1004,'yesterday'),(4,4,1005,'today');
> 6 rows affected (3.699 seconds)
> {code}
> Data from ORC file (wrong partitioning):
> {code}
> /apps/hive/warehouse/acid_ptn_bucket1/ds=today/delta_0000003_0000003_0000/bucket_00000
> {"operation": 0, "originalTransaction": 3, "bucket": 536870912, "rowId": 0, "currentTransaction": 3, "row": {"a": 3, "b": 3, "c": 1001}}
> /apps/hive/warehouse/acid_ptn_bucket1/ds=today/delta_0000003_0000003_0000/bucket_00001
> {"operation": 0, "originalTransaction": 3, "bucket": 536936448, "rowId": 0, "currentTransaction": 3, "row": {"a": 4, "b": 4, "c": 1005}}
> {"operation": 0, "originalTransaction": 3, "bucket": 536936448, "rowId": 1, "currentTransaction": 3, "row": {"a": 4, "b": 2, "c": 1003}}
> {code}
> As seen above, the expected behaviour is that new data with column 'a' being 3 should go to bucket1 and column 'a' being 4 should go to bucket0, but the partitioning is wrong.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)