You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Eugene Koifman (JIRA)" <ji...@apache.org> on 2016/12/01 00:23:59 UTC
[jira] [Comment Edited] (HIVE-15202) Concurrent compactions for the
same partition may generate malformed folder structure
[ https://issues.apache.org/jira/browse/HIVE-15202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15709447#comment-15709447 ]
Eugene Koifman edited comment on HIVE-15202 at 12/1/16 12:23 AM:
-----------------------------------------------------------------
It's not. The cleaner can only remove files that are obsolete, i.e. there is another (better/wider) delta/base that includes the same data. No reader (including compaction) reads obsolete data.
AcidUtils.getAcidState() encapsulates this
was (Author: ekoifman):
It's not. The cleaner can only remove files that are obsolete, i.e. there is another (better/wider) delta/base that includes the same data. No reader (including compaction) reads obsolete data
> Concurrent compactions for the same partition may generate malformed folder structure
> -------------------------------------------------------------------------------------
>
> Key: HIVE-15202
> URL: https://issues.apache.org/jira/browse/HIVE-15202
> Project: Hive
> Issue Type: Bug
> Components: Transactions
> Reporter: Rui Li
> Assignee: Eugene Koifman
> Attachments: HIVE-15202.01.patch, HIVE-15202.02.patch, HIVE-15202.03.patch
>
>
> If two compactions run concurrently on a single partition, it may generate folder structure like this: (nested base dir)
> {noformat}
> drwxr-xr-x - root supergroup 0 2016-11-14 22:23 /user/hive/warehouse/test/z=1/base_0000007/base_0000007
> -rw-r--r-- 3 root supergroup 201 2016-11-14 21:46 /user/hive/warehouse/test/z=1/base_0000007/bucket_00000
> -rw-r--r-- 3 root supergroup 611 2016-11-14 21:46 /user/hive/warehouse/test/z=1/base_0000007/bucket_00001
> -rw-r--r-- 3 root supergroup 614 2016-11-14 21:46 /user/hive/warehouse/test/z=1/base_0000007/bucket_00002
> -rw-r--r-- 3 root supergroup 621 2016-11-14 21:46 /user/hive/warehouse/test/z=1/base_0000007/bucket_00003
> -rw-r--r-- 3 root supergroup 621 2016-11-14 21:46 /user/hive/warehouse/test/z=1/base_0000007/bucket_00004
> -rw-r--r-- 3 root supergroup 201 2016-11-14 21:46 /user/hive/warehouse/test/z=1/base_0000007/bucket_00005
> -rw-r--r-- 3 root supergroup 201 2016-11-14 21:46 /user/hive/warehouse/test/z=1/base_0000007/bucket_00006
> -rw-r--r-- 3 root supergroup 201 2016-11-14 21:46 /user/hive/warehouse/test/z=1/base_0000007/bucket_00007
> -rw-r--r-- 3 root supergroup 201 2016-11-14 21:46 /user/hive/warehouse/test/z=1/base_0000007/bucket_00008
> -rw-r--r-- 3 root supergroup 201 2016-11-14 21:46 /user/hive/warehouse/test/z=1/base_0000007/bucket_00009
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)