You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Eugene Koifman (JIRA)" <ji...@apache.org> on 2018/03/09 21:52:00 UTC
[jira] [Updated] (HIVE-17206) make a version of Compactor specific
to unbucketed tables
[ https://issues.apache.org/jira/browse/HIVE-17206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eugene Koifman updated HIVE-17206:
----------------------------------
Description:
current Compactor will work but is not optimized/flexible enough
The current compactor is designed to generate the number of splits equal to the number of buckets in the table. That is the degree of parallelism.
For unbucketed tables, the same is used but the "number of buckets" is derived from the files found in the deltas. For small writes, there will likely be just 1 bucket_00000 file. For large writes, the parallelism of the write determines the number of output files.
Need to make sure Compactor can control parallelism for unbucketed tables as it wishes. For example, hash partition all records (by ROW__ID?) into N disjoint sets.
was:current Compactor will work but is not optimized/flexible enough
> make a version of Compactor specific to unbucketed tables
> ---------------------------------------------------------
>
> Key: HIVE-17206
> URL: https://issues.apache.org/jira/browse/HIVE-17206
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
> Reporter: Eugene Koifman
> Assignee: Eugene Koifman
> Priority: Major
>
> current Compactor will work but is not optimized/flexible enough
> The current compactor is designed to generate the number of splits equal to the number of buckets in the table. That is the degree of parallelism.
> For unbucketed tables, the same is used but the "number of buckets" is derived from the files found in the deltas. For small writes, there will likely be just 1 bucket_00000 file. For large writes, the parallelism of the write determines the number of output files.
> Need to make sure Compactor can control parallelism for unbucketed tables as it wishes. For example, hash partition all records (by ROW__ID?) into N disjoint sets.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)