You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "Andrew Purtell (JIRA)" <ji...@apache.org> on 2015/04/11 02:55:13 UTC
[jira] [Resolved] (HBASE-3404) Compaction Ordering for Bulk Import
Files
[ https://issues.apache.org/jira/browse/HBASE-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell resolved HBASE-3404.
-----------------------------------
Resolution: Fixed
> Compaction Ordering for Bulk Import Files
> -----------------------------------------
>
> Key: HBASE-3404
> URL: https://issues.apache.org/jira/browse/HBASE-3404
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.0, 0.90.1, 0.92.0
> Reporter: Nicolas Spiegelberg
> Assignee: Nicolas Spiegelberg
>
> We got into an issue today where we were using HFileOutputFormat to perform an incremental load on an already-large cluster. Because bulk-loaded files don't have a sequence ID, they are put in the front of the StoreFile list. This resulted in the following StoreFile ordering
> 2GB (bulk) => 25GB => 2GB => ...
> So this triggered a 30+GB major compaction for every single region. Optimally, we would like bulk import files to be ordered in the compaction list at the time of insertion so this can be a much smaller compaction and rely on StoreFile age for major compaction trigger.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)