[jira] Resolved: (HBASE-2181) remove or provide config to completely disable all aspects of 'table fragmentation'

["Lars George (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HBASE-2181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Lars George resolved HBASE-2181.
--------------------------------

    Resolution: Duplicate

Duplicates HBASE-2165

> remove or provide config to completely disable all aspects of 'table fragmentation' 
> ------------------------------------------------------------------------------------
>
>                 Key: HBASE-2181
>                 URL: https://issues.apache.org/jira/browse/HBASE-2181
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: ryan rawson
>             Fix For: 0.21.0
>
>
> Given the potentially low and misleading value of the metric, and how much effort must be expended to collect them, I would argue at least we should allow users to disable the feature completely.
> The first problem is the data the metric delivers is not very useful. On any given busy system, this value is often 100%. On a sample system here, 12% of the  tables were at either 0 or 100%.  Furthermore the 100% metric is not particularly informative. If a table has 100% 'fragmentation' it does not necessarily imply that this table is in dire need of compaction.  The HBase compaction code will generally keep at least 2 store files around - it refuses to minor compact older and larger files, preferring to merge small files.  Thus on a table taking writes on all regions, the expected value of fragmentation is in fact 100%. And this is not a bad thing either.  Considering that compacting a 500GB table will take an hour and hammer a cluster, misleading users into striving to get to 0% is non ideal.
> The other major problem of this feature is collecting the data is non-trivial on larger clusters.  I did a test where I did a lsr on a hadoop cluster, and to generate 15k lines of output, it pegged the namenode at over 100% cpu for a few seconds. On a cluster with 7000 regions, we can clearly easily have 14,000 (2 store files per region is typical) files thus causing spikes against the namenode to generate this statistic.
> I would propose 3 courses of actions:
> - allow complete disablement of the feature, including the background thread and the UI display
> - change the metric to mean '# of regions with > 5 store files'
> - replacing the metric with a completely different one that attempts to capture the spirit of the intent but with less load.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.