[jira] [Created] (HBASE-12626) Archive cleaner cannot keep up; it maxes out at about 400k deletes/hour

["stack (JIRA)" <ji...@apache.org>]

stack created HBASE-12626:
-----------------------------

             Summary: Archive cleaner cannot keep up; it maxes out at about 400k deletes/hour
                 Key: HBASE-12626
                 URL: https://issues.apache.org/jira/browse/HBASE-12626
             Project: HBase
          Issue Type: Improvement
          Components: scaling, master
    Affects Versions: 0.94.25
            Reporter: stack
            Assignee: stack
            Priority: Critical


On big clusters, it is possible to overrun the archive cleaning thread.  Make it able to do more work per cycle when needed.

We saw this on a user's cluster. The rate at which files were being moved to the archive exceeded our delete rate such that the archive had tens of millions of files putting a friction on all cluster ops.

The cluster had ~500 nodes.  It that was RAM constrained (other processes on box also need RAM). Over a period of days, the loading was thrown off kilter because it started taking double writes going from one schema to another (Cluster was running hot before the double loading).  The master was deleting an archived file every 9ms on average, about 400k deletes an hour.  The constrained RAM and their having 4-5 column famiilies had them creating files in excess of this rate so we backed up.

For some helpful background/input, see the dev thread http://search-hadoop.com/m/DHED4UYSF9



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)