You are viewing a plain text version of this content. The canonical link for it is here.

Posted to hdfs-dev@hadoop.apache.org by "Liang Xie (JIRA)" <ji...@apache.org> on 2014/01/08 05:07:53 UTC

[jira] [Created] (HDFS-5727) introduce a self-maintain io queue handling mechanism

Liang Xie created HDFS-5727:
-------------------------------

             Summary: introduce a self-maintain io queue handling mechanism
                 Key: HDFS-5727
                 URL: https://issues.apache.org/jira/browse/HDFS-5727
             Project: Hadoop HDFS
          Issue Type: New Feature
          Components: datanode
    Affects Versions: 3.0.0
            Reporter: Liang Xie
            Assignee: Liang Xie


Currently the datanode read/write SLA is dfficult to be ganranteed for HBase online requirement. One of major reasons is we don't support io priority or io reqeust reorder inside datanode.
I proposal introducing a self-maintain io queue mechanism to handle io request priority. Image there're lots of concurrent read/write reqeust from HBase side, and a background datanode block scanner is running(default is every 21 days, IIRC) just in time, then the HBase read/write 99% or 99.9% percentile latency would be vulnerable despite we have a bg thread throttling...
the reorder stuf i have not thought clearly enough, but definitely the reorder in the queue in the app side would beat the currently relying OS's io queue merge.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)