[jira] [Commented] (HBASE-6772) Make the Distributed Split HDFS Location aware

["Nicolas Liochon (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-6772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13731790#comment-13731790 ] 

Nicolas Liochon commented on HBASE-6772:
----------------------------------------

I'm going to try this. Something I had not though about is that it's simpler is that as the master adds a field in ZK, the regionserver must be able to read it. If it cannot make it into the 0.95.x, I will make it backward compatible.
                
> Make the Distributed Split HDFS Location aware
> ----------------------------------------------
>
>                 Key: HBASE-6772
>                 URL: https://issues.apache.org/jira/browse/HBASE-6772
>             Project: HBase
>          Issue Type: Improvement
>          Components: master, regionserver
>    Affects Versions: 0.95.2
>            Reporter: Nicolas Liochon
>            Assignee: Nicolas Liochon
>
> During a hlog split, each log file (a single hdfs block) is allocated to a different region server. This region server reads the file and creates the recovery edit files.
> The allocation to the region server is random. We could take into account the locations of the log file to split:
> - the reads would be local, hence faster. This allows short circuit as well.
> - less network i/o used during a failure (and this is important)
> - we would be sure to read from a working datanode, hence we're sure we won't have read errors. Read errors slow the split process a lot, as we often enter the "timeouted world". 
> We need to limit the calls to the namenode however.
> Typical algo could be:
> - the master gets the locations of the hlog files
> - it writes it into ZK, if possible in one transaction (this way all the tasks are visible alltogether, allowing some arbitrage by the region server).
> - when the regionserver receives the event, it checks for all logs and all locations.
> - if there is a match, it takes it
> - if not it waits something like 0.2s (to give the time to other regionserver to take it if the location matches), and take any remaining task.
> Drawbacks are:
> - a 0.2s delay added if there is no regionserver available on one of the locations. It's likely possible to remove it with some extra synchronization.
> - Small increase in complexity and dependency to HDFS
> Considering the advantages, it's worth it imho.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira