[jira] [Commented] (HADOOP-10599) Support prioritization of DN RPCs over client RPCs

["Daryn Sharp (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-10599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13996401#comment-13996401 ] 

Daryn Sharp commented on HADOOP-10599:
--------------------------------------

With the current global fair lock, I wonder if DNs could effectively starve clients.  The block updates may create a glut of write ops at the front of the call queue.  With no prioritization, clients create read bubbles in between the write ops.  

I think you've touched upon the angle we are using to attack the problem.  Now once I can finally get away from webhdfs hardening which derailed the efforts, we're working on fine grain FSN locking + distinct BM locking + BM has its own RPC service.  This should effectively achieve prioritization of DNs with much better performance characteristics overall.

> Support prioritization of DN RPCs over client RPCs
> --------------------------------------------------
>
>                 Key: HADOOP-10599
>                 URL: https://issues.apache.org/jira/browse/HADOOP-10599
>             Project: Hadoop Common
>          Issue Type: Sub-task
>            Reporter: Ming Ma
>
> We might need to prioritize DN RPC over client RPC so that no matter what application do to NN RPC and FSNamesystem's global lock, DN's requests will be processed timely. After a cluster is configured to have service RPC server separated from client RPC server, it is mitigated to some degree with fair FSNamesystem's global lock. Also if the NN global lock can be made more fine grained; such need becomes less important. Still, it will be good to evaluate if this is a good option.



--
This message was sent by Atlassian JIRA
(v6.2#6252)