[jira] [Work logged] (HDFS-16283) RBF: improve renewLease() to call only a specific NameNode rather than make fan-out calls

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HDFS-16283?focusedWorklogId=786994&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-786994 ]

ASF GitHub Bot logged work on HDFS-16283:
-----------------------------------------

                Author: ASF GitHub Bot
            Created on: 01/Jul/22 08:44
            Start Date: 01/Jul/22 08:44
    Worklog Time Spent: 10m 
      Work Description: ZanderXu opened a new pull request, #4524:
URL: https://github.com/apache/hadoop/pull/4524

   ### Description of PR
   [HDFS-16283](https://issues.apache.org/jira/browse/HDFS-16283): RBF: improve renewLease() to call only a specific NameNode rather than make fan-out calls
   
   Currently RBF will forward the renewLease() rpc to all the available name services. So the forwarding efficiency will be affected by the unhealthy downstream name services. And along with as more as NSs are monitored by RBF, this problem will become more and more serious. 
   
   In our prod cluster, there are 70+ nameservices, the phenomenon that renewLease() rpc is blocked often occurs. 
   
   This patch is be used to fix this problem and work well on our cluster, and the main ideas is:
   - Carrying nsId to client when creating a new file
   - Store this nsId in DFSOutputStream
   - Client renewLease() rpc carries all nsIds of all DFSOutputStream to RBF
   - RBF parses the nsIds and forwards the renewLease rpc to the corresponding name services
   




Issue Time Tracking
-------------------

    Worklog Id:     (was: 786994)
    Time Spent: 2h 20m  (was: 2h 10m)

> RBF: improve renewLease() to call only a specific NameNode rather than make fan-out calls
> -----------------------------------------------------------------------------------------
>
>                 Key: HDFS-16283
>                 URL: https://issues.apache.org/jira/browse/HDFS-16283
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: rbf
>            Reporter: Aihua Xu
>            Assignee: Aihua Xu
>            Priority: Major
>              Labels: pull-request-available
>         Attachments: RBF_ improve renewLease() to call only a specific NameNode rather than make fan-out calls.pdf
>
>          Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> Currently renewLease() against a router will make fan-out to all the NameNodes. Since renewLease() call is so frequent and if one of the NameNodes are slow, then eventually the router queues are blocked by all renewLease() and cause router degradation. 
> We will make a change in the client side to keep track of NameNode Id in additional to current fileId so routers understand which NameNodes the client is renewing lease against.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org