[jira] [Updated] (HADOOP-10597) Evaluate if we can have RPC client back off when server is under heavy load

["Ming Ma (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HADOOP-10597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ming Ma updated HADOOP-10597:
-----------------------------

    Attachment: HADOOP-10597.patch

Steve, thanks for the suggestions. Yes, it is useful to have server provide backoff parameters to client.

Here is the initial patch to cover the basic idea. After we believe this is the right direction, unit test and load tests will follow.

1. Enhance RetriableException for this purpose. When RPC queue is full, RetriableException will be thrown back to the client with the backoff parameters in PB. The backoff parameters are based on exponential back off.

2. Client/RetryPolicy can decide if and how to use the server hint. It is up to each retry policy implementation. For NN HA RetryPolicy failoverOnNetworkException, it will use the server hint when it is available. For NN non-HA scenario, HDFS-6478 needs to be fixed first. A new policy called RetryUpToMaximumCountWithFixedSleepAndServerHint is provided as an example.

3. Backoff feature is turned off by default in RPC server.

> Evaluate if we can have RPC client back off when server is under heavy load
> ---------------------------------------------------------------------------
>
>                 Key: HADOOP-10597
>                 URL: https://issues.apache.org/jira/browse/HADOOP-10597
>             Project: Hadoop Common
>          Issue Type: Sub-task
>            Reporter: Ming Ma
>         Attachments: HADOOP-10597.patch
>
>
> Currently if an application hits NN too hard, RPC requests be in blocking state, assuming OS connection doesn't run out. Alternatively RPC or NN can throw some well defined exception back to the client based on certain policies when it is under heavy load; client will understand such exception and do exponential back off, as another implementation of RetryInvocationHandler.



--
This message was sent by Atlassian JIRA
(v6.2#6252)