[jira] [Reopened] (HADOOP-15703) ABFS - Implement client-side throttling

["Thomas Marquardt (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HADOOP-15703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Thomas Marquardt reopened HADOOP-15703:
---------------------------------------
      Assignee: Thomas Marquardt  (was: Sneha Varma)

I'll provide a patch to fix the Yetus issues.  I could not get Yetus to run previously, so lets see if it will run on the patch to fix this.

> ABFS - Implement client-side throttling 
> ----------------------------------------
>
>                 Key: HADOOP-15703
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15703
>             Project: Hadoop Common
>          Issue Type: Sub-task
>            Reporter: Sneha Varma
>            Assignee: Thomas Marquardt
>            Priority: Major
>         Attachments: HADOOP-15703-HADOOP-15407-001.patch, HADOOP-15703-HADOOP-15407-002.patch
>
>
> Big data workloads frequently exceed the AzureBlobFS max ingress and egress limits (https://docs.microsoft.com/en-us/azure/storage/common/storage-scalability-targets). For example, the max ingress limit for a GRS account in the United States is currently 10 Gbps. When the limit is exceeded, the AzureBlobFS service fails a percentage of incoming requests, and this causes the client to initiate the retry policy. The retry policy delays requests by sleeping, but the sleep duration is independent of the client throughput and account limit. This results in low throughput, due to the high number of failed requests and thrashing causes by the retry policy.
> To fix this, we introduce a client-side throttle which minimizes failed requests and maximizes throughput. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org