You are viewing a plain text version of this content. The canonical link for it is here.

Posted to common-issues@hadoop.apache.org by "Steve Loughran (Jira)" <ji...@apache.org> on 2024/03/21 22:22:00 UTC

[jira] [Updated] (HADOOP-19093) Improve rate limiting through ABFS in Manifest Committer

     [ https://issues.apache.org/jira/browse/HADOOP-19093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Steve Loughran updated HADOOP-19093:
------------------------------------
    Summary: Improve rate limiting through ABFS in Manifest Committer  (was: add load tests for abfs rename resilience)

> Improve rate limiting through ABFS in Manifest Committer
> --------------------------------------------------------
>
>                 Key: HADOOP-19093
>                 URL: https://issues.apache.org/jira/browse/HADOOP-19093
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/azure, test
>    Affects Versions: 3.3.6
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>            Priority: Major
>              Labels: pull-request-available
>
> I need a load test to verify that the rename resilience of the manifest committer actually works as intended
> * test suite with name ILoadTest* prefix (as with s3)
> * parallel test running with many threads doing many renames
> * verify that rename recovery should be detected
> * and that all renames MUST NOT fail.
> maybe also: metrics for this in fs and doc update. 
> Possibly; LogExactlyOnce to warn of load issues



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org