[jira] [Created] (MAPREDUCE-7474) [ABFS] Improve commit resilience and performance in Manifest Committer

["Steve Loughran (Jira)" <ji...@apache.org>]

Steve Loughran created MAPREDUCE-7474:
-----------------------------------------

             Summary: [ABFS] Improve commit resilience and performance in Manifest Committer
                 Key: MAPREDUCE-7474
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7474
             Project: Hadoop Map/Reduce
          Issue Type: Bug
          Components: client
    Affects Versions: 3.3.6, 3.4.0
            Reporter: Steve Loughran



* Manifest committer is not resilient to rename failures on task commit without HADOOP-18012 rename recovery enabled. 
* large burst of delete calls noted: are they needed


relates to HADOOP-19093 but takes a more minimal approach with goal of changes in manifest committer only.

Initial proposed changes
* retry recovery on task commit rename, always (repeat save, delete, rename)
* audit delete use and see if it can be pruned
* maybe: rate limit some IO internally, but not delegate to abfs



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org