[jira] [Commented] (HADOOP-18657) Tune ABFS create() retry logic

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-18657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17722087#comment-17722087 ] 

ASF GitHub Bot commented on HADOOP-18657:
-----------------------------------------

steveloughran commented on PR #5462:
URL: https://github.com/apache/hadoop/pull/5462#issuecomment-1545445339

   reviewing this; too many other things have got in my way.
   
   I agree, with create overwrite=false, we must fail with a concurrency error
   
   what we don't want to do is overreact if we are doing overwrite=true and something does happen partway.
   
   I'll look at this in more detail, maybe focus purely on being meaningful on errors, in particular making sure that if the file is deleted before the error is raised, keep raising that concurrency error.




> Tune ABFS create() retry logic
> ------------------------------
>
>                 Key: HADOOP-18657
>                 URL: https://issues.apache.org/jira/browse/HADOOP-18657
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs/azure
>    Affects Versions: 3.3.5
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>            Priority: Major
>              Labels: pull-request-available
>
> Based on experience trying to debug this happening
> # add debug statements when create() fails
> # generated exception text to reference string shared with tests, path and error code
> # generated exception to include inner exception for full stack trace
> Currently the retry logic is
> # create(overwrite=false)
> # if HTTP_CONFLICT/409 raised; call HEAD
> # use etag in create(path, overwrite=true, etag)
> # special handling of error HTTP_PRECON_FAILED = 412
> There's a race condition here, which is if between 1 and 2 the file which exists is deleted. The retry should succeed, but currently a 404 from the head is escalated to a failure
> proposed changes
> # if HEAD is 404, leave etag == null and continue
> # special handling of 412 also to handle 409



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org