[jira] [Commented] (BEAM-3638) S3 occasionally fails to delete

["Jacob Marble (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BEAM-3638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16355751#comment-16355751 ] 

Jacob Marble commented on BEAM-3638:
------------------------------------

I believe this is a mistake. The "subsequent errors" are emitted by Filesystems.copy(). Closing this for now, will re-open if I find something concrete.

> S3 occasionally fails to delete
> -------------------------------
>
>                 Key: BEAM-3638
>                 URL: https://issues.apache.org/jira/browse/BEAM-3638
>             Project: Beam
>          Issue Type: Bug
>          Components: sdk-java-extensions
>    Affects Versions: 2.3.0
>            Reporter: Jacob Marble
>            Assignee: Jacob Marble
>            Priority: Critical
>             Fix For: 2.3.0
>
>
> When AmazonS3Client.deleteObjects() is called, the S3 service occasionally returns a retryable error, and also deletes the object. Beam retries the request, and the S3 service correctly returns a "404 Not Found" error. The job fails needlessly.
> I propose to fix this by logging and allowing "404 Not Found" errors; if the object doesn't exist then the current state is also the desired state. Should have a PR later today.
> The first error looks like this:
> {{java.io.IOException: com.amazonaws.services.s3.model.MultiObjectDeleteException: One or more objects could not be deleted (Service: null; Status Code: 200; Error Code: null; Request ID: [redacted]; S3 Extended Request ID: [redacted]), S3 Extended Request ID: [redacted]}}
> Subsequent errors:
> {{java.io.IOException: com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: [redacted]; S3 Extended Request ID: [redacted]), S3 Extended Request ID: [redacted]}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)