[jira] [Created] (HADOOP-16632) Partitioned S3A magic committers can leaving pending files, maybe upload data

["Steve Loughran (Jira)" <ji...@apache.org>]

Steve Loughran created HADOOP-16632:
---------------------------------------

             Summary: Partitioned S3A magic committers can leaving pending files, maybe upload data
                 Key: HADOOP-16632
                 URL: https://issues.apache.org/jira/browse/HADOOP-16632
             Project: Hadoop Common
          Issue Type: Sub-task
          Components: fs/s3
    Affects Versions: 3.1.3, 3.2.1
            Reporter: Steve Loughran
            Assignee: Steve Loughran


Partitioned S3A magic committers can leaving pending files, maybe upload data

This surfaced in an assertion failure on a parallel test run.

I thought it was actually a test failure, but with HADOOP-16207 all the docs are preserved in the local FS and I can understand what happened.

h3. Junit process
{code}
[INFO] 
[ERROR] Failures: 
[ERROR] ITestS3ACommitterMRJob.test_200_execute:344->customPostExecutionValidation:356 Expected a java.io.FileNotFoundException to be thrown, but got the result: : "Found magic dir which should have been deleted at S3AFileStatus{path=s3a://hwdev-steve-ireland-new/fork-0001/test/ITestS3ACommitterMRJob-execute-magic/__magic; isDirectory=true; modification_time=0; access_time=0; owner=stevel; group=stevel; permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=true; isErasureCoded=false} isEmptyDirectory=UNKNOWN eTag=null versionId=null
[s3a://hwdev-steve-ireland-new/fork-0001/test/ITestS3ACommitterMRJob-execute-magic/__magic/app-attempt-0001/tasks/attempt_1570197469968_0003_m_000008_1/__base/part-m-00008
s3a://hwdev-steve-ireland-new/fork-0001/test/ITestS3ACommitterMRJob-execute-magic/__magic/app-attempt-0001/tasks/attempt_1570197469968_0003_m_000008_1/__base/part-m-00008.pending
{code}

Full details to follow in the comment as they are, well, detailed.

 

Key point: AM-side job and task cleanup can happen before the worker task finishes its writes. This will result in files under __magic. It may result in pending uploads too -but only if the write began after the AM job cleanup did a list + abort of all pending uploads under the destination directory



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org