[jira] [Work logged] (HADOOP-17833) Improve Magic Committer Performane

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HADOOP-17833?focusedWorklogId=636556&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-636556 ]

ASF GitHub Bot logged work on HADOOP-17833:
-------------------------------------------

                Author: ASF GitHub Bot
            Created on: 10/Aug/21 18:37
            Start Date: 10/Aug/21 18:37
    Worklog Time Spent: 10m 
      Work Description: steveloughran opened a new pull request #3289:
URL: https://github.com/apache/hadoop/pull/3289


   
   Speeding up the committer with key changes being
   
   * All writes under __magic trigger marker retention
     (no DELETEs after file/dir creation)
   * create(path, overwrite) skips all overwrite checks, including
     the LIST call intended to stop files being created over dirs
   * thread pool used for more parallelism in task commit.
   
   This is still WiP as it needs
   * cost tests to verify the optimisations are active
   * testing through spark
   
   Lots of changes in the tests because the committer has added
   a CommitContext class which manages the lifecycle of
   the thread pool and a set of thread local JSON serializers;
   this is now what is passed around in internal committer methods,
   so breaking tests calling in to them.
   
   It is a better design (one we should have done from the start);
   manifest committer is even better as all its operations "stages"
   are modular. Just means that a lot of tests stopped compiling.
   And as usual, mock tests played up.
   
   Finally, removed the injection/handling of inconsistent S3
   from the committer tests. Not needed, and simply complicating
   the code needlessly.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Issue Time Tracking
-------------------

            Worklog Id:     (was: 636556)
    Remaining Estimate: 0h
            Time Spent: 10m

> Improve Magic Committer Performane
> ----------------------------------
>
>                 Key: HADOOP-17833
>                 URL: https://issues.apache.org/jira/browse/HADOOP-17833
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs/s3
>    Affects Versions: 3.3.1
>            Reporter: Steve Loughran
>            Priority: Minor
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> Magic committer tasks can be slow because every file created with overwrite=false triggers a HEAD (verify there's no file) and a LIST (that there's no dir). And because of delayed manifestations, it may not behave as expected.
> ParquetOutputFormat is one example of a library which does this.
> we could fix parquet to use overwrite=true, but (a) there may be surprises in other uses (b) it'd still leave the list and (c) do nothing for other formats call
> Proposed: createFile() under a magic path to skip all probes for file/dir at end of path
> Only a single task attempt Will be writing to that directory and it should know what it is doing. If there is conflicting file names and parts across tasks that won't even get picked up at this point. Oh and none of the committers ever check for this: you'll get the last file manifested (s3a) or renamed (file)
> If we skip the checks we will save 2 HTTP requests/file.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org