[jira] [Assigned] (YARN-11997) Serialize concurrent Docker image pulls by image name

["Yuan Luo (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/YARN-11997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Yuan Luo reassigned YARN-11997:
-------------------------------

    Assignee: Yuan Luo

> Serialize concurrent Docker image pulls by image name
> -----------------------------------------------------
>
>                 Key: YARN-11997
>                 URL: https://issues.apache.org/jira/browse/YARN-11997
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: nodemanager
>            Reporter: Yuan Luo
>            Assignee: Yuan Luo
>            Priority: Major
>              Labels: pull-request-available
>
> h3. Problem
> When multiple containers request the same Docker image at the same time, each NodeManager launch thread can invoke a separate {{docker pull}} command for that image. These duplicate pulls contend on the same image and perform redundant work.
> Pulls for different images do not need to block one another.
> h3. Proposed fix
> Serialize concurrent Docker pulls by image name. Use a per-image lock with reference counting so that:
> * pulls for the same image execute one at a time;
> * pulls for different images can execute concurrently;
> * unused locks are removed after the last waiter finishes; 
> * locks are released when a pull fails.
> h3. Tests
> Add focused coverage for:
> * concurrent pulls for the same image are serialized;
> * concurrent pulls for different images run in parallel; 
> * a failed pull releases its image lock.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org