[jira] [Commented] (YARN-9229) Document docker registry deployment with NFS Gateway

["Eric Yang (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-9229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16752563#comment-16752563 ] 

Eric Yang commented on YARN-9229:
---------------------------------

Environment: 5 Nodes VM cluster.
Model name: Intel Xeon E312xx (Sandy Bridge)
Cpu MHz: 2399.996
Network: 1GB Ethernet
Software: CentOS 7.3, Hadoop 3.3.0-SNAPSHOT and ZooKeeper 3.4.5, Kerberos Enabled cluster.
Standard Hadoop configuration with NFS configuration:
{code}
    <property>
      <name>nfs.exports.allowed.hosts</name>
      <value>* rw</value>
    </property>

    <property>
      <name>nfs.file.dump.dir</name>
      <value>/tmp/.hdfs-nfs</value>
    </property>

    <property>
      <name>nfs.kerberos.principal</name>
      <value>nfs/_HOST@EXAMPLE.COM</value>
    </property>

    <property>
      <name>nfs.keytab.file</name>
      <value>/etc/security/keytabs/nfs.service.keytab</value>
    </property>
{code}

Run NFS Gateway on all datanodes as hdfs user using: {code}$ $HADOOP_HOME/bin/hdfs nfs3{code}

On each datanode, nfs mount point is exposed to /hdfs, using:
{code}# mount -t nfs -o vers=3,proto=tcp,nolock,noacl,sync $DN_IP:/ /hdfs{code}

Docker Registry is started using YARN service:

registry.json
{code}{
  "name": "docker-registry",
  "version": "1.0",
  "kerberos_principal" : {
    "principal_name" : "hbase/_HOST@EXAMPLE.COM",
    "keytab" : "file:///etc/security/keytabs/hbase.service.keytab"
  },
  "components" :
  [
    {
      "name": "registry",
      "number_of_containers": 1,
      "artifact": {
        "id": "registry:latest",
        "type": "DOCKER"
      },
      "resource": {
        "cpus": 1,
        "memory": "256"
      },
      "configuration": {
        "env": {
          "YARN_CONTAINER_RUNTIME_DOCKER_RUN_OVERRIDE_DISABLE":"true",
          "YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS":"/hdfs/apps/docker/registry:/var/lib/registry"
        },
        "properties": {
          "docker.network": "host"
        }
      }
    }
  ]
}
{code}

YARN service configures docker mounts from /hdfs/apps/docker/registry to /var/lib/registry inside docker container.

{code}yarn app -launch docker-reg /tmp/registry.json{code}


> Document docker registry deployment with NFS Gateway
> ----------------------------------------------------
>
>                 Key: YARN-9229
>                 URL: https://issues.apache.org/jira/browse/YARN-9229
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Eric Yang
>            Assignee: Eric Yang
>            Priority: Major
>
> The goal of this task is to demonstrate running docker pull of image from HDFS via NFS gateway. Document the pros and cons for using NFS gateway.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org