[jira] [Commented] (YARN-8675) Setting hostname of docker container breaks with "host" networking mode for Apps which do not run as a YARN service

["Eric Yang (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-8675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16587873#comment-16587873 ] 

Eric Yang commented on YARN-8675:
---------------------------------

Docker only prevents --hostname and --net=host flag to be combined in older version of docker (1.13) . See [Docker issue|https://github.com/moby/moby/pull/29144].  The following table illustrate the possible combination when we should and should not support customized hostname:

| YARN Registry DNS | YARN Service | Custom AM | Network Type | Custom Hostname |
| Enabled | Yes | No | Host | Yes |
| Enabled | No | Yes | Host | No |
| Disabled | No | Yes | Host | No |
| Disabled | No | Yes | Bridge | N/A |

Today registryDNS and YARN service are coupled together, only YARN service knows how to populate hostname information to registryDNS.  If custom AM creates it's own logic to generate custom hostname, it must have some way to populate RegistryDNS to translate correct name.  Without using YARN service, there is no programmable API to customize hostname.  This is the reason that Spark on YARN cluster on docker mode fails with buzzard hostname composition.

For resolving spark issue, it is entirely possible to run spark using YARN service API without making any code changes to spark standalone mode.  To fix this issue properly, it would be best to provide hint to docker runtime to decide if custom hostname can be supported.  Custom hostname is a new concept that doesn't exist prior to docker container.  Therefore, only new application using YARN service should be supported in my view.

> Setting hostname of docker container breaks with "host" networking mode for Apps which do not run as a YARN service
> -------------------------------------------------------------------------------------------------------------------
>
>                 Key: YARN-8675
>                 URL: https://issues.apache.org/jira/browse/YARN-8675
>             Project: Hadoop YARN
>          Issue Type: Bug
>            Reporter: Yesha Vora
>            Assignee: Suma Shivaprasad
>            Priority: Major
>
> Applications like the Spark AM currently do not run as a YARN service and setting hostname breaks driver/executor communication if docker version >=1.13.1 , especially with wire-encryption turned on.
> YARN-8027 sets the hostname if YARN DNS is enabled. But the cluster could have a mix of YARN service/native Applications.
> The proposal is to not set the hostname when "host" networking mode is enabled.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org