You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Alvaro Brandon <al...@gmail.com> on 2017/12/05 15:50:15 UTC
Avoiding using hostname for YARN nodemanagers
Hello:
I'm using Docker images to build a YARN cluster. I have a problem when the
node managers register with the resource manager.
Since they are containers they use the hash that the Docker engine assigns
to them as the hostname.
*17/12/05 14:56:16 INFO nodemanager.NodeStatusUpdaterImpl: Registered with
ResourceManager as ba3aeecd656a:45989 with total resource of <memory:4096,
vCores:1>*
This, of course, is a problem when the resourcemanager tries to contact the
node.
*17/12/05 15:42:04 ERROR scheduler.SchedulerApplicationAttempt: Error
trying to assign container token and NM token to an allocated container
container_1512485507238_0003_01_000001*
*java.lang.IllegalArgumentException: java.net.UnknownHostException:
ba3aeecd656a*
With HDFS I had no problems since you can always set the
*dfs.datanode.use.datanode.hostname *and similar configuration options to
avoid the problem. However, I cannot find a similar option in YARN node
managers.
Is there an option to not use the hostname when registering with the
resource manager?
Re: Avoiding using hostname for YARN nodemanagers
Posted by Susheel Kumar Gadalay <sk...@gmail.com>.
Check properties yarn.nodemanager.hostname,
yarn.resourcemanager.hostname under yarn-site.xml.
On 12/5/17, Alvaro Brandon <al...@gmail.com> wrote:
> Thanks for your answer Vinay:
>
> The thing is that I'm using Marathon and not the Docker engine per se. I
> don't want to set a -h parameter to each instance that is launched, since
> this is the responsibility of the container orchestrator platform. That's
> why I need an option like the HDFS one.
>
> Alvaro
>
> 2017-12-05 17:03 GMT+01:00 Vinayakumar B <vi...@apache.org>:
>
>> Hi Alvaro,
>>
>> I think you can configure to use custom hostname for docker containers as
>> well.
>> Hostname should be provided durin launch of containers using -h
>> parameter.
>>
>> And with user created docker network DNS resolution of these hostnames
>> among the containers is possible. provide --network-alias parameter to
>> add
>> hostname for DNS
>>
>> Check if that works for you.
>> -Vinay
>>
>>
>> On 5 Dec 2017 9:20 pm, "Alvaro Brandon" <al...@gmail.com> wrote:
>>
>> Hello:
>>
>> I'm using Docker images to build a YARN cluster. I have a problem when
>> the
>> node managers register with the resource manager.
>>
>> Since they are containers they use the hash that the Docker engine
>> assigns
>> to them as the hostname.
>>
>> *17/12/05 14:56:16 INFO nodemanager.NodeStatusUpdaterImpl: Registered
>> with
>> ResourceManager as ba3aeecd656a:45989 with total resource of
>> <memory:4096,
>> vCores:1>*
>>
>> This, of course, is a problem when the resourcemanager tries to contact
>> the node.
>>
>> *17/12/05 15:42:04 ERROR scheduler.SchedulerApplicationAttempt: Error
>> trying to assign container token and NM token to an allocated container
>> container_1512485507238_0003_01_000001*
>> *java.lang.IllegalArgumentException: java.net.UnknownHostException:
>> ba3aeecd656a*
>>
>> With HDFS I had no problems since you can always set the
>> *dfs.datanode.use.datanode.hostname *and similar configuration options to
>> avoid the problem. However, I cannot find a similar option in YARN node
>> managers.
>>
>> Is there an option to not use the hostname when registering with the
>> resource manager?
>>
>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
For additional commands, e-mail: user-help@hadoop.apache.org
Re: Avoiding using hostname for YARN nodemanagers
Posted by Alvaro Brandon <al...@gmail.com>.
Thanks for your answer Vinay:
The thing is that I'm using Marathon and not the Docker engine per se. I
don't want to set a -h parameter to each instance that is launched, since
this is the responsibility of the container orchestrator platform. That's
why I need an option like the HDFS one.
Alvaro
2017-12-05 17:03 GMT+01:00 Vinayakumar B <vi...@apache.org>:
> Hi Alvaro,
>
> I think you can configure to use custom hostname for docker containers as
> well.
> Hostname should be provided durin launch of containers using -h parameter.
>
> And with user created docker network DNS resolution of these hostnames
> among the containers is possible. provide --network-alias parameter to add
> hostname for DNS
>
> Check if that works for you.
> -Vinay
>
>
> On 5 Dec 2017 9:20 pm, "Alvaro Brandon" <al...@gmail.com> wrote:
>
> Hello:
>
> I'm using Docker images to build a YARN cluster. I have a problem when the
> node managers register with the resource manager.
>
> Since they are containers they use the hash that the Docker engine assigns
> to them as the hostname.
>
> *17/12/05 14:56:16 INFO nodemanager.NodeStatusUpdaterImpl: Registered with
> ResourceManager as ba3aeecd656a:45989 with total resource of <memory:4096,
> vCores:1>*
>
> This, of course, is a problem when the resourcemanager tries to contact
> the node.
>
> *17/12/05 15:42:04 ERROR scheduler.SchedulerApplicationAttempt: Error
> trying to assign container token and NM token to an allocated container
> container_1512485507238_0003_01_000001*
> *java.lang.IllegalArgumentException: java.net.UnknownHostException:
> ba3aeecd656a*
>
> With HDFS I had no problems since you can always set the
> *dfs.datanode.use.datanode.hostname *and similar configuration options to
> avoid the problem. However, I cannot find a similar option in YARN node
> managers.
>
> Is there an option to not use the hostname when registering with the
> resource manager?
>
>
>
Re: Avoiding using hostname for YARN nodemanagers
Posted by Vinayakumar B <vi...@apache.org>.
Hi Alvaro,
I think you can configure to use custom hostname for docker containers as
well.
Hostname should be provided durin launch of containers using -h parameter.
And with user created docker network DNS resolution of these hostnames
among the containers is possible. provide --network-alias parameter to add
hostname for DNS
Check if that works for you.
-Vinay
On 5 Dec 2017 9:20 pm, "Alvaro Brandon" <al...@gmail.com> wrote:
Hello:
I'm using Docker images to build a YARN cluster. I have a problem when the
node managers register with the resource manager.
Since they are containers they use the hash that the Docker engine assigns
to them as the hostname.
*17/12/05 14:56:16 INFO nodemanager.NodeStatusUpdaterImpl: Registered with
ResourceManager as ba3aeecd656a:45989 with total resource of <memory:4096,
vCores:1>*
This, of course, is a problem when the resourcemanager tries to contact the
node.
*17/12/05 15:42:04 ERROR scheduler.SchedulerApplicationAttempt: Error
trying to assign container token and NM token to an allocated container
container_1512485507238_0003_01_000001*
*java.lang.IllegalArgumentException: java.net.UnknownHostException:
ba3aeecd656a*
With HDFS I had no problems since you can always set the
*dfs.datanode.use.datanode.hostname *and similar configuration options to
avoid the problem. However, I cannot find a similar option in YARN node
managers.
Is there an option to not use the hostname when registering with the
resource manager?