After cluster rolling reboot, nodemanager could not authenticate to resource manager

[Manoj Samel <ma...@gmail.com>]

Environment is Hadoop 2.3.0, CDH 5.0,  RM and NN in HA, Kerberos Security

Rolling reboot of cluster was done. Services on each node was not stopped
before, the machines were just shut down, rebooted and services started on
each after reboot. Nodes were shut down in rolling manner such that one RM,
NN etc. were available at all times.

After restart, all services came up but node managers failed to connect to
Resource managers with "invalid Yarn Credentials". Kerberos keytabs etc.
are valid. The Resource managers services had to be restarted again for
node managers to connect successfully.

Thoughts?

Thanks,

Re: After cluster rolling reboot, nodemanager could not authenticate to resource manager

[daemeon reiydelle <da...@gmail.com>]

Check your ip addresses and host names of the RM (could be an issue around
which interface the nodes are now using?)



*.......*






*“Life should not be a journey to the grave with the intention of arriving
safely in apretty and well preserved body, but rather to skid in broadside
in a cloud of smoke,thoroughly used up, totally worn out, and loudly
proclaiming “Wow! What a Ride!” - Hunter ThompsonDaemeon C.M. ReiydelleUSA
(+1) 415.501.0198London (+44) (0) 20 8144 9872*

On Tue, Jan 27, 2015 at 6:06 PM, Manoj Samel <ma...@gmail.com>
wrote:

> Environment is Hadoop 2.3.0, CDH 5.0,  RM and NN in HA, Kerberos Security
>
> Rolling reboot of cluster was done. Services on each node was not stopped
> before, the machines were just shut down, rebooted and services started on
> each after reboot. Nodes were shut down in rolling manner such that one RM,
> NN etc. were available at all times.
>
> After restart, all services came up but node managers failed to connect to
> Resource managers with "invalid Yarn Credentials". Kerberos keytabs etc.
> are valid. The Resource managers services had to be restarted again for
> node managers to connect successfully.
>
> Thoughts?
>
> Thanks,
>
>
>
>

Re: After cluster rolling reboot, nodemanager could not authenticate to resource manager

[daemeon reiydelle <da...@gmail.com>]

Check your ip addresses and host names of the RM (could be an issue around
which interface the nodes are now using?)



*.......*






*“Life should not be a journey to the grave with the intention of arriving
safely in apretty and well preserved body, but rather to skid in broadside
in a cloud of smoke,thoroughly used up, totally worn out, and loudly
proclaiming “Wow! What a Ride!” - Hunter ThompsonDaemeon C.M. ReiydelleUSA
(+1) 415.501.0198London (+44) (0) 20 8144 9872*

On Tue, Jan 27, 2015 at 6:06 PM, Manoj Samel <ma...@gmail.com>
wrote:

> Environment is Hadoop 2.3.0, CDH 5.0,  RM and NN in HA, Kerberos Security
>
> Rolling reboot of cluster was done. Services on each node was not stopped
> before, the machines were just shut down, rebooted and services started on
> each after reboot. Nodes were shut down in rolling manner such that one RM,
> NN etc. were available at all times.
>
> After restart, all services came up but node managers failed to connect to
> Resource managers with "invalid Yarn Credentials". Kerberos keytabs etc.
> are valid. The Resource managers services had to be restarted again for
> node managers to connect successfully.
>
> Thoughts?
>
> Thanks,
>
>
>
>

Re: After cluster rolling reboot, nodemanager could not authenticate to resource manager

[daemeon reiydelle <da...@gmail.com>]

Check your ip addresses and host names of the RM (could be an issue around
which interface the nodes are now using?)



*.......*






*“Life should not be a journey to the grave with the intention of arriving
safely in apretty and well preserved body, but rather to skid in broadside
in a cloud of smoke,thoroughly used up, totally worn out, and loudly
proclaiming “Wow! What a Ride!” - Hunter ThompsonDaemeon C.M. ReiydelleUSA
(+1) 415.501.0198London (+44) (0) 20 8144 9872*

On Tue, Jan 27, 2015 at 6:06 PM, Manoj Samel <ma...@gmail.com>
wrote:

> Environment is Hadoop 2.3.0, CDH 5.0,  RM and NN in HA, Kerberos Security
>
> Rolling reboot of cluster was done. Services on each node was not stopped
> before, the machines were just shut down, rebooted and services started on
> each after reboot. Nodes were shut down in rolling manner such that one RM,
> NN etc. were available at all times.
>
> After restart, all services came up but node managers failed to connect to
> Resource managers with "invalid Yarn Credentials". Kerberos keytabs etc.
> are valid. The Resource managers services had to be restarted again for
> node managers to connect successfully.
>
> Thoughts?
>
> Thanks,
>
>
>
>

Re: After cluster rolling reboot, nodemanager could not authenticate to resource manager

[daemeon reiydelle <da...@gmail.com>]

Check your ip addresses and host names of the RM (could be an issue around
which interface the nodes are now using?)



*.......*






*“Life should not be a journey to the grave with the intention of arriving
safely in apretty and well preserved body, but rather to skid in broadside
in a cloud of smoke,thoroughly used up, totally worn out, and loudly
proclaiming “Wow! What a Ride!” - Hunter ThompsonDaemeon C.M. ReiydelleUSA
(+1) 415.501.0198London (+44) (0) 20 8144 9872*

On Tue, Jan 27, 2015 at 6:06 PM, Manoj Samel <ma...@gmail.com>
wrote:

> Environment is Hadoop 2.3.0, CDH 5.0,  RM and NN in HA, Kerberos Security
>
> Rolling reboot of cluster was done. Services on each node was not stopped
> before, the machines were just shut down, rebooted and services started on
> each after reboot. Nodes were shut down in rolling manner such that one RM,
> NN etc. were available at all times.
>
> After restart, all services came up but node managers failed to connect to
> Resource managers with "invalid Yarn Credentials". Kerberos keytabs etc.
> are valid. The Resource managers services had to be restarted again for
> node managers to connect successfully.
>
> Thoughts?
>
> Thanks,
>
>
>
>