You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@ambari.apache.org by Andrew Onischuk <ao...@hortonworks.com> on 2016/02/26 18:07:28 UTC

Review Request 44086: Registering 700 hosts gets stuck without timeout on 8 hosts

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44086/
-----------------------------------------------------------

Review request for Ambari and Myroslav Papirkovskyy.


Bugs: AMBARI-15205
    https://issues.apache.org/jira/browse/AMBARI-15205


Repository: ambari


Description
-------

When adding 700 hosts through the UI's add-host-wizard, all hosts successfully
registered except for 8 of them. These 8 hosts were stuck in RUNNING state and
would not timeout even after 20 minutes.

I had to kill the bootstrap process on ambari-server to make the operation to
complete.  
On my second attempt I was able to recreate the problem

Below is the output from `/api/v1/bootstrap/2` for the failing hosts
(bootstrap response attached)

    
    
    <bootStrapStatus>
      <status>RUNNING</status>
      .....
    
    
      <hostsStatus>
        <hostName>perf-d-511.c.pramod-thangali.internal</hostName>
        <status>RUNNING</status>
        <log>
    ==========================
    Creating target directory...
    ==========================
    
    Command start time 2016-02-17 18:55:54
    chmod: cannot access `/var/lib/ambari-agent/data': No such file or directory
    
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
    and the key for the corresponding IP address 10.240.5.174
    has a different value. This could either mean that
    DNS SPOOFING is happening or the IP address for the host
    and its host key have changed at the same time.
    Offending key for IP in /root/.ssh/known_hosts:1418
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
    Someone could be eavesdropping on you right now (man-in-the-middle attack)!
    It is also possible that the RSA host key has just been changed.
    The fingerprint for the RSA key sent by the remote host is
    66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
    Please contact your system administrator.
    Add correct host key in /root/.ssh/known_hosts to get rid of this message.
    Offending key in /root/.ssh/known_hosts:1417
    Password authentication is disabled to avoid man-in-the-middle attacks.
    Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
    Connection to perf-d-511.c.pramod-thangali.internal closed.
    SSH command execution finished
    host=perf-d-511.c.pramod-thangali.internal, exitcode=0
    Command end time 2016-02-17 18:55:55
    
    ==========================
    Copying common functions script...
    ==========================
    
    Command start time 2016-02-17 18:55:55
    
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
    and the key for the corresponding IP address 10.240.5.174
    has a different value. This could either mean that
    DNS SPOOFING is happening or the IP address for the host
    and its host key have changed at the same time.
    Offending key for IP in /root/.ssh/known_hosts:1418
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
    Someone could be eavesdropping on you right now (man-in-the-middle attack)!
    It is also possible that the RSA host key has just been changed.
    The fingerprint for the RSA key sent by the remote host is
    66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
    Please contact your system administrator.
    Add correct host key in /root/.ssh/known_hosts to get rid of this message.
    Offending key in /root/.ssh/known_hosts:1417
    Password authentication is disabled to avoid man-in-the-middle attacks.
    Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
    scp /usr/lib/python2.6/site-packages/ambari_commons
    host=perf-d-511.c.pramod-thangali.internal, exitcode=0
    Command end time 2016-02-17 18:55:56
    
    ==========================
    Copying OS type check script...
    ==========================
    
    Command start time 2016-02-17 18:55:56
    
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
    and the key for the corresponding IP address 10.240.5.174
    has a different value. This could either mean that
    DNS SPOOFING is happening or the IP address for the host
    and its host key have changed at the same time.
    Offending key for IP in /root/.ssh/known_hosts:1418
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
    Someone could be eavesdropping on you right now (man-in-the-middle attack)!
    It is also possible that the RSA host key has just been changed.
    The fingerprint for the RSA key sent by the remote host is
    66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
    Please contact your system administrator.
    Add correct host key in /root/.ssh/known_hosts to get rid of this message.
    Offending key in /root/.ssh/known_hosts:1417
    Password authentication is disabled to avoid man-in-the-middle attacks.
    Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
    scp /usr/lib/python2.6/site-packages/ambari_server/os_check_type.py
    host=perf-d-511.c.pramod-thangali.internal, exitcode=0
    Command end time 2016-02-17 18:55:56
    
    ==========================
    Running OS type check...
    ==========================
    
    Command start time 2016-02-17 18:55:56
    Cluster primary/cluster OS family is redhat6 and local/current OS family is redhat6
    
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
    and the key for the corresponding IP address 10.240.5.174
    has a different value. This could either mean that
    DNS SPOOFING is happening or the IP address for the host
    and its host key have changed at the same time.
    Offending key for IP in /root/.ssh/known_hosts:1418
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
    Someone could be eavesdropping on you right now (man-in-the-middle attack)!
    It is also possible that the RSA host key has just been changed.
    The fingerprint for the RSA key sent by the remote host is
    66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
    Please contact your system administrator.
    Add correct host key in /root/.ssh/known_hosts to get rid of this message.
    Offending key in /root/.ssh/known_hosts:1417
    Password authentication is disabled to avoid man-in-the-middle attacks.
    Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
    Connection to perf-d-511.c.pramod-thangali.internal closed.
    SSH command execution finished
    host=perf-d-511.c.pramod-thangali.internal, exitcode=0
    Command end time 2016-02-17 18:55:57
    
    ==========================
    Checking 'sudo' package on remote host...
    ==========================
    
    Command start time 2016-02-17 18:55:57
    sudo-1.8.6p3-20.el6_7.x86_64
    
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
    and the key for the corresponding IP address 10.240.5.174
    has a different value. This could either mean that
    DNS SPOOFING is happening or the IP address for the host
    and its host key have changed at the same time.
    Offending key for IP in /root/.ssh/known_hosts:1418
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
    Someone could be eavesdropping on you right now (man-in-the-middle attack)!
    It is also possible that the RSA host key has just been changed.
    The fingerprint for the RSA key sent by the remote host is
    66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
    Please contact your system administrator.
    Add correct host key in /root/.ssh/known_hosts to get rid of this message.
    Offending key in /root/.ssh/known_hosts:1417
    Password authentication is disabled to avoid man-in-the-middle attacks.
    Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
    Connection to perf-d-511.c.pramod-thangali.internal closed.
    SSH command execution finished
    host=perf-d-511.c.pramod-thangali.internal, exitcode=0
    Command end time 2016-02-17 18:55:58
    
    ==========================
    Copying repo file to 'tmp' folder...
    ==========================
    
    Command start time 2016-02-17 18:55:58
    
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
    and the key for the corresponding IP address 10.240.5.174
    has a different value. This could either mean that
    DNS SPOOFING is happening or the IP address for the host
    and its host key have changed at the same time.
    Offending key for IP in /root/.ssh/known_hosts:1418
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
    Someone could be eavesdropping on you right now (man-in-the-middle attack)!
    It is also possible that the RSA host key has just been changed.
    The fingerprint for the RSA key sent by the remote host is
    66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
    Please contact your system administrator.
    Add correct host key in /root/.ssh/known_hosts to get rid of this message.
    Offending key in /root/.ssh/known_hosts:1417
    Password authentication is disabled to avoid man-in-the-middle attacks.
    Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
    scp /etc/yum.repos.d/ambari.repo
    host=perf-d-511.c.pramod-thangali.internal, exitcode=0
    Command end time 2016-02-17 18:55:59
    
    ==========================
    Moving file to repo dir...
    ==========================
    
    Command start time 2016-02-17 18:55:59
    
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
    and the key for the corresponding IP address 10.240.5.174
    has a different value. This could either mean that
    DNS SPOOFING is happening or the IP address for the host
    and its host key have changed at the same time.
    Offending key for IP in /root/.ssh/known_hosts:1418
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
    Someone could be eavesdropping on you right now (man-in-the-middle attack)!
    It is also possible that the RSA host key has just been changed.
    The fingerprint for the RSA key sent by the remote host is
    66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
    Please contact your system administrator.
    Add correct host key in /root/.ssh/known_hosts to get rid of this message.
    Offending key in /root/.ssh/known_hosts:1417
    Password authentication is disabled to avoid man-in-the-middle attacks.
    Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
    Connection to perf-d-511.c.pramod-thangali.internal closed.
    SSH command execution finished
    host=perf-d-511.c.pramod-thangali.internal, exitcode=0
    Command end time 2016-02-17 18:55:59
    
    ==========================
    Changing permissions for ambari.repo...
    ==========================
    
    Command start time 2016-02-17 18:55:59
    
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
    and the key for the corresponding IP address 10.240.5.174
    has a different value. This could either mean that
    DNS SPOOFING is happening or the IP address for the host
    and its host key have changed at the same time.
    Offending key for IP in /root/.ssh/known_hosts:1418
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
    Someone could be eavesdropping on you right now (man-in-the-middle attack)!
    It is also possible that the RSA host key has just been changed.
    The fingerprint for the RSA key sent by the remote host is
    66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
    Please contact your system administrator.
    Add correct host key in /root/.ssh/known_hosts to get rid of this message.
    Offending key in /root/.ssh/known_hosts:1417
    Password authentication is disabled to avoid man-in-the-middle attacks.
    Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
    Connection to perf-d-511.c.pramod-thangali.internal closed.
    SSH command execution finished
    host=perf-d-511.c.pramod-thangali.internal, exitcode=0
    Command end time 2016-02-17 18:56:00
    
    ==========================
    Copying setup script file...
    ==========================
    
    Command start time 2016-02-17 18:56:00
    
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
    and the key for the corresponding IP address 10.240.5.174
    has a different value. This could either mean that
    DNS SPOOFING is happening or the IP address for the host
    and its host key have changed at the same time.
    Offending key for IP in /root/.ssh/known_hosts:1418
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
    Someone could be eavesdropping on you right now (man-in-the-middle attack)!
    It is also possible that the RSA host key has just been changed.
    The fingerprint for the RSA key sent by the remote host is
    66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
    Please contact your system administrator.
    Add correct host key in /root/.ssh/known_hosts to get rid of this message.
    Offending key in /root/.ssh/known_hosts:1417
    Password authentication is disabled to avoid man-in-the-middle attacks.
    Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
    scp /usr/lib/python2.6/site-packages/ambari_server/setupAgent.py
    host=perf-d-511.c.pramod-thangali.internal, exitcode=0
    Command end time 2016-02-17 18:56:00
    
    ==========================
    Running setup agent script...
    ==========================
    
    Command start time 2016-02-17 18:56:00
    Repository base is listed more than once in the configuration
    Repository updates is listed more than once in the configuration
    Repository extras is listed more than once in the configuration
    Error: Cannot find a valid baseurl for repo: base
    Repository base is listed more than once in the configuration
    Repository updates is listed more than once in the configuration
    Repository extras is listed more than once in the configuration
    Error: Cannot find a valid baseurl for repo: base
    ('', None)
    
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
    and the key for the corresponding IP address 10.240.5.174
    has a different value. This could either mean that
    DNS SPOOFING is happening or the IP address for the host
    and its host key have changed at the same time.
    Offending key for IP in /root/.ssh/known_hosts:1418
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
    Someone could be eavesdropping on you right now (man-in-the-middle attack)!
    It is also possible that the RSA host key has just been changed.
    The fingerprint for the RSA key sent by the remote host is
    66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
    Please contact your system administrator.
    Add correct host key in /root/.ssh/known_hosts to get rid of this message.
    Offending key in /root/.ssh/known_hosts:1417
    Password authentication is disabled to avoid man-in-the-middle attacks.
    Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
    Connection to perf-d-511.c.pramod-thangali.internal closed.
    SSH command execution finished
    host=perf-d-511.c.pramod-thangali.internal, exitcode=1
    Command end time 2016-02-17 18:56:47
    
    ERROR: Bootstrap of host perf-d-511.c.pramod-thangali.internal fails because previous action finished with non-zero exit code (1)
    ERROR MESSAGE: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
    and the key for the corresponding IP address 10.240.5.174
    has a different value. This could either mean that
    DNS SPOOFING is happening or the IP address for the host
    and its host key have changed at the same time.
    Offending key for IP in /root/.ssh/known_hosts:1418
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
    Someone could be eavesdropping on you right now (man-in-the-middle attack)!
    It is also possible that the RSA host key has just been changed.
    The fingerprint for the RSA key sent by the remote host is
    66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
    Please contact your system administrator.
    Add correct host key in /root/.ssh/known_hosts to get rid of this message.
    Offending key in /root/.ssh/known_hosts:1417
    Password authentication is disabled to avoid man-in-the-middle attacks.
    Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
    Connection to perf-d-511.c.pramod-thangali.internal closed.
    
    STDOUT: Repository base is listed more than once in the configuration
    Repository updates is listed more than once in the configuration
    Repository extras is listed more than once in the configuration
    Error: Cannot find a valid baseurl for repo: base
    Repository base is listed more than once in the configuration
    Repository updates is listed more than once in the configuration
    Repository extras is listed more than once in the configuration
    Error: Cannot find a valid baseurl for repo: base
    ('', None)
    
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
    and the key for the corresponding IP address 10.240.5.174
    has a different value. This could either mean that
    DNS SPOOFING is happening or the IP address for the host
    and its host key have changed at the same time.
    Offending key for IP in /root/.ssh/known_hosts:1418
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
    Someone could be eavesdropping on you right now (man-in-the-middle attack)!
    It is also possible that the RSA host key has just been changed.
    The fingerprint for the RSA key sent by the remote host is
    66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
    Please contact your system administrator.
    Add correct host key in /root/.ssh/known_hosts to get rid of this message.
    Offending key in /root/.ssh/known_hosts:1417
    Password authentication is disabled to avoid man-in-the-middle attacks.
    Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
    Connection to perf-d-511.c.pramod-thangali.internal closed.
      </log>
    </hostsStatus>
    
    
    ...


Diffs
-----

  ambari-server/src/main/java/org/apache/ambari/server/bootstrap/BSRunner.java bf43bc9 

Diff: https://reviews.apache.org/r/44086/diff/


Testing
-------

mvn clean test


Thanks,

Andrew Onischuk

Re: Review Request 44086: Registering 700 hosts gets stuck without timeout on 8 hosts

Posted by Alejandro Fernandez <af...@hortonworks.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44086/#review120933
-----------------------------------------------------------


Ship it!




Ship It!

- Alejandro Fernandez


On Feb. 26, 2016, 5:07 p.m., Andrew Onischuk wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44086/
> -----------------------------------------------------------
> 
> (Updated Feb. 26, 2016, 5:07 p.m.)
> 
> 
> Review request for Ambari and Myroslav Papirkovskyy.
> 
> 
> Bugs: AMBARI-15205
>     https://issues.apache.org/jira/browse/AMBARI-15205
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> When adding 700 hosts through the UI's add-host-wizard, all hosts successfully
> registered except for 8 of them. These 8 hosts were stuck in RUNNING state and
> would not timeout even after 20 minutes.
> 
> I had to kill the bootstrap process on ambari-server to make the operation to
> complete.  
> On my second attempt I was able to recreate the problem
> 
> Below is the output from `/api/v1/bootstrap/2` for the failing hosts
> (bootstrap response attached)
> 
>     
>     
>     <bootStrapStatus>
>       <status>RUNNING</status>
>       .....
>     
>     
>       <hostsStatus>
>         <hostName>perf-d-511.c.pramod-thangali.internal</hostName>
>         <status>RUNNING</status>
>         <log>
>     ==========================
>     Creating target directory...
>     ==========================
>     
>     Command start time 2016-02-17 18:55:54
>     chmod: cannot access `/var/lib/ambari-agent/data': No such file or directory
>     
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
>     and the key for the corresponding IP address 10.240.5.174
>     has a different value. This could either mean that
>     DNS SPOOFING is happening or the IP address for the host
>     and its host key have changed at the same time.
>     Offending key for IP in /root/.ssh/known_hosts:1418
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
>     Someone could be eavesdropping on you right now (man-in-the-middle attack)!
>     It is also possible that the RSA host key has just been changed.
>     The fingerprint for the RSA key sent by the remote host is
>     66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
>     Please contact your system administrator.
>     Add correct host key in /root/.ssh/known_hosts to get rid of this message.
>     Offending key in /root/.ssh/known_hosts:1417
>     Password authentication is disabled to avoid man-in-the-middle attacks.
>     Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
>     Connection to perf-d-511.c.pramod-thangali.internal closed.
>     SSH command execution finished
>     host=perf-d-511.c.pramod-thangali.internal, exitcode=0
>     Command end time 2016-02-17 18:55:55
>     
>     ==========================
>     Copying common functions script...
>     ==========================
>     
>     Command start time 2016-02-17 18:55:55
>     
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
>     and the key for the corresponding IP address 10.240.5.174
>     has a different value. This could either mean that
>     DNS SPOOFING is happening or the IP address for the host
>     and its host key have changed at the same time.
>     Offending key for IP in /root/.ssh/known_hosts:1418
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
>     Someone could be eavesdropping on you right now (man-in-the-middle attack)!
>     It is also possible that the RSA host key has just been changed.
>     The fingerprint for the RSA key sent by the remote host is
>     66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
>     Please contact your system administrator.
>     Add correct host key in /root/.ssh/known_hosts to get rid of this message.
>     Offending key in /root/.ssh/known_hosts:1417
>     Password authentication is disabled to avoid man-in-the-middle attacks.
>     Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
>     scp /usr/lib/python2.6/site-packages/ambari_commons
>     host=perf-d-511.c.pramod-thangali.internal, exitcode=0
>     Command end time 2016-02-17 18:55:56
>     
>     ==========================
>     Copying OS type check script...
>     ==========================
>     
>     Command start time 2016-02-17 18:55:56
>     
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
>     and the key for the corresponding IP address 10.240.5.174
>     has a different value. This could either mean that
>     DNS SPOOFING is happening or the IP address for the host
>     and its host key have changed at the same time.
>     Offending key for IP in /root/.ssh/known_hosts:1418
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
>     Someone could be eavesdropping on you right now (man-in-the-middle attack)!
>     It is also possible that the RSA host key has just been changed.
>     The fingerprint for the RSA key sent by the remote host is
>     66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
>     Please contact your system administrator.
>     Add correct host key in /root/.ssh/known_hosts to get rid of this message.
>     Offending key in /root/.ssh/known_hosts:1417
>     Password authentication is disabled to avoid man-in-the-middle attacks.
>     Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
>     scp /usr/lib/python2.6/site-packages/ambari_server/os_check_type.py
>     host=perf-d-511.c.pramod-thangali.internal, exitcode=0
>     Command end time 2016-02-17 18:55:56
>     
>     ==========================
>     Running OS type check...
>     ==========================
>     
>     Command start time 2016-02-17 18:55:56
>     Cluster primary/cluster OS family is redhat6 and local/current OS family is redhat6
>     
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
>     and the key for the corresponding IP address 10.240.5.174
>     has a different value. This could either mean that
>     DNS SPOOFING is happening or the IP address for the host
>     and its host key have changed at the same time.
>     Offending key for IP in /root/.ssh/known_hosts:1418
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
>     Someone could be eavesdropping on you right now (man-in-the-middle attack)!
>     It is also possible that the RSA host key has just been changed.
>     The fingerprint for the RSA key sent by the remote host is
>     66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
>     Please contact your system administrator.
>     Add correct host key in /root/.ssh/known_hosts to get rid of this message.
>     Offending key in /root/.ssh/known_hosts:1417
>     Password authentication is disabled to avoid man-in-the-middle attacks.
>     Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
>     Connection to perf-d-511.c.pramod-thangali.internal closed.
>     SSH command execution finished
>     host=perf-d-511.c.pramod-thangali.internal, exitcode=0
>     Command end time 2016-02-17 18:55:57
>     
>     ==========================
>     Checking 'sudo' package on remote host...
>     ==========================
>     
>     Command start time 2016-02-17 18:55:57
>     sudo-1.8.6p3-20.el6_7.x86_64
>     
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
>     and the key for the corresponding IP address 10.240.5.174
>     has a different value. This could either mean that
>     DNS SPOOFING is happening or the IP address for the host
>     and its host key have changed at the same time.
>     Offending key for IP in /root/.ssh/known_hosts:1418
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
>     Someone could be eavesdropping on you right now (man-in-the-middle attack)!
>     It is also possible that the RSA host key has just been changed.
>     The fingerprint for the RSA key sent by the remote host is
>     66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
>     Please contact your system administrator.
>     Add correct host key in /root/.ssh/known_hosts to get rid of this message.
>     Offending key in /root/.ssh/known_hosts:1417
>     Password authentication is disabled to avoid man-in-the-middle attacks.
>     Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
>     Connection to perf-d-511.c.pramod-thangali.internal closed.
>     SSH command execution finished
>     host=perf-d-511.c.pramod-thangali.internal, exitcode=0
>     Command end time 2016-02-17 18:55:58
>     
>     ==========================
>     Copying repo file to 'tmp' folder...
>     ==========================
>     
>     Command start time 2016-02-17 18:55:58
>     
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
>     and the key for the corresponding IP address 10.240.5.174
>     has a different value. This could either mean that
>     DNS SPOOFING is happening or the IP address for the host
>     and its host key have changed at the same time.
>     Offending key for IP in /root/.ssh/known_hosts:1418
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
>     Someone could be eavesdropping on you right now (man-in-the-middle attack)!
>     It is also possible that the RSA host key has just been changed.
>     The fingerprint for the RSA key sent by the remote host is
>     66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
>     Please contact your system administrator.
>     Add correct host key in /root/.ssh/known_hosts to get rid of this message.
>     Offending key in /root/.ssh/known_hosts:1417
>     Password authentication is disabled to avoid man-in-the-middle attacks.
>     Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
>     scp /etc/yum.repos.d/ambari.repo
>     host=perf-d-511.c.pramod-thangali.internal, exitcode=0
>     Command end time 2016-02-17 18:55:59
>     
>     ==========================
>     Moving file to repo dir...
>     ==========================
>     
>     Command start time 2016-02-17 18:55:59
>     
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
>     and the key for the corresponding IP address 10.240.5.174
>     has a different value. This could either mean that
>     DNS SPOOFING is happening or the IP address for the host
>     and its host key have changed at the same time.
>     Offending key for IP in /root/.ssh/known_hosts:1418
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
>     Someone could be eavesdropping on you right now (man-in-the-middle attack)!
>     It is also possible that the RSA host key has just been changed.
>     The fingerprint for the RSA key sent by the remote host is
>     66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
>     Please contact your system administrator.
>     Add correct host key in /root/.ssh/known_hosts to get rid of this message.
>     Offending key in /root/.ssh/known_hosts:1417
>     Password authentication is disabled to avoid man-in-the-middle attacks.
>     Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
>     Connection to perf-d-511.c.pramod-thangali.internal closed.
>     SSH command execution finished
>     host=perf-d-511.c.pramod-thangali.internal, exitcode=0
>     Command end time 2016-02-17 18:55:59
>     
>     ==========================
>     Changing permissions for ambari.repo...
>     ==========================
>     
>     Command start time 2016-02-17 18:55:59
>     
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
>     and the key for the corresponding IP address 10.240.5.174
>     has a different value. This could either mean that
>     DNS SPOOFING is happening or the IP address for the host
>     and its host key have changed at the same time.
>     Offending key for IP in /root/.ssh/known_hosts:1418
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
>     Someone could be eavesdropping on you right now (man-in-the-middle attack)!
>     It is also possible that the RSA host key has just been changed.
>     The fingerprint for the RSA key sent by the remote host is
>     66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
>     Please contact your system administrator.
>     Add correct host key in /root/.ssh/known_hosts to get rid of this message.
>     Offending key in /root/.ssh/known_hosts:1417
>     Password authentication is disabled to avoid man-in-the-middle attacks.
>     Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
>     Connection to perf-d-511.c.pramod-thangali.internal closed.
>     SSH command execution finished
>     host=perf-d-511.c.pramod-thangali.internal, exitcode=0
>     Command end time 2016-02-17 18:56:00
>     
>     ==========================
>     Copying setup script file...
>     ==========================
>     
>     Command start time 2016-02-17 18:56:00
>     
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
>     and the key for the corresponding IP address 10.240.5.174
>     has a different value. This could either mean that
>     DNS SPOOFING is happening or the IP address for the host
>     and its host key have changed at the same time.
>     Offending key for IP in /root/.ssh/known_hosts:1418
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
>     Someone could be eavesdropping on you right now (man-in-the-middle attack)!
>     It is also possible that the RSA host key has just been changed.
>     The fingerprint for the RSA key sent by the remote host is
>     66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
>     Please contact your system administrator.
>     Add correct host key in /root/.ssh/known_hosts to get rid of this message.
>     Offending key in /root/.ssh/known_hosts:1417
>     Password authentication is disabled to avoid man-in-the-middle attacks.
>     Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
>     scp /usr/lib/python2.6/site-packages/ambari_server/setupAgent.py
>     host=perf-d-511.c.pramod-thangali.internal, exitcode=0
>     Command end time 2016-02-17 18:56:00
>     
>     ==========================
>     Running setup agent script...
>     ==========================
>     
>     Command start time 2016-02-17 18:56:00
>     Repository base is listed more than once in the configuration
>     Repository updates is listed more than once in the configuration
>     Repository extras is listed more than once in the configuration
>     Error: Cannot find a valid baseurl for repo: base
>     Repository base is listed more than once in the configuration
>     Repository updates is listed more than once in the configuration
>     Repository extras is listed more than once in the configuration
>     Error: Cannot find a valid baseurl for repo: base
>     ('', None)
>     
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
>     and the key for the corresponding IP address 10.240.5.174
>     has a different value. This could either mean that
>     DNS SPOOFING is happening or the IP address for the host
>     and its host key have changed at the same time.
>     Offending key for IP in /root/.ssh/known_hosts:1418
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
>     Someone could be eavesdropping on you right now (man-in-the-middle attack)!
>     It is also possible that the RSA host key has just been changed.
>     The fingerprint for the RSA key sent by the remote host is
>     66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
>     Please contact your system administrator.
>     Add correct host key in /root/.ssh/known_hosts to get rid of this message.
>     Offending key in /root/.ssh/known_hosts:1417
>     Password authentication is disabled to avoid man-in-the-middle attacks.
>     Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
>     Connection to perf-d-511.c.pramod-thangali.internal closed.
>     SSH command execution finished
>     host=perf-d-511.c.pramod-thangali.internal, exitcode=1
>     Command end time 2016-02-17 18:56:47
>     
>     ERROR: Bootstrap of host perf-d-511.c.pramod-thangali.internal fails because previous action finished with non-zero exit code (1)
>     ERROR MESSAGE: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
>     and the key for the corresponding IP address 10.240.5.174
>     has a different value. This could either mean that
>     DNS SPOOFING is happening or the IP address for the host
>     and its host key have changed at the same time.
>     Offending key for IP in /root/.ssh/known_hosts:1418
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
>     Someone could be eavesdropping on you right now (man-in-the-middle attack)!
>     It is also possible that the RSA host key has just been changed.
>     The fingerprint for the RSA key sent by the remote host is
>     66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
>     Please contact your system administrator.
>     Add correct host key in /root/.ssh/known_hosts to get rid of this message.
>     Offending key in /root/.ssh/known_hosts:1417
>     Password authentication is disabled to avoid man-in-the-middle attacks.
>     Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
>     Connection to perf-d-511.c.pramod-thangali.internal closed.
>     
>     STDOUT: Repository base is listed more than once in the configuration
>     Repository updates is listed more than once in the configuration
>     Repository extras is listed more than once in the configuration
>     Error: Cannot find a valid baseurl for repo: base
>     Repository base is listed more than once in the configuration
>     Repository updates is listed more than once in the configuration
>     Repository extras is listed more than once in the configuration
>     Error: Cannot find a valid baseurl for repo: base
>     ('', None)
>     
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
>     and the key for the corresponding IP address 10.240.5.174
>     has a different value. This could either mean that
>     DNS SPOOFING is happening or the IP address for the host
>     and its host key have changed at the same time.
>     Offending key for IP in /root/.ssh/known_hosts:1418
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
>     Someone could be eavesdropping on you right now (man-in-the-middle attack)!
>     It is also possible that the RSA host key has just been changed.
>     The fingerprint for the RSA key sent by the remote host is
>     66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
>     Please contact your system administrator.
>     Add correct host key in /root/.ssh/known_hosts to get rid of this message.
>     Offending key in /root/.ssh/known_hosts:1417
>     Password authentication is disabled to avoid man-in-the-middle attacks.
>     Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
>     Connection to perf-d-511.c.pramod-thangali.internal closed.
>       </log>
>     </hostsStatus>
>     
>     
>     ...
> 
> 
> Diffs
> -----
> 
>   ambari-server/src/main/java/org/apache/ambari/server/bootstrap/BSRunner.java bf43bc9 
> 
> Diff: https://reviews.apache.org/r/44086/diff/
> 
> 
> Testing
> -------
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Andrew Onischuk
> 
>

Re: Review Request 44086: Registering 700 hosts gets stuck without timeout on 8 hosts

Posted by Sumit Mohanty <sm...@hortonworks.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44086/#review120889
-----------------------------------------------------------




ambari-server/src/main/java/org/apache/ambari/server/bootstrap/BSRunner.java (line 43)
<https://reviews.apache.org/r/44086/#comment182414>

    Why 320?



ambari-server/src/main/java/org/apache/ambari/server/bootstrap/BSRunner.java (line 160)
<https://reviews.apache.org/r/44086/#comment182415>

    Too frequent? Why not 1 minute?


- Sumit Mohanty


On Feb. 26, 2016, 5:07 p.m., Andrew Onischuk wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44086/
> -----------------------------------------------------------
> 
> (Updated Feb. 26, 2016, 5:07 p.m.)
> 
> 
> Review request for Ambari and Myroslav Papirkovskyy.
> 
> 
> Bugs: AMBARI-15205
>     https://issues.apache.org/jira/browse/AMBARI-15205
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> When adding 700 hosts through the UI's add-host-wizard, all hosts successfully
> registered except for 8 of them. These 8 hosts were stuck in RUNNING state and
> would not timeout even after 20 minutes.
> 
> I had to kill the bootstrap process on ambari-server to make the operation to
> complete.  
> On my second attempt I was able to recreate the problem
> 
> Below is the output from `/api/v1/bootstrap/2` for the failing hosts
> (bootstrap response attached)
> 
>     
>     
>     <bootStrapStatus>
>       <status>RUNNING</status>
>       .....
>     
>     
>       <hostsStatus>
>         <hostName>perf-d-511.c.pramod-thangali.internal</hostName>
>         <status>RUNNING</status>
>         <log>
>     ==========================
>     Creating target directory...
>     ==========================
>     
>     Command start time 2016-02-17 18:55:54
>     chmod: cannot access `/var/lib/ambari-agent/data': No such file or directory
>     
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
>     and the key for the corresponding IP address 10.240.5.174
>     has a different value. This could either mean that
>     DNS SPOOFING is happening or the IP address for the host
>     and its host key have changed at the same time.
>     Offending key for IP in /root/.ssh/known_hosts:1418
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
>     Someone could be eavesdropping on you right now (man-in-the-middle attack)!
>     It is also possible that the RSA host key has just been changed.
>     The fingerprint for the RSA key sent by the remote host is
>     66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
>     Please contact your system administrator.
>     Add correct host key in /root/.ssh/known_hosts to get rid of this message.
>     Offending key in /root/.ssh/known_hosts:1417
>     Password authentication is disabled to avoid man-in-the-middle attacks.
>     Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
>     Connection to perf-d-511.c.pramod-thangali.internal closed.
>     SSH command execution finished
>     host=perf-d-511.c.pramod-thangali.internal, exitcode=0
>     Command end time 2016-02-17 18:55:55
>     
>     ==========================
>     Copying common functions script...
>     ==========================
>     
>     Command start time 2016-02-17 18:55:55
>     
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
>     and the key for the corresponding IP address 10.240.5.174
>     has a different value. This could either mean that
>     DNS SPOOFING is happening or the IP address for the host
>     and its host key have changed at the same time.
>     Offending key for IP in /root/.ssh/known_hosts:1418
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
>     Someone could be eavesdropping on you right now (man-in-the-middle attack)!
>     It is also possible that the RSA host key has just been changed.
>     The fingerprint for the RSA key sent by the remote host is
>     66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
>     Please contact your system administrator.
>     Add correct host key in /root/.ssh/known_hosts to get rid of this message.
>     Offending key in /root/.ssh/known_hosts:1417
>     Password authentication is disabled to avoid man-in-the-middle attacks.
>     Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
>     scp /usr/lib/python2.6/site-packages/ambari_commons
>     host=perf-d-511.c.pramod-thangali.internal, exitcode=0
>     Command end time 2016-02-17 18:55:56
>     
>     ==========================
>     Copying OS type check script...
>     ==========================
>     
>     Command start time 2016-02-17 18:55:56
>     
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
>     and the key for the corresponding IP address 10.240.5.174
>     has a different value. This could either mean that
>     DNS SPOOFING is happening or the IP address for the host
>     and its host key have changed at the same time.
>     Offending key for IP in /root/.ssh/known_hosts:1418
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
>     Someone could be eavesdropping on you right now (man-in-the-middle attack)!
>     It is also possible that the RSA host key has just been changed.
>     The fingerprint for the RSA key sent by the remote host is
>     66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
>     Please contact your system administrator.
>     Add correct host key in /root/.ssh/known_hosts to get rid of this message.
>     Offending key in /root/.ssh/known_hosts:1417
>     Password authentication is disabled to avoid man-in-the-middle attacks.
>     Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
>     scp /usr/lib/python2.6/site-packages/ambari_server/os_check_type.py
>     host=perf-d-511.c.pramod-thangali.internal, exitcode=0
>     Command end time 2016-02-17 18:55:56
>     
>     ==========================
>     Running OS type check...
>     ==========================
>     
>     Command start time 2016-02-17 18:55:56
>     Cluster primary/cluster OS family is redhat6 and local/current OS family is redhat6
>     
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
>     and the key for the corresponding IP address 10.240.5.174
>     has a different value. This could either mean that
>     DNS SPOOFING is happening or the IP address for the host
>     and its host key have changed at the same time.
>     Offending key for IP in /root/.ssh/known_hosts:1418
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
>     Someone could be eavesdropping on you right now (man-in-the-middle attack)!
>     It is also possible that the RSA host key has just been changed.
>     The fingerprint for the RSA key sent by the remote host is
>     66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
>     Please contact your system administrator.
>     Add correct host key in /root/.ssh/known_hosts to get rid of this message.
>     Offending key in /root/.ssh/known_hosts:1417
>     Password authentication is disabled to avoid man-in-the-middle attacks.
>     Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
>     Connection to perf-d-511.c.pramod-thangali.internal closed.
>     SSH command execution finished
>     host=perf-d-511.c.pramod-thangali.internal, exitcode=0
>     Command end time 2016-02-17 18:55:57
>     
>     ==========================
>     Checking 'sudo' package on remote host...
>     ==========================
>     
>     Command start time 2016-02-17 18:55:57
>     sudo-1.8.6p3-20.el6_7.x86_64
>     
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
>     and the key for the corresponding IP address 10.240.5.174
>     has a different value. This could either mean that
>     DNS SPOOFING is happening or the IP address for the host
>     and its host key have changed at the same time.
>     Offending key for IP in /root/.ssh/known_hosts:1418
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
>     Someone could be eavesdropping on you right now (man-in-the-middle attack)!
>     It is also possible that the RSA host key has just been changed.
>     The fingerprint for the RSA key sent by the remote host is
>     66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
>     Please contact your system administrator.
>     Add correct host key in /root/.ssh/known_hosts to get rid of this message.
>     Offending key in /root/.ssh/known_hosts:1417
>     Password authentication is disabled to avoid man-in-the-middle attacks.
>     Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
>     Connection to perf-d-511.c.pramod-thangali.internal closed.
>     SSH command execution finished
>     host=perf-d-511.c.pramod-thangali.internal, exitcode=0
>     Command end time 2016-02-17 18:55:58
>     
>     ==========================
>     Copying repo file to 'tmp' folder...
>     ==========================
>     
>     Command start time 2016-02-17 18:55:58
>     
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
>     and the key for the corresponding IP address 10.240.5.174
>     has a different value. This could either mean that
>     DNS SPOOFING is happening or the IP address for the host
>     and its host key have changed at the same time.
>     Offending key for IP in /root/.ssh/known_hosts:1418
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
>     Someone could be eavesdropping on you right now (man-in-the-middle attack)!
>     It is also possible that the RSA host key has just been changed.
>     The fingerprint for the RSA key sent by the remote host is
>     66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
>     Please contact your system administrator.
>     Add correct host key in /root/.ssh/known_hosts to get rid of this message.
>     Offending key in /root/.ssh/known_hosts:1417
>     Password authentication is disabled to avoid man-in-the-middle attacks.
>     Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
>     scp /etc/yum.repos.d/ambari.repo
>     host=perf-d-511.c.pramod-thangali.internal, exitcode=0
>     Command end time 2016-02-17 18:55:59
>     
>     ==========================
>     Moving file to repo dir...
>     ==========================
>     
>     Command start time 2016-02-17 18:55:59
>     
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
>     and the key for the corresponding IP address 10.240.5.174
>     has a different value. This could either mean that
>     DNS SPOOFING is happening or the IP address for the host
>     and its host key have changed at the same time.
>     Offending key for IP in /root/.ssh/known_hosts:1418
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
>     Someone could be eavesdropping on you right now (man-in-the-middle attack)!
>     It is also possible that the RSA host key has just been changed.
>     The fingerprint for the RSA key sent by the remote host is
>     66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
>     Please contact your system administrator.
>     Add correct host key in /root/.ssh/known_hosts to get rid of this message.
>     Offending key in /root/.ssh/known_hosts:1417
>     Password authentication is disabled to avoid man-in-the-middle attacks.
>     Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
>     Connection to perf-d-511.c.pramod-thangali.internal closed.
>     SSH command execution finished
>     host=perf-d-511.c.pramod-thangali.internal, exitcode=0
>     Command end time 2016-02-17 18:55:59
>     
>     ==========================
>     Changing permissions for ambari.repo...
>     ==========================
>     
>     Command start time 2016-02-17 18:55:59
>     
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
>     and the key for the corresponding IP address 10.240.5.174
>     has a different value. This could either mean that
>     DNS SPOOFING is happening or the IP address for the host
>     and its host key have changed at the same time.
>     Offending key for IP in /root/.ssh/known_hosts:1418
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
>     Someone could be eavesdropping on you right now (man-in-the-middle attack)!
>     It is also possible that the RSA host key has just been changed.
>     The fingerprint for the RSA key sent by the remote host is
>     66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
>     Please contact your system administrator.
>     Add correct host key in /root/.ssh/known_hosts to get rid of this message.
>     Offending key in /root/.ssh/known_hosts:1417
>     Password authentication is disabled to avoid man-in-the-middle attacks.
>     Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
>     Connection to perf-d-511.c.pramod-thangali.internal closed.
>     SSH command execution finished
>     host=perf-d-511.c.pramod-thangali.internal, exitcode=0
>     Command end time 2016-02-17 18:56:00
>     
>     ==========================
>     Copying setup script file...
>     ==========================
>     
>     Command start time 2016-02-17 18:56:00
>     
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
>     and the key for the corresponding IP address 10.240.5.174
>     has a different value. This could either mean that
>     DNS SPOOFING is happening or the IP address for the host
>     and its host key have changed at the same time.
>     Offending key for IP in /root/.ssh/known_hosts:1418
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
>     Someone could be eavesdropping on you right now (man-in-the-middle attack)!
>     It is also possible that the RSA host key has just been changed.
>     The fingerprint for the RSA key sent by the remote host is
>     66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
>     Please contact your system administrator.
>     Add correct host key in /root/.ssh/known_hosts to get rid of this message.
>     Offending key in /root/.ssh/known_hosts:1417
>     Password authentication is disabled to avoid man-in-the-middle attacks.
>     Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
>     scp /usr/lib/python2.6/site-packages/ambari_server/setupAgent.py
>     host=perf-d-511.c.pramod-thangali.internal, exitcode=0
>     Command end time 2016-02-17 18:56:00
>     
>     ==========================
>     Running setup agent script...
>     ==========================
>     
>     Command start time 2016-02-17 18:56:00
>     Repository base is listed more than once in the configuration
>     Repository updates is listed more than once in the configuration
>     Repository extras is listed more than once in the configuration
>     Error: Cannot find a valid baseurl for repo: base
>     Repository base is listed more than once in the configuration
>     Repository updates is listed more than once in the configuration
>     Repository extras is listed more than once in the configuration
>     Error: Cannot find a valid baseurl for repo: base
>     ('', None)
>     
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
>     and the key for the corresponding IP address 10.240.5.174
>     has a different value. This could either mean that
>     DNS SPOOFING is happening or the IP address for the host
>     and its host key have changed at the same time.
>     Offending key for IP in /root/.ssh/known_hosts:1418
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
>     Someone could be eavesdropping on you right now (man-in-the-middle attack)!
>     It is also possible that the RSA host key has just been changed.
>     The fingerprint for the RSA key sent by the remote host is
>     66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
>     Please contact your system administrator.
>     Add correct host key in /root/.ssh/known_hosts to get rid of this message.
>     Offending key in /root/.ssh/known_hosts:1417
>     Password authentication is disabled to avoid man-in-the-middle attacks.
>     Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
>     Connection to perf-d-511.c.pramod-thangali.internal closed.
>     SSH command execution finished
>     host=perf-d-511.c.pramod-thangali.internal, exitcode=1
>     Command end time 2016-02-17 18:56:47
>     
>     ERROR: Bootstrap of host perf-d-511.c.pramod-thangali.internal fails because previous action finished with non-zero exit code (1)
>     ERROR MESSAGE: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
>     and the key for the corresponding IP address 10.240.5.174
>     has a different value. This could either mean that
>     DNS SPOOFING is happening or the IP address for the host
>     and its host key have changed at the same time.
>     Offending key for IP in /root/.ssh/known_hosts:1418
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
>     Someone could be eavesdropping on you right now (man-in-the-middle attack)!
>     It is also possible that the RSA host key has just been changed.
>     The fingerprint for the RSA key sent by the remote host is
>     66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
>     Please contact your system administrator.
>     Add correct host key in /root/.ssh/known_hosts to get rid of this message.
>     Offending key in /root/.ssh/known_hosts:1417
>     Password authentication is disabled to avoid man-in-the-middle attacks.
>     Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
>     Connection to perf-d-511.c.pramod-thangali.internal closed.
>     
>     STDOUT: Repository base is listed more than once in the configuration
>     Repository updates is listed more than once in the configuration
>     Repository extras is listed more than once in the configuration
>     Error: Cannot find a valid baseurl for repo: base
>     Repository base is listed more than once in the configuration
>     Repository updates is listed more than once in the configuration
>     Repository extras is listed more than once in the configuration
>     Error: Cannot find a valid baseurl for repo: base
>     ('', None)
>     
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
>     and the key for the corresponding IP address 10.240.5.174
>     has a different value. This could either mean that
>     DNS SPOOFING is happening or the IP address for the host
>     and its host key have changed at the same time.
>     Offending key for IP in /root/.ssh/known_hosts:1418
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
>     Someone could be eavesdropping on you right now (man-in-the-middle attack)!
>     It is also possible that the RSA host key has just been changed.
>     The fingerprint for the RSA key sent by the remote host is
>     66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
>     Please contact your system administrator.
>     Add correct host key in /root/.ssh/known_hosts to get rid of this message.
>     Offending key in /root/.ssh/known_hosts:1417
>     Password authentication is disabled to avoid man-in-the-middle attacks.
>     Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
>     Connection to perf-d-511.c.pramod-thangali.internal closed.
>       </log>
>     </hostsStatus>
>     
>     
>     ...
> 
> 
> Diffs
> -----
> 
>   ambari-server/src/main/java/org/apache/ambari/server/bootstrap/BSRunner.java bf43bc9 
> 
> Diff: https://reviews.apache.org/r/44086/diff/
> 
> 
> Testing
> -------
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Andrew Onischuk
> 
>

Re: Review Request 44086: Registering 700 hosts gets stuck without timeout on 8 hosts

Posted by Alejandro Fernandez <af...@hortonworks.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44086/#review121279
-----------------------------------------------------------


Ship it!




Ship It!

- Alejandro Fernandez


On Feb. 29, 2016, 5:48 p.m., Andrew Onischuk wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44086/
> -----------------------------------------------------------
> 
> (Updated Feb. 29, 2016, 5:48 p.m.)
> 
> 
> Review request for Ambari and Myroslav Papirkovskyy.
> 
> 
> Bugs: AMBARI-15205
>     https://issues.apache.org/jira/browse/AMBARI-15205
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> When adding 700 hosts through the UI's add-host-wizard, all hosts successfully
> registered except for 8 of them. These 8 hosts were stuck in RUNNING state and
> would not timeout even after 20 minutes.
> 
> I had to kill the bootstrap process on ambari-server to make the operation to
> complete.  
> On my second attempt I was able to recreate the problem
> 
> Below is the output from `/api/v1/bootstrap/2` for the failing hosts
> (bootstrap response attached)
> 
>     
>     
>     <bootStrapStatus>
>       <status>RUNNING</status>
>       .....
>     
>     
>       <hostsStatus>
>         <hostName>perf-d-511.c.pramod-thangali.internal</hostName>
>         <status>RUNNING</status>
>         <log>
>     ==========================
>     Creating target directory...
>     ==========================
>     
>     Command start time 2016-02-17 18:55:54
>     chmod: cannot access `/var/lib/ambari-agent/data': No such file or directory
>     
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
>     and the key for the corresponding IP address 10.240.5.174
>     has a different value. This could either mean that
>     DNS SPOOFING is happening or the IP address for the host
>     and its host key have changed at the same time.
>     Offending key for IP in /root/.ssh/known_hosts:1418
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
>     Someone could be eavesdropping on you right now (man-in-the-middle attack)!
>     It is also possible that the RSA host key has just been changed.
>     The fingerprint for the RSA key sent by the remote host is
>     66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
>     Please contact your system administrator.
>     Add correct host key in /root/.ssh/known_hosts to get rid of this message.
>     Offending key in /root/.ssh/known_hosts:1417
>     Password authentication is disabled to avoid man-in-the-middle attacks.
>     Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
>     Connection to perf-d-511.c.pramod-thangali.internal closed.
>     SSH command execution finished
>     host=perf-d-511.c.pramod-thangali.internal, exitcode=0
>     Command end time 2016-02-17 18:55:55
>     
>     ==========================
>     Copying common functions script...
>     ==========================
>     
>     Command start time 2016-02-17 18:55:55
>     
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
>     and the key for the corresponding IP address 10.240.5.174
>     has a different value. This could either mean that
>     DNS SPOOFING is happening or the IP address for the host
>     and its host key have changed at the same time.
>     Offending key for IP in /root/.ssh/known_hosts:1418
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
>     Someone could be eavesdropping on you right now (man-in-the-middle attack)!
>     It is also possible that the RSA host key has just been changed.
>     The fingerprint for the RSA key sent by the remote host is
>     66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
>     Please contact your system administrator.
>     Add correct host key in /root/.ssh/known_hosts to get rid of this message.
>     Offending key in /root/.ssh/known_hosts:1417
>     Password authentication is disabled to avoid man-in-the-middle attacks.
>     Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
>     scp /usr/lib/python2.6/site-packages/ambari_commons
>     host=perf-d-511.c.pramod-thangali.internal, exitcode=0
>     Command end time 2016-02-17 18:55:56
>     
>     ==========================
>     Copying OS type check script...
>     ==========================
>     
>     Command start time 2016-02-17 18:55:56
>     
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
>     and the key for the corresponding IP address 10.240.5.174
>     has a different value. This could either mean that
>     DNS SPOOFING is happening or the IP address for the host
>     and its host key have changed at the same time.
>     Offending key for IP in /root/.ssh/known_hosts:1418
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
>     Someone could be eavesdropping on you right now (man-in-the-middle attack)!
>     It is also possible that the RSA host key has just been changed.
>     The fingerprint for the RSA key sent by the remote host is
>     66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
>     Please contact your system administrator.
>     Add correct host key in /root/.ssh/known_hosts to get rid of this message.
>     Offending key in /root/.ssh/known_hosts:1417
>     Password authentication is disabled to avoid man-in-the-middle attacks.
>     Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
>     scp /usr/lib/python2.6/site-packages/ambari_server/os_check_type.py
>     host=perf-d-511.c.pramod-thangali.internal, exitcode=0
>     Command end time 2016-02-17 18:55:56
>     
>     ==========================
>     Running OS type check...
>     ==========================
>     
>     Command start time 2016-02-17 18:55:56
>     Cluster primary/cluster OS family is redhat6 and local/current OS family is redhat6
>     
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
>     and the key for the corresponding IP address 10.240.5.174
>     has a different value. This could either mean that
>     DNS SPOOFING is happening or the IP address for the host
>     and its host key have changed at the same time.
>     Offending key for IP in /root/.ssh/known_hosts:1418
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
>     Someone could be eavesdropping on you right now (man-in-the-middle attack)!
>     It is also possible that the RSA host key has just been changed.
>     The fingerprint for the RSA key sent by the remote host is
>     66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
>     Please contact your system administrator.
>     Add correct host key in /root/.ssh/known_hosts to get rid of this message.
>     Offending key in /root/.ssh/known_hosts:1417
>     Password authentication is disabled to avoid man-in-the-middle attacks.
>     Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
>     Connection to perf-d-511.c.pramod-thangali.internal closed.
>     SSH command execution finished
>     host=perf-d-511.c.pramod-thangali.internal, exitcode=0
>     Command end time 2016-02-17 18:55:57
>     
>     ==========================
>     Checking 'sudo' package on remote host...
>     ==========================
>     
>     Command start time 2016-02-17 18:55:57
>     sudo-1.8.6p3-20.el6_7.x86_64
>     
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
>     and the key for the corresponding IP address 10.240.5.174
>     has a different value. This could either mean that
>     DNS SPOOFING is happening or the IP address for the host
>     and its host key have changed at the same time.
>     Offending key for IP in /root/.ssh/known_hosts:1418
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
>     Someone could be eavesdropping on you right now (man-in-the-middle attack)!
>     It is also possible that the RSA host key has just been changed.
>     The fingerprint for the RSA key sent by the remote host is
>     66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
>     Please contact your system administrator.
>     Add correct host key in /root/.ssh/known_hosts to get rid of this message.
>     Offending key in /root/.ssh/known_hosts:1417
>     Password authentication is disabled to avoid man-in-the-middle attacks.
>     Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
>     Connection to perf-d-511.c.pramod-thangali.internal closed.
>     SSH command execution finished
>     host=perf-d-511.c.pramod-thangali.internal, exitcode=0
>     Command end time 2016-02-17 18:55:58
>     
>     ==========================
>     Copying repo file to 'tmp' folder...
>     ==========================
>     
>     Command start time 2016-02-17 18:55:58
>     
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
>     and the key for the corresponding IP address 10.240.5.174
>     has a different value. This could either mean that
>     DNS SPOOFING is happening or the IP address for the host
>     and its host key have changed at the same time.
>     Offending key for IP in /root/.ssh/known_hosts:1418
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
>     Someone could be eavesdropping on you right now (man-in-the-middle attack)!
>     It is also possible that the RSA host key has just been changed.
>     The fingerprint for the RSA key sent by the remote host is
>     66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
>     Please contact your system administrator.
>     Add correct host key in /root/.ssh/known_hosts to get rid of this message.
>     Offending key in /root/.ssh/known_hosts:1417
>     Password authentication is disabled to avoid man-in-the-middle attacks.
>     Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
>     scp /etc/yum.repos.d/ambari.repo
>     host=perf-d-511.c.pramod-thangali.internal, exitcode=0
>     Command end time 2016-02-17 18:55:59
>     
>     ==========================
>     Moving file to repo dir...
>     ==========================
>     
>     Command start time 2016-02-17 18:55:59
>     
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
>     and the key for the corresponding IP address 10.240.5.174
>     has a different value. This could either mean that
>     DNS SPOOFING is happening or the IP address for the host
>     and its host key have changed at the same time.
>     Offending key for IP in /root/.ssh/known_hosts:1418
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
>     Someone could be eavesdropping on you right now (man-in-the-middle attack)!
>     It is also possible that the RSA host key has just been changed.
>     The fingerprint for the RSA key sent by the remote host is
>     66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
>     Please contact your system administrator.
>     Add correct host key in /root/.ssh/known_hosts to get rid of this message.
>     Offending key in /root/.ssh/known_hosts:1417
>     Password authentication is disabled to avoid man-in-the-middle attacks.
>     Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
>     Connection to perf-d-511.c.pramod-thangali.internal closed.
>     SSH command execution finished
>     host=perf-d-511.c.pramod-thangali.internal, exitcode=0
>     Command end time 2016-02-17 18:55:59
>     
>     ==========================
>     Changing permissions for ambari.repo...
>     ==========================
>     
>     Command start time 2016-02-17 18:55:59
>     
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
>     and the key for the corresponding IP address 10.240.5.174
>     has a different value. This could either mean that
>     DNS SPOOFING is happening or the IP address for the host
>     and its host key have changed at the same time.
>     Offending key for IP in /root/.ssh/known_hosts:1418
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
>     Someone could be eavesdropping on you right now (man-in-the-middle attack)!
>     It is also possible that the RSA host key has just been changed.
>     The fingerprint for the RSA key sent by the remote host is
>     66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
>     Please contact your system administrator.
>     Add correct host key in /root/.ssh/known_hosts to get rid of this message.
>     Offending key in /root/.ssh/known_hosts:1417
>     Password authentication is disabled to avoid man-in-the-middle attacks.
>     Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
>     Connection to perf-d-511.c.pramod-thangali.internal closed.
>     SSH command execution finished
>     host=perf-d-511.c.pramod-thangali.internal, exitcode=0
>     Command end time 2016-02-17 18:56:00
>     
>     ==========================
>     Copying setup script file...
>     ==========================
>     
>     Command start time 2016-02-17 18:56:00
>     
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
>     and the key for the corresponding IP address 10.240.5.174
>     has a different value. This could either mean that
>     DNS SPOOFING is happening or the IP address for the host
>     and its host key have changed at the same time.
>     Offending key for IP in /root/.ssh/known_hosts:1418
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
>     Someone could be eavesdropping on you right now (man-in-the-middle attack)!
>     It is also possible that the RSA host key has just been changed.
>     The fingerprint for the RSA key sent by the remote host is
>     66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
>     Please contact your system administrator.
>     Add correct host key in /root/.ssh/known_hosts to get rid of this message.
>     Offending key in /root/.ssh/known_hosts:1417
>     Password authentication is disabled to avoid man-in-the-middle attacks.
>     Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
>     scp /usr/lib/python2.6/site-packages/ambari_server/setupAgent.py
>     host=perf-d-511.c.pramod-thangali.internal, exitcode=0
>     Command end time 2016-02-17 18:56:00
>     
>     ==========================
>     Running setup agent script...
>     ==========================
>     
>     Command start time 2016-02-17 18:56:00
>     Repository base is listed more than once in the configuration
>     Repository updates is listed more than once in the configuration
>     Repository extras is listed more than once in the configuration
>     Error: Cannot find a valid baseurl for repo: base
>     Repository base is listed more than once in the configuration
>     Repository updates is listed more than once in the configuration
>     Repository extras is listed more than once in the configuration
>     Error: Cannot find a valid baseurl for repo: base
>     ('', None)
>     
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
>     and the key for the corresponding IP address 10.240.5.174
>     has a different value. This could either mean that
>     DNS SPOOFING is happening or the IP address for the host
>     and its host key have changed at the same time.
>     Offending key for IP in /root/.ssh/known_hosts:1418
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
>     Someone could be eavesdropping on you right now (man-in-the-middle attack)!
>     It is also possible that the RSA host key has just been changed.
>     The fingerprint for the RSA key sent by the remote host is
>     66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
>     Please contact your system administrator.
>     Add correct host key in /root/.ssh/known_hosts to get rid of this message.
>     Offending key in /root/.ssh/known_hosts:1417
>     Password authentication is disabled to avoid man-in-the-middle attacks.
>     Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
>     Connection to perf-d-511.c.pramod-thangali.internal closed.
>     SSH command execution finished
>     host=perf-d-511.c.pramod-thangali.internal, exitcode=1
>     Command end time 2016-02-17 18:56:47
>     
>     ERROR: Bootstrap of host perf-d-511.c.pramod-thangali.internal fails because previous action finished with non-zero exit code (1)
>     ERROR MESSAGE: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
>     and the key for the corresponding IP address 10.240.5.174
>     has a different value. This could either mean that
>     DNS SPOOFING is happening or the IP address for the host
>     and its host key have changed at the same time.
>     Offending key for IP in /root/.ssh/known_hosts:1418
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
>     Someone could be eavesdropping on you right now (man-in-the-middle attack)!
>     It is also possible that the RSA host key has just been changed.
>     The fingerprint for the RSA key sent by the remote host is
>     66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
>     Please contact your system administrator.
>     Add correct host key in /root/.ssh/known_hosts to get rid of this message.
>     Offending key in /root/.ssh/known_hosts:1417
>     Password authentication is disabled to avoid man-in-the-middle attacks.
>     Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
>     Connection to perf-d-511.c.pramod-thangali.internal closed.
>     
>     STDOUT: Repository base is listed more than once in the configuration
>     Repository updates is listed more than once in the configuration
>     Repository extras is listed more than once in the configuration
>     Error: Cannot find a valid baseurl for repo: base
>     Repository base is listed more than once in the configuration
>     Repository updates is listed more than once in the configuration
>     Repository extras is listed more than once in the configuration
>     Error: Cannot find a valid baseurl for repo: base
>     ('', None)
>     
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
>     and the key for the corresponding IP address 10.240.5.174
>     has a different value. This could either mean that
>     DNS SPOOFING is happening or the IP address for the host
>     and its host key have changed at the same time.
>     Offending key for IP in /root/.ssh/known_hosts:1418
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
>     Someone could be eavesdropping on you right now (man-in-the-middle attack)!
>     It is also possible that the RSA host key has just been changed.
>     The fingerprint for the RSA key sent by the remote host is
>     66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
>     Please contact your system administrator.
>     Add correct host key in /root/.ssh/known_hosts to get rid of this message.
>     Offending key in /root/.ssh/known_hosts:1417
>     Password authentication is disabled to avoid man-in-the-middle attacks.
>     Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
>     Connection to perf-d-511.c.pramod-thangali.internal closed.
>       </log>
>     </hostsStatus>
>     
>     
>     ...
> 
> 
> Diffs
> -----
> 
>   ambari-server/src/main/java/org/apache/ambari/server/bootstrap/BSRunner.java bf43bc9 
> 
> Diff: https://reviews.apache.org/r/44086/diff/
> 
> 
> Testing
> -------
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Andrew Onischuk
> 
>

Re: Review Request 44086: Registering 700 hosts gets stuck without timeout on 8 hosts

Posted by Andrew Onischuk <ao...@hortonworks.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44086/
-----------------------------------------------------------

(Updated Feb. 29, 2016, 5:48 p.m.)


Review request for Ambari and Myroslav Papirkovskyy.


Bugs: AMBARI-15205
    https://issues.apache.org/jira/browse/AMBARI-15205


Repository: ambari


Description
-------

When adding 700 hosts through the UI's add-host-wizard, all hosts successfully
registered except for 8 of them. These 8 hosts were stuck in RUNNING state and
would not timeout even after 20 minutes.

I had to kill the bootstrap process on ambari-server to make the operation to
complete.  
On my second attempt I was able to recreate the problem

Below is the output from `/api/v1/bootstrap/2` for the failing hosts
(bootstrap response attached)

    
    
    <bootStrapStatus>
      <status>RUNNING</status>
      .....
    
    
      <hostsStatus>
        <hostName>perf-d-511.c.pramod-thangali.internal</hostName>
        <status>RUNNING</status>
        <log>
    ==========================
    Creating target directory...
    ==========================
    
    Command start time 2016-02-17 18:55:54
    chmod: cannot access `/var/lib/ambari-agent/data': No such file or directory
    
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
    and the key for the corresponding IP address 10.240.5.174
    has a different value. This could either mean that
    DNS SPOOFING is happening or the IP address for the host
    and its host key have changed at the same time.
    Offending key for IP in /root/.ssh/known_hosts:1418
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
    Someone could be eavesdropping on you right now (man-in-the-middle attack)!
    It is also possible that the RSA host key has just been changed.
    The fingerprint for the RSA key sent by the remote host is
    66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
    Please contact your system administrator.
    Add correct host key in /root/.ssh/known_hosts to get rid of this message.
    Offending key in /root/.ssh/known_hosts:1417
    Password authentication is disabled to avoid man-in-the-middle attacks.
    Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
    Connection to perf-d-511.c.pramod-thangali.internal closed.
    SSH command execution finished
    host=perf-d-511.c.pramod-thangali.internal, exitcode=0
    Command end time 2016-02-17 18:55:55
    
    ==========================
    Copying common functions script...
    ==========================
    
    Command start time 2016-02-17 18:55:55
    
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
    and the key for the corresponding IP address 10.240.5.174
    has a different value. This could either mean that
    DNS SPOOFING is happening or the IP address for the host
    and its host key have changed at the same time.
    Offending key for IP in /root/.ssh/known_hosts:1418
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
    Someone could be eavesdropping on you right now (man-in-the-middle attack)!
    It is also possible that the RSA host key has just been changed.
    The fingerprint for the RSA key sent by the remote host is
    66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
    Please contact your system administrator.
    Add correct host key in /root/.ssh/known_hosts to get rid of this message.
    Offending key in /root/.ssh/known_hosts:1417
    Password authentication is disabled to avoid man-in-the-middle attacks.
    Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
    scp /usr/lib/python2.6/site-packages/ambari_commons
    host=perf-d-511.c.pramod-thangali.internal, exitcode=0
    Command end time 2016-02-17 18:55:56
    
    ==========================
    Copying OS type check script...
    ==========================
    
    Command start time 2016-02-17 18:55:56
    
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
    and the key for the corresponding IP address 10.240.5.174
    has a different value. This could either mean that
    DNS SPOOFING is happening or the IP address for the host
    and its host key have changed at the same time.
    Offending key for IP in /root/.ssh/known_hosts:1418
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
    Someone could be eavesdropping on you right now (man-in-the-middle attack)!
    It is also possible that the RSA host key has just been changed.
    The fingerprint for the RSA key sent by the remote host is
    66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
    Please contact your system administrator.
    Add correct host key in /root/.ssh/known_hosts to get rid of this message.
    Offending key in /root/.ssh/known_hosts:1417
    Password authentication is disabled to avoid man-in-the-middle attacks.
    Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
    scp /usr/lib/python2.6/site-packages/ambari_server/os_check_type.py
    host=perf-d-511.c.pramod-thangali.internal, exitcode=0
    Command end time 2016-02-17 18:55:56
    
    ==========================
    Running OS type check...
    ==========================
    
    Command start time 2016-02-17 18:55:56
    Cluster primary/cluster OS family is redhat6 and local/current OS family is redhat6
    
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
    and the key for the corresponding IP address 10.240.5.174
    has a different value. This could either mean that
    DNS SPOOFING is happening or the IP address for the host
    and its host key have changed at the same time.
    Offending key for IP in /root/.ssh/known_hosts:1418
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
    Someone could be eavesdropping on you right now (man-in-the-middle attack)!
    It is also possible that the RSA host key has just been changed.
    The fingerprint for the RSA key sent by the remote host is
    66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
    Please contact your system administrator.
    Add correct host key in /root/.ssh/known_hosts to get rid of this message.
    Offending key in /root/.ssh/known_hosts:1417
    Password authentication is disabled to avoid man-in-the-middle attacks.
    Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
    Connection to perf-d-511.c.pramod-thangali.internal closed.
    SSH command execution finished
    host=perf-d-511.c.pramod-thangali.internal, exitcode=0
    Command end time 2016-02-17 18:55:57
    
    ==========================
    Checking 'sudo' package on remote host...
    ==========================
    
    Command start time 2016-02-17 18:55:57
    sudo-1.8.6p3-20.el6_7.x86_64
    
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
    and the key for the corresponding IP address 10.240.5.174
    has a different value. This could either mean that
    DNS SPOOFING is happening or the IP address for the host
    and its host key have changed at the same time.
    Offending key for IP in /root/.ssh/known_hosts:1418
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
    Someone could be eavesdropping on you right now (man-in-the-middle attack)!
    It is also possible that the RSA host key has just been changed.
    The fingerprint for the RSA key sent by the remote host is
    66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
    Please contact your system administrator.
    Add correct host key in /root/.ssh/known_hosts to get rid of this message.
    Offending key in /root/.ssh/known_hosts:1417
    Password authentication is disabled to avoid man-in-the-middle attacks.
    Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
    Connection to perf-d-511.c.pramod-thangali.internal closed.
    SSH command execution finished
    host=perf-d-511.c.pramod-thangali.internal, exitcode=0
    Command end time 2016-02-17 18:55:58
    
    ==========================
    Copying repo file to 'tmp' folder...
    ==========================
    
    Command start time 2016-02-17 18:55:58
    
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
    and the key for the corresponding IP address 10.240.5.174
    has a different value. This could either mean that
    DNS SPOOFING is happening or the IP address for the host
    and its host key have changed at the same time.
    Offending key for IP in /root/.ssh/known_hosts:1418
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
    Someone could be eavesdropping on you right now (man-in-the-middle attack)!
    It is also possible that the RSA host key has just been changed.
    The fingerprint for the RSA key sent by the remote host is
    66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
    Please contact your system administrator.
    Add correct host key in /root/.ssh/known_hosts to get rid of this message.
    Offending key in /root/.ssh/known_hosts:1417
    Password authentication is disabled to avoid man-in-the-middle attacks.
    Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
    scp /etc/yum.repos.d/ambari.repo
    host=perf-d-511.c.pramod-thangali.internal, exitcode=0
    Command end time 2016-02-17 18:55:59
    
    ==========================
    Moving file to repo dir...
    ==========================
    
    Command start time 2016-02-17 18:55:59
    
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
    and the key for the corresponding IP address 10.240.5.174
    has a different value. This could either mean that
    DNS SPOOFING is happening or the IP address for the host
    and its host key have changed at the same time.
    Offending key for IP in /root/.ssh/known_hosts:1418
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
    Someone could be eavesdropping on you right now (man-in-the-middle attack)!
    It is also possible that the RSA host key has just been changed.
    The fingerprint for the RSA key sent by the remote host is
    66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
    Please contact your system administrator.
    Add correct host key in /root/.ssh/known_hosts to get rid of this message.
    Offending key in /root/.ssh/known_hosts:1417
    Password authentication is disabled to avoid man-in-the-middle attacks.
    Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
    Connection to perf-d-511.c.pramod-thangali.internal closed.
    SSH command execution finished
    host=perf-d-511.c.pramod-thangali.internal, exitcode=0
    Command end time 2016-02-17 18:55:59
    
    ==========================
    Changing permissions for ambari.repo...
    ==========================
    
    Command start time 2016-02-17 18:55:59
    
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
    and the key for the corresponding IP address 10.240.5.174
    has a different value. This could either mean that
    DNS SPOOFING is happening or the IP address for the host
    and its host key have changed at the same time.
    Offending key for IP in /root/.ssh/known_hosts:1418
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
    Someone could be eavesdropping on you right now (man-in-the-middle attack)!
    It is also possible that the RSA host key has just been changed.
    The fingerprint for the RSA key sent by the remote host is
    66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
    Please contact your system administrator.
    Add correct host key in /root/.ssh/known_hosts to get rid of this message.
    Offending key in /root/.ssh/known_hosts:1417
    Password authentication is disabled to avoid man-in-the-middle attacks.
    Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
    Connection to perf-d-511.c.pramod-thangali.internal closed.
    SSH command execution finished
    host=perf-d-511.c.pramod-thangali.internal, exitcode=0
    Command end time 2016-02-17 18:56:00
    
    ==========================
    Copying setup script file...
    ==========================
    
    Command start time 2016-02-17 18:56:00
    
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
    and the key for the corresponding IP address 10.240.5.174
    has a different value. This could either mean that
    DNS SPOOFING is happening or the IP address for the host
    and its host key have changed at the same time.
    Offending key for IP in /root/.ssh/known_hosts:1418
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
    Someone could be eavesdropping on you right now (man-in-the-middle attack)!
    It is also possible that the RSA host key has just been changed.
    The fingerprint for the RSA key sent by the remote host is
    66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
    Please contact your system administrator.
    Add correct host key in /root/.ssh/known_hosts to get rid of this message.
    Offending key in /root/.ssh/known_hosts:1417
    Password authentication is disabled to avoid man-in-the-middle attacks.
    Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
    scp /usr/lib/python2.6/site-packages/ambari_server/setupAgent.py
    host=perf-d-511.c.pramod-thangali.internal, exitcode=0
    Command end time 2016-02-17 18:56:00
    
    ==========================
    Running setup agent script...
    ==========================
    
    Command start time 2016-02-17 18:56:00
    Repository base is listed more than once in the configuration
    Repository updates is listed more than once in the configuration
    Repository extras is listed more than once in the configuration
    Error: Cannot find a valid baseurl for repo: base
    Repository base is listed more than once in the configuration
    Repository updates is listed more than once in the configuration
    Repository extras is listed more than once in the configuration
    Error: Cannot find a valid baseurl for repo: base
    ('', None)
    
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
    and the key for the corresponding IP address 10.240.5.174
    has a different value. This could either mean that
    DNS SPOOFING is happening or the IP address for the host
    and its host key have changed at the same time.
    Offending key for IP in /root/.ssh/known_hosts:1418
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
    Someone could be eavesdropping on you right now (man-in-the-middle attack)!
    It is also possible that the RSA host key has just been changed.
    The fingerprint for the RSA key sent by the remote host is
    66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
    Please contact your system administrator.
    Add correct host key in /root/.ssh/known_hosts to get rid of this message.
    Offending key in /root/.ssh/known_hosts:1417
    Password authentication is disabled to avoid man-in-the-middle attacks.
    Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
    Connection to perf-d-511.c.pramod-thangali.internal closed.
    SSH command execution finished
    host=perf-d-511.c.pramod-thangali.internal, exitcode=1
    Command end time 2016-02-17 18:56:47
    
    ERROR: Bootstrap of host perf-d-511.c.pramod-thangali.internal fails because previous action finished with non-zero exit code (1)
    ERROR MESSAGE: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
    and the key for the corresponding IP address 10.240.5.174
    has a different value. This could either mean that
    DNS SPOOFING is happening or the IP address for the host
    and its host key have changed at the same time.
    Offending key for IP in /root/.ssh/known_hosts:1418
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
    Someone could be eavesdropping on you right now (man-in-the-middle attack)!
    It is also possible that the RSA host key has just been changed.
    The fingerprint for the RSA key sent by the remote host is
    66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
    Please contact your system administrator.
    Add correct host key in /root/.ssh/known_hosts to get rid of this message.
    Offending key in /root/.ssh/known_hosts:1417
    Password authentication is disabled to avoid man-in-the-middle attacks.
    Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
    Connection to perf-d-511.c.pramod-thangali.internal closed.
    
    STDOUT: Repository base is listed more than once in the configuration
    Repository updates is listed more than once in the configuration
    Repository extras is listed more than once in the configuration
    Error: Cannot find a valid baseurl for repo: base
    Repository base is listed more than once in the configuration
    Repository updates is listed more than once in the configuration
    Repository extras is listed more than once in the configuration
    Error: Cannot find a valid baseurl for repo: base
    ('', None)
    
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
    and the key for the corresponding IP address 10.240.5.174
    has a different value. This could either mean that
    DNS SPOOFING is happening or the IP address for the host
    and its host key have changed at the same time.
    Offending key for IP in /root/.ssh/known_hosts:1418
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
    Someone could be eavesdropping on you right now (man-in-the-middle attack)!
    It is also possible that the RSA host key has just been changed.
    The fingerprint for the RSA key sent by the remote host is
    66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
    Please contact your system administrator.
    Add correct host key in /root/.ssh/known_hosts to get rid of this message.
    Offending key in /root/.ssh/known_hosts:1417
    Password authentication is disabled to avoid man-in-the-middle attacks.
    Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
    Connection to perf-d-511.c.pramod-thangali.internal closed.
      </log>
    </hostsStatus>
    
    
    ...


Diffs (updated)
-----

  ambari-server/src/main/java/org/apache/ambari/server/bootstrap/BSRunner.java bf43bc9 

Diff: https://reviews.apache.org/r/44086/diff/


Testing
-------

mvn clean test


Thanks,

Andrew Onischuk

Re: Review Request 44086: Registering 700 hosts gets stuck without timeout on 8 hosts

Posted by Andrew Onischuk <ao...@hortonworks.com>.


> On Feb. 26, 2016, 6:40 p.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/bootstrap/BSRunner.java, line 422
> > <https://reviews.apache.org/r/44086/diff/1/?file=1272177#file1272177line422>
> >
> >     Under what conditions would the file not exist?

If bootstrap timed out some .done files will be missing (when bootstrap.py is destroyed).

Old behavior (before the patch): in case of error in bootstrap.py BSRunner rewrites all .done files to '99' and api shows all tasks as failed.

We have the following case: successful bootstrap for some hosts and timed out for other. We need to create .done files for timed out hosts (if they are absent), but we won't change '0' for successful


- Andrew


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44086/#review120911
-----------------------------------------------------------


On Feb. 26, 2016, 5:07 p.m., Andrew Onischuk wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44086/
> -----------------------------------------------------------
> 
> (Updated Feb. 26, 2016, 5:07 p.m.)
> 
> 
> Review request for Ambari and Myroslav Papirkovskyy.
> 
> 
> Bugs: AMBARI-15205
>     https://issues.apache.org/jira/browse/AMBARI-15205
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> When adding 700 hosts through the UI's add-host-wizard, all hosts successfully
> registered except for 8 of them. These 8 hosts were stuck in RUNNING state and
> would not timeout even after 20 minutes.
> 
> I had to kill the bootstrap process on ambari-server to make the operation to
> complete.  
> On my second attempt I was able to recreate the problem
> 
> Below is the output from `/api/v1/bootstrap/2` for the failing hosts
> (bootstrap response attached)
> 
>     
>     
>     <bootStrapStatus>
>       <status>RUNNING</status>
>       .....
>     
>     
>       <hostsStatus>
>         <hostName>perf-d-511.c.pramod-thangali.internal</hostName>
>         <status>RUNNING</status>
>         <log>
>     ==========================
>     Creating target directory...
>     ==========================
>     
>     Command start time 2016-02-17 18:55:54
>     chmod: cannot access `/var/lib/ambari-agent/data': No such file or directory
>     
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
>     and the key for the corresponding IP address 10.240.5.174
>     has a different value. This could either mean that
>     DNS SPOOFING is happening or the IP address for the host
>     and its host key have changed at the same time.
>     Offending key for IP in /root/.ssh/known_hosts:1418
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
>     Someone could be eavesdropping on you right now (man-in-the-middle attack)!
>     It is also possible that the RSA host key has just been changed.
>     The fingerprint for the RSA key sent by the remote host is
>     66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
>     Please contact your system administrator.
>     Add correct host key in /root/.ssh/known_hosts to get rid of this message.
>     Offending key in /root/.ssh/known_hosts:1417
>     Password authentication is disabled to avoid man-in-the-middle attacks.
>     Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
>     Connection to perf-d-511.c.pramod-thangali.internal closed.
>     SSH command execution finished
>     host=perf-d-511.c.pramod-thangali.internal, exitcode=0
>     Command end time 2016-02-17 18:55:55
>     
>     ==========================
>     Copying common functions script...
>     ==========================
>     
>     Command start time 2016-02-17 18:55:55
>     
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
>     and the key for the corresponding IP address 10.240.5.174
>     has a different value. This could either mean that
>     DNS SPOOFING is happening or the IP address for the host
>     and its host key have changed at the same time.
>     Offending key for IP in /root/.ssh/known_hosts:1418
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
>     Someone could be eavesdropping on you right now (man-in-the-middle attack)!
>     It is also possible that the RSA host key has just been changed.
>     The fingerprint for the RSA key sent by the remote host is
>     66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
>     Please contact your system administrator.
>     Add correct host key in /root/.ssh/known_hosts to get rid of this message.
>     Offending key in /root/.ssh/known_hosts:1417
>     Password authentication is disabled to avoid man-in-the-middle attacks.
>     Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
>     scp /usr/lib/python2.6/site-packages/ambari_commons
>     host=perf-d-511.c.pramod-thangali.internal, exitcode=0
>     Command end time 2016-02-17 18:55:56
>     
>     ==========================
>     Copying OS type check script...
>     ==========================
>     
>     Command start time 2016-02-17 18:55:56
>     
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
>     and the key for the corresponding IP address 10.240.5.174
>     has a different value. This could either mean that
>     DNS SPOOFING is happening or the IP address for the host
>     and its host key have changed at the same time.
>     Offending key for IP in /root/.ssh/known_hosts:1418
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
>     Someone could be eavesdropping on you right now (man-in-the-middle attack)!
>     It is also possible that the RSA host key has just been changed.
>     The fingerprint for the RSA key sent by the remote host is
>     66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
>     Please contact your system administrator.
>     Add correct host key in /root/.ssh/known_hosts to get rid of this message.
>     Offending key in /root/.ssh/known_hosts:1417
>     Password authentication is disabled to avoid man-in-the-middle attacks.
>     Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
>     scp /usr/lib/python2.6/site-packages/ambari_server/os_check_type.py
>     host=perf-d-511.c.pramod-thangali.internal, exitcode=0
>     Command end time 2016-02-17 18:55:56
>     
>     ==========================
>     Running OS type check...
>     ==========================
>     
>     Command start time 2016-02-17 18:55:56
>     Cluster primary/cluster OS family is redhat6 and local/current OS family is redhat6
>     
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
>     and the key for the corresponding IP address 10.240.5.174
>     has a different value. This could either mean that
>     DNS SPOOFING is happening or the IP address for the host
>     and its host key have changed at the same time.
>     Offending key for IP in /root/.ssh/known_hosts:1418
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
>     Someone could be eavesdropping on you right now (man-in-the-middle attack)!
>     It is also possible that the RSA host key has just been changed.
>     The fingerprint for the RSA key sent by the remote host is
>     66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
>     Please contact your system administrator.
>     Add correct host key in /root/.ssh/known_hosts to get rid of this message.
>     Offending key in /root/.ssh/known_hosts:1417
>     Password authentication is disabled to avoid man-in-the-middle attacks.
>     Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
>     Connection to perf-d-511.c.pramod-thangali.internal closed.
>     SSH command execution finished
>     host=perf-d-511.c.pramod-thangali.internal, exitcode=0
>     Command end time 2016-02-17 18:55:57
>     
>     ==========================
>     Checking 'sudo' package on remote host...
>     ==========================
>     
>     Command start time 2016-02-17 18:55:57
>     sudo-1.8.6p3-20.el6_7.x86_64
>     
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
>     and the key for the corresponding IP address 10.240.5.174
>     has a different value. This could either mean that
>     DNS SPOOFING is happening or the IP address for the host
>     and its host key have changed at the same time.
>     Offending key for IP in /root/.ssh/known_hosts:1418
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
>     Someone could be eavesdropping on you right now (man-in-the-middle attack)!
>     It is also possible that the RSA host key has just been changed.
>     The fingerprint for the RSA key sent by the remote host is
>     66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
>     Please contact your system administrator.
>     Add correct host key in /root/.ssh/known_hosts to get rid of this message.
>     Offending key in /root/.ssh/known_hosts:1417
>     Password authentication is disabled to avoid man-in-the-middle attacks.
>     Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
>     Connection to perf-d-511.c.pramod-thangali.internal closed.
>     SSH command execution finished
>     host=perf-d-511.c.pramod-thangali.internal, exitcode=0
>     Command end time 2016-02-17 18:55:58
>     
>     ==========================
>     Copying repo file to 'tmp' folder...
>     ==========================
>     
>     Command start time 2016-02-17 18:55:58
>     
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
>     and the key for the corresponding IP address 10.240.5.174
>     has a different value. This could either mean that
>     DNS SPOOFING is happening or the IP address for the host
>     and its host key have changed at the same time.
>     Offending key for IP in /root/.ssh/known_hosts:1418
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
>     Someone could be eavesdropping on you right now (man-in-the-middle attack)!
>     It is also possible that the RSA host key has just been changed.
>     The fingerprint for the RSA key sent by the remote host is
>     66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
>     Please contact your system administrator.
>     Add correct host key in /root/.ssh/known_hosts to get rid of this message.
>     Offending key in /root/.ssh/known_hosts:1417
>     Password authentication is disabled to avoid man-in-the-middle attacks.
>     Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
>     scp /etc/yum.repos.d/ambari.repo
>     host=perf-d-511.c.pramod-thangali.internal, exitcode=0
>     Command end time 2016-02-17 18:55:59
>     
>     ==========================
>     Moving file to repo dir...
>     ==========================
>     
>     Command start time 2016-02-17 18:55:59
>     
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
>     and the key for the corresponding IP address 10.240.5.174
>     has a different value. This could either mean that
>     DNS SPOOFING is happening or the IP address for the host
>     and its host key have changed at the same time.
>     Offending key for IP in /root/.ssh/known_hosts:1418
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
>     Someone could be eavesdropping on you right now (man-in-the-middle attack)!
>     It is also possible that the RSA host key has just been changed.
>     The fingerprint for the RSA key sent by the remote host is
>     66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
>     Please contact your system administrator.
>     Add correct host key in /root/.ssh/known_hosts to get rid of this message.
>     Offending key in /root/.ssh/known_hosts:1417
>     Password authentication is disabled to avoid man-in-the-middle attacks.
>     Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
>     Connection to perf-d-511.c.pramod-thangali.internal closed.
>     SSH command execution finished
>     host=perf-d-511.c.pramod-thangali.internal, exitcode=0
>     Command end time 2016-02-17 18:55:59
>     
>     ==========================
>     Changing permissions for ambari.repo...
>     ==========================
>     
>     Command start time 2016-02-17 18:55:59
>     
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
>     and the key for the corresponding IP address 10.240.5.174
>     has a different value. This could either mean that
>     DNS SPOOFING is happening or the IP address for the host
>     and its host key have changed at the same time.
>     Offending key for IP in /root/.ssh/known_hosts:1418
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
>     Someone could be eavesdropping on you right now (man-in-the-middle attack)!
>     It is also possible that the RSA host key has just been changed.
>     The fingerprint for the RSA key sent by the remote host is
>     66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
>     Please contact your system administrator.
>     Add correct host key in /root/.ssh/known_hosts to get rid of this message.
>     Offending key in /root/.ssh/known_hosts:1417
>     Password authentication is disabled to avoid man-in-the-middle attacks.
>     Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
>     Connection to perf-d-511.c.pramod-thangali.internal closed.
>     SSH command execution finished
>     host=perf-d-511.c.pramod-thangali.internal, exitcode=0
>     Command end time 2016-02-17 18:56:00
>     
>     ==========================
>     Copying setup script file...
>     ==========================
>     
>     Command start time 2016-02-17 18:56:00
>     
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
>     and the key for the corresponding IP address 10.240.5.174
>     has a different value. This could either mean that
>     DNS SPOOFING is happening or the IP address for the host
>     and its host key have changed at the same time.
>     Offending key for IP in /root/.ssh/known_hosts:1418
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
>     Someone could be eavesdropping on you right now (man-in-the-middle attack)!
>     It is also possible that the RSA host key has just been changed.
>     The fingerprint for the RSA key sent by the remote host is
>     66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
>     Please contact your system administrator.
>     Add correct host key in /root/.ssh/known_hosts to get rid of this message.
>     Offending key in /root/.ssh/known_hosts:1417
>     Password authentication is disabled to avoid man-in-the-middle attacks.
>     Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
>     scp /usr/lib/python2.6/site-packages/ambari_server/setupAgent.py
>     host=perf-d-511.c.pramod-thangali.internal, exitcode=0
>     Command end time 2016-02-17 18:56:00
>     
>     ==========================
>     Running setup agent script...
>     ==========================
>     
>     Command start time 2016-02-17 18:56:00
>     Repository base is listed more than once in the configuration
>     Repository updates is listed more than once in the configuration
>     Repository extras is listed more than once in the configuration
>     Error: Cannot find a valid baseurl for repo: base
>     Repository base is listed more than once in the configuration
>     Repository updates is listed more than once in the configuration
>     Repository extras is listed more than once in the configuration
>     Error: Cannot find a valid baseurl for repo: base
>     ('', None)
>     
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
>     and the key for the corresponding IP address 10.240.5.174
>     has a different value. This could either mean that
>     DNS SPOOFING is happening or the IP address for the host
>     and its host key have changed at the same time.
>     Offending key for IP in /root/.ssh/known_hosts:1418
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
>     Someone could be eavesdropping on you right now (man-in-the-middle attack)!
>     It is also possible that the RSA host key has just been changed.
>     The fingerprint for the RSA key sent by the remote host is
>     66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
>     Please contact your system administrator.
>     Add correct host key in /root/.ssh/known_hosts to get rid of this message.
>     Offending key in /root/.ssh/known_hosts:1417
>     Password authentication is disabled to avoid man-in-the-middle attacks.
>     Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
>     Connection to perf-d-511.c.pramod-thangali.internal closed.
>     SSH command execution finished
>     host=perf-d-511.c.pramod-thangali.internal, exitcode=1
>     Command end time 2016-02-17 18:56:47
>     
>     ERROR: Bootstrap of host perf-d-511.c.pramod-thangali.internal fails because previous action finished with non-zero exit code (1)
>     ERROR MESSAGE: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
>     and the key for the corresponding IP address 10.240.5.174
>     has a different value. This could either mean that
>     DNS SPOOFING is happening or the IP address for the host
>     and its host key have changed at the same time.
>     Offending key for IP in /root/.ssh/known_hosts:1418
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
>     Someone could be eavesdropping on you right now (man-in-the-middle attack)!
>     It is also possible that the RSA host key has just been changed.
>     The fingerprint for the RSA key sent by the remote host is
>     66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
>     Please contact your system administrator.
>     Add correct host key in /root/.ssh/known_hosts to get rid of this message.
>     Offending key in /root/.ssh/known_hosts:1417
>     Password authentication is disabled to avoid man-in-the-middle attacks.
>     Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
>     Connection to perf-d-511.c.pramod-thangali.internal closed.
>     
>     STDOUT: Repository base is listed more than once in the configuration
>     Repository updates is listed more than once in the configuration
>     Repository extras is listed more than once in the configuration
>     Error: Cannot find a valid baseurl for repo: base
>     Repository base is listed more than once in the configuration
>     Repository updates is listed more than once in the configuration
>     Repository extras is listed more than once in the configuration
>     Error: Cannot find a valid baseurl for repo: base
>     ('', None)
>     
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
>     and the key for the corresponding IP address 10.240.5.174
>     has a different value. This could either mean that
>     DNS SPOOFING is happening or the IP address for the host
>     and its host key have changed at the same time.
>     Offending key for IP in /root/.ssh/known_hosts:1418
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
>     Someone could be eavesdropping on you right now (man-in-the-middle attack)!
>     It is also possible that the RSA host key has just been changed.
>     The fingerprint for the RSA key sent by the remote host is
>     66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
>     Please contact your system administrator.
>     Add correct host key in /root/.ssh/known_hosts to get rid of this message.
>     Offending key in /root/.ssh/known_hosts:1417
>     Password authentication is disabled to avoid man-in-the-middle attacks.
>     Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
>     Connection to perf-d-511.c.pramod-thangali.internal closed.
>       </log>
>     </hostsStatus>
>     
>     
>     ...
> 
> 
> Diffs
> -----
> 
>   ambari-server/src/main/java/org/apache/ambari/server/bootstrap/BSRunner.java bf43bc9 
> 
> Diff: https://reviews.apache.org/r/44086/diff/
> 
> 
> Testing
> -------
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Andrew Onischuk
> 
>

Re: Review Request 44086: Registering 700 hosts gets stuck without timeout on 8 hosts

Posted by Alejandro Fernandez <af...@hortonworks.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44086/#review120911
-----------------------------------------------------------




ambari-server/src/main/java/org/apache/ambari/server/bootstrap/BSRunner.java (line 403)
<https://reviews.apache.org/r/44086/#comment182438>

    Under what conditions would the file not exist?


- Alejandro Fernandez


On Feb. 26, 2016, 5:07 p.m., Andrew Onischuk wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44086/
> -----------------------------------------------------------
> 
> (Updated Feb. 26, 2016, 5:07 p.m.)
> 
> 
> Review request for Ambari and Myroslav Papirkovskyy.
> 
> 
> Bugs: AMBARI-15205
>     https://issues.apache.org/jira/browse/AMBARI-15205
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> When adding 700 hosts through the UI's add-host-wizard, all hosts successfully
> registered except for 8 of them. These 8 hosts were stuck in RUNNING state and
> would not timeout even after 20 minutes.
> 
> I had to kill the bootstrap process on ambari-server to make the operation to
> complete.  
> On my second attempt I was able to recreate the problem
> 
> Below is the output from `/api/v1/bootstrap/2` for the failing hosts
> (bootstrap response attached)
> 
>     
>     
>     <bootStrapStatus>
>       <status>RUNNING</status>
>       .....
>     
>     
>       <hostsStatus>
>         <hostName>perf-d-511.c.pramod-thangali.internal</hostName>
>         <status>RUNNING</status>
>         <log>
>     ==========================
>     Creating target directory...
>     ==========================
>     
>     Command start time 2016-02-17 18:55:54
>     chmod: cannot access `/var/lib/ambari-agent/data': No such file or directory
>     
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
>     and the key for the corresponding IP address 10.240.5.174
>     has a different value. This could either mean that
>     DNS SPOOFING is happening or the IP address for the host
>     and its host key have changed at the same time.
>     Offending key for IP in /root/.ssh/known_hosts:1418
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
>     Someone could be eavesdropping on you right now (man-in-the-middle attack)!
>     It is also possible that the RSA host key has just been changed.
>     The fingerprint for the RSA key sent by the remote host is
>     66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
>     Please contact your system administrator.
>     Add correct host key in /root/.ssh/known_hosts to get rid of this message.
>     Offending key in /root/.ssh/known_hosts:1417
>     Password authentication is disabled to avoid man-in-the-middle attacks.
>     Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
>     Connection to perf-d-511.c.pramod-thangali.internal closed.
>     SSH command execution finished
>     host=perf-d-511.c.pramod-thangali.internal, exitcode=0
>     Command end time 2016-02-17 18:55:55
>     
>     ==========================
>     Copying common functions script...
>     ==========================
>     
>     Command start time 2016-02-17 18:55:55
>     
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
>     and the key for the corresponding IP address 10.240.5.174
>     has a different value. This could either mean that
>     DNS SPOOFING is happening or the IP address for the host
>     and its host key have changed at the same time.
>     Offending key for IP in /root/.ssh/known_hosts:1418
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
>     Someone could be eavesdropping on you right now (man-in-the-middle attack)!
>     It is also possible that the RSA host key has just been changed.
>     The fingerprint for the RSA key sent by the remote host is
>     66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
>     Please contact your system administrator.
>     Add correct host key in /root/.ssh/known_hosts to get rid of this message.
>     Offending key in /root/.ssh/known_hosts:1417
>     Password authentication is disabled to avoid man-in-the-middle attacks.
>     Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
>     scp /usr/lib/python2.6/site-packages/ambari_commons
>     host=perf-d-511.c.pramod-thangali.internal, exitcode=0
>     Command end time 2016-02-17 18:55:56
>     
>     ==========================
>     Copying OS type check script...
>     ==========================
>     
>     Command start time 2016-02-17 18:55:56
>     
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
>     and the key for the corresponding IP address 10.240.5.174
>     has a different value. This could either mean that
>     DNS SPOOFING is happening or the IP address for the host
>     and its host key have changed at the same time.
>     Offending key for IP in /root/.ssh/known_hosts:1418
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
>     Someone could be eavesdropping on you right now (man-in-the-middle attack)!
>     It is also possible that the RSA host key has just been changed.
>     The fingerprint for the RSA key sent by the remote host is
>     66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
>     Please contact your system administrator.
>     Add correct host key in /root/.ssh/known_hosts to get rid of this message.
>     Offending key in /root/.ssh/known_hosts:1417
>     Password authentication is disabled to avoid man-in-the-middle attacks.
>     Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
>     scp /usr/lib/python2.6/site-packages/ambari_server/os_check_type.py
>     host=perf-d-511.c.pramod-thangali.internal, exitcode=0
>     Command end time 2016-02-17 18:55:56
>     
>     ==========================
>     Running OS type check...
>     ==========================
>     
>     Command start time 2016-02-17 18:55:56
>     Cluster primary/cluster OS family is redhat6 and local/current OS family is redhat6
>     
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
>     and the key for the corresponding IP address 10.240.5.174
>     has a different value. This could either mean that
>     DNS SPOOFING is happening or the IP address for the host
>     and its host key have changed at the same time.
>     Offending key for IP in /root/.ssh/known_hosts:1418
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
>     Someone could be eavesdropping on you right now (man-in-the-middle attack)!
>     It is also possible that the RSA host key has just been changed.
>     The fingerprint for the RSA key sent by the remote host is
>     66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
>     Please contact your system administrator.
>     Add correct host key in /root/.ssh/known_hosts to get rid of this message.
>     Offending key in /root/.ssh/known_hosts:1417
>     Password authentication is disabled to avoid man-in-the-middle attacks.
>     Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
>     Connection to perf-d-511.c.pramod-thangali.internal closed.
>     SSH command execution finished
>     host=perf-d-511.c.pramod-thangali.internal, exitcode=0
>     Command end time 2016-02-17 18:55:57
>     
>     ==========================
>     Checking 'sudo' package on remote host...
>     ==========================
>     
>     Command start time 2016-02-17 18:55:57
>     sudo-1.8.6p3-20.el6_7.x86_64
>     
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
>     and the key for the corresponding IP address 10.240.5.174
>     has a different value. This could either mean that
>     DNS SPOOFING is happening or the IP address for the host
>     and its host key have changed at the same time.
>     Offending key for IP in /root/.ssh/known_hosts:1418
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
>     Someone could be eavesdropping on you right now (man-in-the-middle attack)!
>     It is also possible that the RSA host key has just been changed.
>     The fingerprint for the RSA key sent by the remote host is
>     66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
>     Please contact your system administrator.
>     Add correct host key in /root/.ssh/known_hosts to get rid of this message.
>     Offending key in /root/.ssh/known_hosts:1417
>     Password authentication is disabled to avoid man-in-the-middle attacks.
>     Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
>     Connection to perf-d-511.c.pramod-thangali.internal closed.
>     SSH command execution finished
>     host=perf-d-511.c.pramod-thangali.internal, exitcode=0
>     Command end time 2016-02-17 18:55:58
>     
>     ==========================
>     Copying repo file to 'tmp' folder...
>     ==========================
>     
>     Command start time 2016-02-17 18:55:58
>     
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
>     and the key for the corresponding IP address 10.240.5.174
>     has a different value. This could either mean that
>     DNS SPOOFING is happening or the IP address for the host
>     and its host key have changed at the same time.
>     Offending key for IP in /root/.ssh/known_hosts:1418
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
>     Someone could be eavesdropping on you right now (man-in-the-middle attack)!
>     It is also possible that the RSA host key has just been changed.
>     The fingerprint for the RSA key sent by the remote host is
>     66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
>     Please contact your system administrator.
>     Add correct host key in /root/.ssh/known_hosts to get rid of this message.
>     Offending key in /root/.ssh/known_hosts:1417
>     Password authentication is disabled to avoid man-in-the-middle attacks.
>     Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
>     scp /etc/yum.repos.d/ambari.repo
>     host=perf-d-511.c.pramod-thangali.internal, exitcode=0
>     Command end time 2016-02-17 18:55:59
>     
>     ==========================
>     Moving file to repo dir...
>     ==========================
>     
>     Command start time 2016-02-17 18:55:59
>     
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
>     and the key for the corresponding IP address 10.240.5.174
>     has a different value. This could either mean that
>     DNS SPOOFING is happening or the IP address for the host
>     and its host key have changed at the same time.
>     Offending key for IP in /root/.ssh/known_hosts:1418
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
>     Someone could be eavesdropping on you right now (man-in-the-middle attack)!
>     It is also possible that the RSA host key has just been changed.
>     The fingerprint for the RSA key sent by the remote host is
>     66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
>     Please contact your system administrator.
>     Add correct host key in /root/.ssh/known_hosts to get rid of this message.
>     Offending key in /root/.ssh/known_hosts:1417
>     Password authentication is disabled to avoid man-in-the-middle attacks.
>     Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
>     Connection to perf-d-511.c.pramod-thangali.internal closed.
>     SSH command execution finished
>     host=perf-d-511.c.pramod-thangali.internal, exitcode=0
>     Command end time 2016-02-17 18:55:59
>     
>     ==========================
>     Changing permissions for ambari.repo...
>     ==========================
>     
>     Command start time 2016-02-17 18:55:59
>     
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
>     and the key for the corresponding IP address 10.240.5.174
>     has a different value. This could either mean that
>     DNS SPOOFING is happening or the IP address for the host
>     and its host key have changed at the same time.
>     Offending key for IP in /root/.ssh/known_hosts:1418
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
>     Someone could be eavesdropping on you right now (man-in-the-middle attack)!
>     It is also possible that the RSA host key has just been changed.
>     The fingerprint for the RSA key sent by the remote host is
>     66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
>     Please contact your system administrator.
>     Add correct host key in /root/.ssh/known_hosts to get rid of this message.
>     Offending key in /root/.ssh/known_hosts:1417
>     Password authentication is disabled to avoid man-in-the-middle attacks.
>     Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
>     Connection to perf-d-511.c.pramod-thangali.internal closed.
>     SSH command execution finished
>     host=perf-d-511.c.pramod-thangali.internal, exitcode=0
>     Command end time 2016-02-17 18:56:00
>     
>     ==========================
>     Copying setup script file...
>     ==========================
>     
>     Command start time 2016-02-17 18:56:00
>     
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
>     and the key for the corresponding IP address 10.240.5.174
>     has a different value. This could either mean that
>     DNS SPOOFING is happening or the IP address for the host
>     and its host key have changed at the same time.
>     Offending key for IP in /root/.ssh/known_hosts:1418
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
>     Someone could be eavesdropping on you right now (man-in-the-middle attack)!
>     It is also possible that the RSA host key has just been changed.
>     The fingerprint for the RSA key sent by the remote host is
>     66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
>     Please contact your system administrator.
>     Add correct host key in /root/.ssh/known_hosts to get rid of this message.
>     Offending key in /root/.ssh/known_hosts:1417
>     Password authentication is disabled to avoid man-in-the-middle attacks.
>     Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
>     scp /usr/lib/python2.6/site-packages/ambari_server/setupAgent.py
>     host=perf-d-511.c.pramod-thangali.internal, exitcode=0
>     Command end time 2016-02-17 18:56:00
>     
>     ==========================
>     Running setup agent script...
>     ==========================
>     
>     Command start time 2016-02-17 18:56:00
>     Repository base is listed more than once in the configuration
>     Repository updates is listed more than once in the configuration
>     Repository extras is listed more than once in the configuration
>     Error: Cannot find a valid baseurl for repo: base
>     Repository base is listed more than once in the configuration
>     Repository updates is listed more than once in the configuration
>     Repository extras is listed more than once in the configuration
>     Error: Cannot find a valid baseurl for repo: base
>     ('', None)
>     
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
>     and the key for the corresponding IP address 10.240.5.174
>     has a different value. This could either mean that
>     DNS SPOOFING is happening or the IP address for the host
>     and its host key have changed at the same time.
>     Offending key for IP in /root/.ssh/known_hosts:1418
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
>     Someone could be eavesdropping on you right now (man-in-the-middle attack)!
>     It is also possible that the RSA host key has just been changed.
>     The fingerprint for the RSA key sent by the remote host is
>     66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
>     Please contact your system administrator.
>     Add correct host key in /root/.ssh/known_hosts to get rid of this message.
>     Offending key in /root/.ssh/known_hosts:1417
>     Password authentication is disabled to avoid man-in-the-middle attacks.
>     Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
>     Connection to perf-d-511.c.pramod-thangali.internal closed.
>     SSH command execution finished
>     host=perf-d-511.c.pramod-thangali.internal, exitcode=1
>     Command end time 2016-02-17 18:56:47
>     
>     ERROR: Bootstrap of host perf-d-511.c.pramod-thangali.internal fails because previous action finished with non-zero exit code (1)
>     ERROR MESSAGE: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
>     and the key for the corresponding IP address 10.240.5.174
>     has a different value. This could either mean that
>     DNS SPOOFING is happening or the IP address for the host
>     and its host key have changed at the same time.
>     Offending key for IP in /root/.ssh/known_hosts:1418
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
>     Someone could be eavesdropping on you right now (man-in-the-middle attack)!
>     It is also possible that the RSA host key has just been changed.
>     The fingerprint for the RSA key sent by the remote host is
>     66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
>     Please contact your system administrator.
>     Add correct host key in /root/.ssh/known_hosts to get rid of this message.
>     Offending key in /root/.ssh/known_hosts:1417
>     Password authentication is disabled to avoid man-in-the-middle attacks.
>     Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
>     Connection to perf-d-511.c.pramod-thangali.internal closed.
>     
>     STDOUT: Repository base is listed more than once in the configuration
>     Repository updates is listed more than once in the configuration
>     Repository extras is listed more than once in the configuration
>     Error: Cannot find a valid baseurl for repo: base
>     Repository base is listed more than once in the configuration
>     Repository updates is listed more than once in the configuration
>     Repository extras is listed more than once in the configuration
>     Error: Cannot find a valid baseurl for repo: base
>     ('', None)
>     
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     The RSA host key for perf-d-511.c.pramod-thangali.internal has changed,
>     and the key for the corresponding IP address 10.240.5.174
>     has a different value. This could either mean that
>     DNS SPOOFING is happening or the IP address for the host
>     and its host key have changed at the same time.
>     Offending key for IP in /root/.ssh/known_hosts:1418
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
>     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>     IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
>     Someone could be eavesdropping on you right now (man-in-the-middle attack)!
>     It is also possible that the RSA host key has just been changed.
>     The fingerprint for the RSA key sent by the remote host is
>     66:45:6d:a4:fe:13:5d:47:ee:df:45:cc:32:dc:6a:2f.
>     Please contact your system administrator.
>     Add correct host key in /root/.ssh/known_hosts to get rid of this message.
>     Offending key in /root/.ssh/known_hosts:1417
>     Password authentication is disabled to avoid man-in-the-middle attacks.
>     Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
>     Connection to perf-d-511.c.pramod-thangali.internal closed.
>       </log>
>     </hostsStatus>
>     
>     
>     ...
> 
> 
> Diffs
> -----
> 
>   ambari-server/src/main/java/org/apache/ambari/server/bootstrap/BSRunner.java bf43bc9 
> 
> Diff: https://reviews.apache.org/r/44086/diff/
> 
> 
> Testing
> -------
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Andrew Onischuk
> 
>