You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@vcl.apache.org by "Litchfield, Elwin L." <Le...@sar.usf.edu> on 2015/05/17 15:09:04 UTC

Fw: PROBLEM -- localhost|44:44|timeout|Linux.pm|VM7Cent6VCL1>VM7|vmwarelinux-vm7c2testvcl212-v0|admin

The VM was powered on as scheduled, I logged on to the VM using admin & PW supplied.  Logged off & the VM stayed powered on after the reservation expired.  Ping to the VM works, ICMP Type 0 Code 0.   ICMP Type 3 Code 10 to Port 22 reply is Host Administratively prohibited.  

>From the VM console, on the VMware host,  no administrator user is able to logon.  I have added 2 more admin users & they will not work either.  The admin logon supplied from the reservation worked, but ssh was not active/displayed as a service.

I have a Wireshark 1.8.10 trace of the private network conversation.  I am researching how to upgrade to the current release of Wireshark, but I have not found any information that Wireshark will decode SSH.  

Is there any trace to run to determine what is taking place between the Management Node & the VM?

Thanks    

Lewis Litchfield
Technology Services – Viking Center Lab
University of South Florida Sarasota-Manatee
941-359-4231

________________________________________
From: root@localhost.localdomain <ro...@localhost.localdomain>
Sent: Sunday, May 17, 2015 8:35 AM
To: lewis15@mail.usf.edu
Subject: PROBLEM -- localhost|44:44|timeout|Linux.pm|VM7Cent6VCL1>VM7|vmwarelinux-vm7c2testvcl212-v0|admin

failed to add AllowUsers admin to external_sshd_config
------------------------------------------------------------------------
time: 2015-05-17 08:35:52
caller: Linux.pm:delete_user(894)
( 0) Linux.pm, delete_user (line: 894)
(-1) Linux.pm, sanitize (line: 1177)
(-2) reclaim.pm, call_os_sanitize (line: 271)
(-3) reclaim.pm, process (line: 162)
(-4) vcld, make_new_child (line: 571)
(-5) vcld, main (line: 350)
------------------------------------------------------------------------
management node: localhost
reservation PID: 22905
parent vcld PID: 26123

request ID: 44
reservation ID: 44
request state/laststate: timeout/reserved
request start time: 2015-05-17 08:15:00
request end time: 2015-05-17 09:00:00
for imaging: no
log ID: 51

computer: VM7Cent6VCL1
computer id: 6
computer type: virtualmachine
computer eth0 MAC address: 00:50:56:3F:40:E3
computer eth1 MAC address: 00:50:56:3F:40:ED
computer private IP address: 192.168.10.48
computer public IP address: 131.247.29.132
computer in block allocation: no
provisioning module: VCL::Module::Provisioning::VMware::VMware

vm host: VM7
vm host ID: 2
vm host computer ID: 5
vm profile: VMware ESXi - local storage
vm profile VM path: datastore1
vm profile repository path: <undefined>
vm profile datastore path: datastore1
vm profile disk type: shared

image: vmwarelinux-vm7c2testvcl212-v0
image display name: vm7c2testvcl2
image ID: 12
image revision ID: 12
image size: 5762 MB
use Sysprep: yes
root access: yes
image owner ID: 1
image owner affiliation: Local
image revision date created: 2015-05-01 09:43:57
image revision production: yes
OS module: VCL::Module::OS::Linux

user: admin
user name: vcl admin
user ID: 1
user affiliation: Local
------------------------------------------------------------------------
RECENT LOG ENTRIES FOR THIS PROCESS:
|22905|44:44|timeout| (-1) Linux.pm, delete_user (line: 871)
|22905|44:44|timeout| (-2) Linux.pm, sanitize (line: 1177)
|22905|44:44|timeout| (-3) reclaim.pm, call_os_sanitize (line: 271)
|22905|44:44|timeout| (-4) reclaim.pm, process (line: 162)
|22905|44:44|timeout| (-5) vcld, make_new_child (line: 571)
2015-05-17 08:35:40|22905|44:44|timeout|utils.pm:run_ssh_command(4902)|executing SSH command on VM7Cent6VCL1:
|22905|44:44|timeout| /usr/bin/ssh -i /etc/vcl/vcl.key  -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ConnectionAttempts=1 -o ConnectTimeout=3 -l root -p 22 -x VM7Cent6VCL1 '/usr/sbin/userdel admin' 2>&1
|22905|44:44|timeout| ---- WARNING ----
|22905|44:44|timeout| 2015-05-17 08:35:40|22905|44:44|timeout|utils.pm:run_ssh_command(5006)|attempt 1/3: failed to execute SSH command on VM7Cent6VCL1: '/usr/sbin/userdel admin', exit status: 255, output:
|22905|44:44|timeout| ssh output (/usr/sbin/...): ssh: connect to host VM7Cent6VCL1 port 22: No route to host
|22905|44:44|timeout| ( 0) utils.pm, run_ssh_command (line: 5006)
|22905|44:44|timeout| (-1) Linux.pm, delete_user (line: 881)
|22905|44:44|timeout| (-2) Linux.pm, sanitize (line: 1177)
|22905|44:44|timeout| (-3) reclaim.pm, call_os_sanitize (line: 271)
|22905|44:44|timeout| (-4) reclaim.pm, process (line: 162)
|22905|44:44|timeout| (-5) vcld, make_new_child (line: 571)
2015-05-17 08:35:40|22905|44:44|timeout|utils.pm:run_ssh_command(4894)|sleeping for 2 seconds before making next SSH attempt
2015-05-17 08:35:42|22905|44:44|timeout|utils.pm:run_ssh_command(4906)|attempt 2/3: executing SSH command on VM7Cent6VCL1:
|22905|44:44|timeout| /usr/bin/ssh -i /etc/vcl/vcl.key  -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ConnectionAttempts=1 -o ConnectTimeout=3 -l root -p 22 -x VM7Cent6VCL1 '/usr/sbin/userdel admin' 2>&1
|22905|44:44|timeout| ---- WARNING ----
|22905|44:44|timeout| 2015-05-17 08:35:42|22905|44:44|timeout|utils.pm:run_ssh_command(5006)|attempt 2/3: failed to execute SSH command on VM7Cent6VCL1: '/usr/sbin/userdel admin', exit status: 255, output:
|22905|44:44|timeout| ssh output (/usr/sbin/...): ssh: connect to host VM7Cent6VCL1 port 22: No route to host
|22905|44:44|timeout| ( 0) utils.pm, run_ssh_command (line: 5006)
|22905|44:44|timeout| (-1) Linux.pm, delete_user (line: 881)
|22905|44:44|timeout| (-2) Linux.pm, sanitize (line: 1177)
|22905|44:44|timeout| (-3) reclaim.pm, call_os_sanitize (line: 271)
|22905|44:44|timeout| (-4) reclaim.pm, process (line: 162)
|22905|44:44|timeout| (-5) vcld, make_new_child (line: 571)
2015-05-17 08:35:42|22905|44:44|timeout|utils.pm:run_ssh_command(4894)|sleeping for 4 seconds before making next SSH attempt
2015-05-17 08:35:46|22905|44:44|timeout|utils.pm:run_ssh_command(4906)|attempt 3/3: executing SSH command on VM7Cent6VCL1:
|22905|44:44|timeout| /usr/bin/ssh -i /etc/vcl/vcl.key  -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ConnectionAttempts=1 -o ConnectTimeout=3 -l root -p 22 -x VM7Cent6VCL1 '/usr/sbin/userdel admin' 2>&1
|22905|44:44|timeout| ---- WARNING ----
|22905|44:44|timeout| 2015-05-17 08:35:46|22905|44:44|timeout|utils.pm:run_ssh_command(5006)|attempt 3/3: failed to execute SSH command on VM7Cent6VCL1: '/usr/sbin/userdel admin', exit status: 255, output:
|22905|44:44|timeout| ssh output (/usr/sbin/...): ssh: connect to host VM7Cent6VCL1 port 22: No route to host
|22905|44:44|timeout| ( 0) utils.pm, run_ssh_command (line: 5006)
|22905|44:44|timeout| (-1) Linux.pm, delete_user (line: 881)
|22905|44:44|timeout| (-2) Linux.pm, sanitize (line: 1177)
|22905|44:44|timeout| (-3) reclaim.pm, call_os_sanitize (line: 271)
|22905|44:44|timeout| (-4) reclaim.pm, process (line: 162)
|22905|44:44|timeout| (-5) vcld, make_new_child (line: 571)
|22905|44:44|timeout| ---- WARNING ----
|22905|44:44|timeout| 2015-05-17 08:35:46|22905|44:44|timeout|utils.pm:run_ssh_command(5043)|failed to run SSH command after 3 attempts, command: /usr/bin/ssh -i /etc/vcl/vcl.key  -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ConnectionAttempts=1 -o ConnectTimeout=3 -l root -p 22 -x VM7Cent6VCL1 '/usr/sbin/userdel admin' 2>&1, exit status: 255, output:
|22905|44:44|timeout| ssh output (/usr/sbin/...): ssh: connect to host VM7Cent6VCL1 port 22: No route to host
|22905|44:44|timeout| ( 0) utils.pm, run_ssh_command (line: 5043)
|22905|44:44|timeout| (-1) Linux.pm, delete_user (line: 881)
|22905|44:44|timeout| (-2) Linux.pm, sanitize (line: 1177)
|22905|44:44|timeout| (-3) reclaim.pm, call_os_sanitize (line: 271)
|22905|44:44|timeout| (-4) reclaim.pm, process (line: 162)
|22905|44:44|timeout| (-5) vcld, make_new_child (line: 571)
2015-05-17 08:35:46|22905|44:44|timeout|utils.pm:run_ssh_command(4902)|executing SSH command on VM7Cent6VCL1:
|22905|44:44|timeout| /usr/bin/ssh -i /etc/vcl/vcl.key  -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ConnectionAttempts=1 -o ConnectTimeout=3 -l root -p 22 -x VM7Cent6VCL1 'sed -i -e "/^AllowUsers .*/d" /etc/ssh/external_sshd_config' 2>&1
|22905|44:44|timeout| ---- WARNING ----
|22905|44:44|timeout| 2015-05-17 08:35:46|22905|44:44|timeout|utils.pm:run_ssh_command(5006)|attempt 1/3: failed to execute SSH command on VM7Cent6VCL1: 'sed -i -e "/^AllowUsers .*/d" /etc/ssh/external_sshd_config', exit status: 255, output:
|22905|44:44|timeout| ssh output (sed -i -e ...): ssh: connect to host VM7Cent6VCL1 port 22: No route to host
|22905|44:44|timeout| ( 0) utils.pm, run_ssh_command (line: 5006)
|22905|44:44|timeout| (-1) Linux.pm, delete_user (line: 890)
|22905|44:44|timeout| (-2) Linux.pm, sanitize (line: 1177)
|22905|44:44|timeout| (-3) reclaim.pm, call_os_sanitize (line: 271)
|22905|44:44|timeout| (-4) reclaim.pm, process (line: 162)
|22905|44:44|timeout| (-5) vcld, make_new_child (line: 571)
2015-05-17 08:35:46|22905|44:44|timeout|utils.pm:run_ssh_command(4894)|sleeping for 2 seconds before making next SSH attempt
2015-05-17 08:35:48|22905|44:44|timeout|utils.pm:run_ssh_command(4906)|attempt 2/3: executing SSH command on VM7Cent6VCL1:
|22905|44:44|timeout| /usr/bin/ssh -i /etc/vcl/vcl.key  -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ConnectionAttempts=1 -o ConnectTimeout=3 -l root -p 22 -x VM7Cent6VCL1 'sed -i -e "/^AllowUsers .*/d" /etc/ssh/external_sshd_config' 2>&1
|22905|44:44|timeout| ---- WARNING ----
|22905|44:44|timeout| 2015-05-17 08:35:48|22905|44:44|timeout|utils.pm:run_ssh_command(5006)|attempt 2/3: failed to execute SSH command on VM7Cent6VCL1: 'sed -i -e "/^AllowUsers .*/d" /etc/ssh/external_sshd_config', exit status: 255, output:
|22905|44:44|timeout| ssh output (sed -i -e ...): ssh: connect to host VM7Cent6VCL1 port 22: No route to host
|22905|44:44|timeout| ( 0) utils.pm, run_ssh_command (line: 5006)
|22905|44:44|timeout| (-1) Linux.pm, delete_user (line: 890)
|22905|44:44|timeout| (-2) Linux.pm, sanitize (line: 1177)
|22905|44:44|timeout| (-3) reclaim.pm, call_os_sanitize (line: 271)
|22905|44:44|timeout| (-4) reclaim.pm, process (line: 162)
|22905|44:44|timeout| (-5) vcld, make_new_child (line: 571)
2015-05-17 08:35:48|22905|44:44|timeout|utils.pm:run_ssh_command(4894)|sleeping for 4 seconds before making next SSH attempt
2015-05-17 08:35:52|22905|44:44|timeout|utils.pm:run_ssh_command(4906)|attempt 3/3: executing SSH command on VM7Cent6VCL1:
|22905|44:44|timeout| /usr/bin/ssh -i /etc/vcl/vcl.key  -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ConnectionAttempts=1 -o ConnectTimeout=3 -l root -p 22 -x VM7Cent6VCL1 'sed -i -e "/^AllowUsers .*/d" /etc/ssh/external_sshd_config' 2>&1
|22905|44:44|timeout| ---- WARNING ----
|22905|44:44|timeout| 2015-05-17 08:35:52|22905|44:44|timeout|utils.pm:run_ssh_command(5006)|attempt 3/3: failed to execute SSH command on VM7Cent6VCL1: 'sed -i -e "/^AllowUsers .*/d" /etc/ssh/external_sshd_config', exit status: 255, output:
|22905|44:44|timeout| ssh output (sed -i -e ...): ssh: connect to host VM7Cent6VCL1 port 22: No route to host
|22905|44:44|timeout| ( 0) utils.pm, run_ssh_command (line: 5006)
|22905|44:44|timeout| (-1) Linux.pm, delete_user (line: 890)
|22905|44:44|timeout| (-2) Linux.pm, sanitize (line: 1177)
|22905|44:44|timeout| (-3) reclaim.pm, call_os_sanitize (line: 271)
|22905|44:44|timeout| (-4) reclaim.pm, process (line: 162)
|22905|44:44|timeout| (-5) vcld, make_new_child (line: 571)
|22905|44:44|timeout| ---- WARNING ----
|22905|44:44|timeout| 2015-05-17 08:35:52|22905|44:44|timeout|utils.pm:run_ssh_command(5043)|failed to run SSH command after 3 attempts, command: /usr/bin/ssh -i /etc/vcl/vcl.key  -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ConnectionAttempts=1 -o ConnectTimeout=3 -l root -p 22 -x VM7Cent6VCL1 'sed -i -e "/^AllowUsers .*/d" /etc/ssh/external_sshd_config' 2>&1, exit status: 255, output:
|22905|44:44|timeout| ssh output (sed -i -e ...): ssh: connect to host VM7Cent6VCL1 port 22: No route to host
|22905|44:44|timeout| ( 0) utils.pm, run_ssh_command (line: 5043)
|22905|44:44|timeout| (-1) Linux.pm, delete_user (line: 890)
|22905|44:44|timeout| (-2) Linux.pm, sanitize (line: 1177)
|22905|44:44|timeout| (-3) reclaim.pm, call_os_sanitize (line: 271)
|22905|44:44|timeout| (-4) reclaim.pm, process (line: 162)
|22905|44:44|timeout| (-5) vcld, make_new_child (line: 571)
2015-05-17 08:35:52|22905|44:44|timeout|DataStructure.pm:get_computer_private_ip_address(1641)|attempting to retrieve private IP address for computer: VM7Cent6VCL1
2015-05-17 08:35:52|22905|44:44|timeout|OS.pm:get_file_contents(1897)|retrieved 15 lines from file on localhost: '/etc/hosts'
2015-05-17 08:35:52|22905|44:44|timeout|DataStructure.pm:get_computer_private_ip_address(1697)|returning IP address from /etc/hosts file: 192.168.10.48
2015-05-17 08:35:52|22905|44:44|timeout|utils.pm:is_inblockrequest(5760)|zero rows were returned from database select
2015-05-17 08:35:52|22905|44:44|timeout|DataStructure.pm:get_image_affiliation_name(2121)|image owner id: 1
2015-05-17 08:35:52|22905|44:44|timeout|DataStructure.pm:retrieve_user_data(1412)|attempting to retrieve and store data for user: user.id = '1'
2015-05-17 08:35:52|22905|44:44|timeout|DataStructure.pm:retrieve_user_data(1475)|data has been retrieved for user: admin (id: 1)

Re: Fw: PROBLEM -- localhost|44:44|timeout|Linux.pm|VM7Cent6VCL1>VM7|vmwarelinux-vm7c2testvcl212-v0|admin

Posted by Elwin Litchfield <le...@mail.usf.edu>.
The email was directed to my spam folder.  It looks like 2.4.2 has the
power feature I am looking for.   I am slated for some new servers soon.
The first project will be 2.4.2 when they arrive.

Thanks.

Lewis

On Tue, May 19, 2015 at 8:51 AM, Aaron Peeler <aa...@ncsu.edu> wrote:

> Hello Lewis,
>
> Sorry we don't have a good VCL starter or VCL for dummies sort of
> documentation.
>
> On Tue, May 19, 2015 at 7:44 AM, Elwin Litchfield <le...@mail.usf.edu>
> wrote:
> > Thanks, your explanation clears up a lot of things for me.  I am still
> > looking for some  good documentation/book that explains Apache VCL from
> > beginning to end, IE Apache VCL for Dummies.
> >
> > I will setup some more test & examine the vcld.log file more closely, it
> is
> > Greek to me.  But the more I dig into Apache VCL the clearer things
> become.
> >
> > It looks like I need to rethink my plans about how to use Apache VCL.  I
> was
> > under the impression that the virtual machine would be powered on for the
> > reservation & powered off after the reservation was over.  If I
> understand
> > correctly, using 2.3.2, a user making a reservation may or may not
> receive
> > the same virtual machine each time, but the functionality is the same?
>
>
> In 2.4.2 (the latest release) there is a new reload action which
> powers off the virtual machine(or node) after a reservation.  See this
> link for the details:
>
> https://cwiki.apache.org/confluence/display/VCL/Computer+Predictive+Loading+Modules+and+Power+Off
>
> Correct, the user may or may not get the same machine, but the image
> is generally the same with different IP address and hostname.
>
>
> >
> > Thanks Again.
> >
> > Lewis
> >
> > On Mon, May 18, 2015 at 7:20 AM, Josh Thompson <jo...@ncsu.edu>
> > wrote:
> >>
> >> -----BEGIN PGP SIGNED MESSAGE-----
> >> Hash: SHA1
> >>
> >> Lewis,
> >>
> >> Looking through vcld.log will allow you to see every ssh command that
> vcld
> >> sends to a node.
> >>
> >> Also, let me explain the process VCL uses for provisioning.  We use
> >> something
> >> we call "hot standby" to help minimize the amount of time it takes for
> >> reservations to become ready for users to connect to their reservations.
> >> At
> >> the end of each reservation, the node is reloaded so that it is almost
> >> ready
> >> for another user.  Then, when another user makes a reservation for it,
> >> only a
> >> few steps need to be done to finish provisioning it.  The image selected
> >> to be
> >> reloaded on the node depends on what Predictive Reload module is
> assigned.
> >> Previous to 2.4.2, Predictive Reload modules were assigned per
> management
> >> node, meaning all computers managed by a specific management node would
> be
> >> handled the same way.  With 2.4.2, it is assigned per computer, meaning
> >> each
> >> computer can be handled differently.  Also, 2.4.2 introduced a new
> module
> >> that
> >> doesn't actually reload it, but instead powers it off, and for VMs,
> >> unregisters it.
> >>
> >> All that to say, the behavior you are seeing may be that the node has
> been
> >> reloaded at the end of your reservation to prep for another one.
> >>
> >> One more point to note is that VCL randomizes the root/admin password
> when
> >> it
> >> provisions each node.  You can find the password it selects for each
> >> reservation in the vcld.log file.
> >>
> >> Josh
> >>
> >> On Sun May 17 2015 1:09:04 PM Litchfield, Elwin L. wrote:
> >> > The VM was powered on as scheduled, I logged on to the VM using admin
> &
> >> > PW
> >> > supplied.  Logged off & the VM stayed powered on after the reservation
> >> > expired.  Ping to the VM works, ICMP Type 0 Code 0.   ICMP Type 3 Code
> >> > 10
> >> > to Port 22 reply is Host Administratively prohibited.
> >> >
> >> > From the VM console, on the VMware host,  no administrator user is
> able
> >> > to
> >> > logon.  I have added 2 more admin users & they will not work either.
> >> > The
> >> > admin logon supplied from the reservation worked, but ssh was not
> >> > active/displayed as a service.
> >> >
> >> > I have a Wireshark 1.8.10 trace of the private network conversation.
> I
> >> > am
> >> > researching how to upgrade to the current release of Wireshark, but I
> >> > have
> >> > not found any information that Wireshark will decode SSH.
> >> >
> >> > Is there any trace to run to determine what is taking place between
> the
> >> > Management Node & the VM?
> >> >
> >> > Thanks
> >> >
> >> > Lewis Litchfield
> >> > Technology Services – Viking Center Lab
> >> > University of South Florida Sarasota-Manatee
> >> > 941-359-4231
> >> >
> >> > ________________________________________
> >> > From: root@localhost.localdomain <ro...@localhost.localdomain>
> >> > Sent: Sunday, May 17, 2015 8:35 AM
> >> > To: lewis15@mail.usf.edu
> >> > Subject: PROBLEM --
> >> >
> >> >
> localhost|44:44|timeout|Linux.pm|VM7Cent6VCL1>VM7|vmwarelinux-vm7c2testvcl2
> >> > 12-v0|admin
> >> >
> >> > failed to add AllowUsers admin to external_sshd_config
> >> >
> ------------------------------------------------------------------------
> >> > time: 2015-05-17 08:35:52
> >> > caller: Linux.pm:delete_user(894)
> >> > ( 0) Linux.pm, delete_user (line: 894)
> >> > (-1) Linux.pm, sanitize (line: 1177)
> >> > (-2) reclaim.pm, call_os_sanitize (line: 271)
> >> > (-3) reclaim.pm, process (line: 162)
> >> > (-4) vcld, make_new_child (line: 571)
> >> > (-5) vcld, main (line: 350)
> >> >
> ------------------------------------------------------------------------
> >> > management node: localhost
> >> > reservation PID: 22905
> >> > parent vcld PID: 26123
> >> >
> >> > request ID: 44
> >> > reservation ID: 44
> >> > request state/laststate: timeout/reserved
> >> > request start time: 2015-05-17 08:15:00
> >> > request end time: 2015-05-17 09:00:00
> >> > for imaging: no
> >> > log ID: 51
> >> >
> >> > computer: VM7Cent6VCL1
> >> > computer id: 6
> >> > computer type: virtualmachine
> >> > computer eth0 MAC address: 00:50:56:3F:40:E3
> >> > computer eth1 MAC address: 00:50:56:3F:40:ED
> >> > computer private IP address: 192.168.10.48
> >> > computer public IP address: 131.247.29.132
> >> > computer in block allocation: no
> >> > provisioning module: VCL::Module::Provisioning::VMware::VMware
> >> >
> >> > vm host: VM7
> >> > vm host ID: 2
> >> > vm host computer ID: 5
> >> > vm profile: VMware ESXi - local storage
> >> > vm profile VM path: datastore1
> >> > vm profile repository path: <undefined>
> >> > vm profile datastore path: datastore1
> >> > vm profile disk type: shared
> >> >
> >> > image: vmwarelinux-vm7c2testvcl212-v0
> >> > image display name: vm7c2testvcl2
> >> > image ID: 12
> >> > image revision ID: 12
> >> > image size: 5762 MB
> >> > use Sysprep: yes
> >> > root access: yes
> >> > image owner ID: 1
> >> > image owner affiliation: Local
> >> > image revision date created: 2015-05-01 09:43:57
> >> > image revision production: yes
> >> > OS module: VCL::Module::OS::Linux
> >> >
> >> > user: admin
> >> > user name: vcl admin
> >> > user ID: 1
> >> > user affiliation: Local
> >> >
> ------------------------------------------------------------------------
> >> >
> >> > RECENT LOG ENTRIES FOR THIS PROCESS:
> >> > |22905|44:44|timeout| (-1) Linux.pm, delete_user (line: 871)
> >> > |22905|44:44|timeout| (-2) Linux.pm, sanitize (line: 1177)
> >> > |22905|44:44|timeout| (-3) reclaim.pm, call_os_sanitize (line: 271)
> >> > |22905|44:44|timeout| (-4) reclaim.pm, process (line: 162)
> >> > |22905|44:44|timeout| (-5) vcld, make_new_child (line: 571)
> >> >
> >> > 2015-05-17 08:35:40|22905|44:44|timeout|utils.pm:
> run_ssh_command(4902)|
> >> executing SSH command on VM7Cent6VCL1:
> >> > |22905|44:44|timeout| /usr/bin/ssh -i /etc/vcl/vcl.key  -o
> >> > |StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o
> >> > |ConnectionAttempts=1 -o ConnectTimeout=3 -l root -p 22 -x
> VM7Cent6VCL1
> >> > |'/usr/sbin/userdel admin' 2>&1 22905|44:44|timeout| ---- WARNING ----
> >> > |22905|44:44|timeout| 2015-05-17
> >> > |08:35:40|22905|44:44|timeout|utils.pm:run_ssh_command(5006)|attempt
> >> > 1/3:
> >> > |failed to execute SSH command on VM7Cent6VCL1: '/usr/sbin/userdel
> >> > admin',
> >> > |exit status: 255, output: 22905|44:44|timeout| ssh output
> >> > (/usr/sbin/...):
> >> > |ssh: connect to host VM7Cent6VCL1 port 22: No route to host
> >> > |22905|44:44|timeout| ( 0) utils.pm, run_ssh_command (line: 5006)
> >> > |22905|44:44|timeout| (-1) Linux.pm, delete_user (line: 881)
> >> > |22905|44:44|timeout| (-2) Linux.pm, sanitize (line: 1177)
> >> > |22905|44:44|timeout| (-3) reclaim.pm, call_os_sanitize (line: 271)
> >> > |22905|44:44|timeout| (-4) reclaim.pm, process (line: 162)
> >> > |22905|44:44|timeout| (-5) vcld, make_new_child (line: 571)
> >> >
> >> > 2015-05-17
> >> > 08:35:40|22905|44:44|timeout|utils.pm:run_ssh_command(4894)|sleeping
> for
> >> > 2
> >> > seconds before making next SSH attempt
> >> > 2015-05-17 08:35:42|22905|44:44|timeout|utils.pm:
> run_ssh_command(4906)|
> >> attempt 2/3: executing SSH command on VM7Cent6VCL1:
> >> > |22905|44:44|timeout| /usr/bin/ssh -i /etc/vcl/vcl.key  -o
> >> > |StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o
> >> > |ConnectionAttempts=1 -o ConnectTimeout=3 -l root -p 22 -x
> VM7Cent6VCL1
> >> > |'/usr/sbin/userdel admin' 2>&1 22905|44:44|timeout| ---- WARNING ----
> >> > |22905|44:44|timeout| 2015-05-17
> >> > |08:35:42|22905|44:44|timeout|utils.pm:run_ssh_command(5006)|attempt
> >> > 2/3:
> >> > |failed to execute SSH command on VM7Cent6VCL1: '/usr/sbin/userdel
> >> > admin',
> >> > |exit status: 255, output: 22905|44:44|timeout| ssh output
> >> > (/usr/sbin/...):
> >> > |ssh: connect to host VM7Cent6VCL1 port 22: No route to host
> >> > |22905|44:44|timeout| ( 0) utils.pm, run_ssh_command (line: 5006)
> >> > |22905|44:44|timeout| (-1) Linux.pm, delete_user (line: 881)
> >> > |22905|44:44|timeout| (-2) Linux.pm, sanitize (line: 1177)
> >> > |22905|44:44|timeout| (-3) reclaim.pm, call_os_sanitize (line: 271)
> >> > |22905|44:44|timeout| (-4) reclaim.pm, process (line: 162)
> >> > |22905|44:44|timeout| (-5) vcld, make_new_child (line: 571)
> >> >
> >> > 2015-05-17
> >> > 08:35:42|22905|44:44|timeout|utils.pm:run_ssh_command(4894)|sleeping
> for
> >> > 4
> >> > seconds before making next SSH attempt
> >> > 2015-05-17 08:35:46|22905|44:44|timeout|utils.pm:
> run_ssh_command(4906)|
> >> attempt 3/3: executing SSH command on VM7Cent6VCL1:
> >> > |22905|44:44|timeout| /usr/bin/ssh -i /etc/vcl/vcl.key  -o
> >> > |StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o
> >> > |ConnectionAttempts=1 -o ConnectTimeout=3 -l root -p 22 -x
> VM7Cent6VCL1
> >> > |'/usr/sbin/userdel admin' 2>&1 22905|44:44|timeout| ---- WARNING ----
> >> > |22905|44:44|timeout| 2015-05-17
> >> > |08:35:46|22905|44:44|timeout|utils.pm:run_ssh_command(5006)|attempt
> >> > 3/3:
> >> > |failed to execute SSH command on VM7Cent6VCL1: '/usr/sbin/userdel
> >> > admin',
> >> > |exit status: 255, output: 22905|44:44|timeout| ssh output
> >> > (/usr/sbin/...):
> >> > |ssh: connect to host VM7Cent6VCL1 port 22: No route to host
> >> > |22905|44:44|timeout| ( 0) utils.pm, run_ssh_command (line: 5006)
> >> > |22905|44:44|timeout| (-1) Linux.pm, delete_user (line: 881)
> >> > |22905|44:44|timeout| (-2) Linux.pm, sanitize (line: 1177)
> >> > |22905|44:44|timeout| (-3) reclaim.pm, call_os_sanitize (line: 271)
> >> > |22905|44:44|timeout| (-4) reclaim.pm, process (line: 162)
> >> > |22905|44:44|timeout| (-5) vcld, make_new_child (line: 571)
> >> > |22905|44:44|timeout| ---- WARNING ----
> >> > |22905|44:44|timeout| 2015-05-17
> >> > |08:35:46|22905|44:44|timeout|utils.pm:run_ssh_command(5043)|failed
> to
> >> > run
> >> > |SSH command after 3 attempts, command: /usr/bin/ssh -i
> /etc/vcl/vcl.key
> >> > |-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o
> >> > |ConnectionAttempts=1 -o ConnectTimeout=3 -l root -p 22 -x
> VM7Cent6VCL1
> >> > |'/usr/sbin/userdel admin' 2>&1, exit status: 255, output:
> >> > |22905|44:44|timeout| ssh output (/usr/sbin/...): ssh: connect to host
> >> > |VM7Cent6VCL1 port 22: No route to host 22905|44:44|timeout| ( 0)
> >> > utils.pm,
> >> > |run_ssh_command (line: 5043)
> >> > |22905|44:44|timeout| (-1) Linux.pm, delete_user (line: 881)
> >> > |22905|44:44|timeout| (-2) Linux.pm, sanitize (line: 1177)
> >> > |22905|44:44|timeout| (-3) reclaim.pm, call_os_sanitize (line: 271)
> >> > |22905|44:44|timeout| (-4) reclaim.pm, process (line: 162)
> >> > |22905|44:44|timeout| (-5) vcld, make_new_child (line: 571)
> >> >
> >> > 2015-05-17 08:35:46|22905|44:44|timeout|utils.pm:
> run_ssh_command(4902)|
> >> executing SSH command on VM7Cent6VCL1:
> >> > |22905|44:44|timeout| /usr/bin/ssh -i /etc/vcl/vcl.key  -o
> >> > |StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o
> >> > |ConnectionAttempts=1 -o ConnectTimeout=3 -l root -p 22 -x
> VM7Cent6VCL1
> >> > |'sed -i -e "/^AllowUsers .*/d" /etc/ssh/external_sshd_config' 2>&1
> >> > |22905|44:44|timeout| ---- WARNING ----
> >> > |22905|44:44|timeout| 2015-05-17
> >> > |08:35:46|22905|44:44|timeout|utils.pm:run_ssh_command(5006)|attempt
> >> > 1/3:
> >> > |failed to execute SSH command on VM7Cent6VCL1: 'sed -i -e
> "/^AllowUsers
> >> > |.*/d" /etc/ssh/external_sshd_config', exit status: 255, output:
> >> > |22905|44:44|timeout| ssh output (sed -i -e ...): ssh: connect to host
> >> > |VM7Cent6VCL1 port 22: No route to host 22905|44:44|timeout| ( 0)
> >> > utils.pm,
> >> > |run_ssh_command (line: 5006)
> >> > |22905|44:44|timeout| (-1) Linux.pm, delete_user (line: 890)
> >> > |22905|44:44|timeout| (-2) Linux.pm, sanitize (line: 1177)
> >> > |22905|44:44|timeout| (-3) reclaim.pm, call_os_sanitize (line: 271)
> >> > |22905|44:44|timeout| (-4) reclaim.pm, process (line: 162)
> >> > |22905|44:44|timeout| (-5) vcld, make_new_child (line: 571)
> >> >
> >> > 2015-05-17
> >> > 08:35:46|22905|44:44|timeout|utils.pm:run_ssh_command(4894)|sleeping
> for
> >> > 2
> >> > seconds before making next SSH attempt
> >> > 2015-05-17 08:35:48|22905|44:44|timeout|utils.pm:
> run_ssh_command(4906)|
> >> attempt 2/3: executing SSH command on VM7Cent6VCL1:
> >> > |22905|44:44|timeout| /usr/bin/ssh -i /etc/vcl/vcl.key  -o
> >> > |StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o
> >> > |ConnectionAttempts=1 -o ConnectTimeout=3 -l root -p 22 -x
> VM7Cent6VCL1
> >> > |'sed -i -e "/^AllowUsers .*/d" /etc/ssh/external_sshd_config' 2>&1
> >> > |22905|44:44|timeout| ---- WARNING ----
> >> > |22905|44:44|timeout| 2015-05-17
> >> > |08:35:48|22905|44:44|timeout|utils.pm:run_ssh_command(5006)|attempt
> >> > 2/3:
> >> > |failed to execute SSH command on VM7Cent6VCL1: 'sed -i -e
> "/^AllowUsers
> >> > |.*/d" /etc/ssh/external_sshd_config', exit status: 255, output:
> >> > |22905|44:44|timeout| ssh output (sed -i -e ...): ssh: connect to host
> >> > |VM7Cent6VCL1 port 22: No route to host 22905|44:44|timeout| ( 0)
> >> > utils.pm,
> >> > |run_ssh_command (line: 5006)
> >> > |22905|44:44|timeout| (-1) Linux.pm, delete_user (line: 890)
> >> > |22905|44:44|timeout| (-2) Linux.pm, sanitize (line: 1177)
> >> > |22905|44:44|timeout| (-3) reclaim.pm, call_os_sanitize (line: 271)
> >> > |22905|44:44|timeout| (-4) reclaim.pm, process (line: 162)
> >> > |22905|44:44|timeout| (-5) vcld, make_new_child (line: 571)
> >> >
> >> > 2015-05-17
> >> > 08:35:48|22905|44:44|timeout|utils.pm:run_ssh_command(4894)|sleeping
> for
> >> > 4
> >> > seconds before making next SSH attempt
> >> > 2015-05-17 08:35:52|22905|44:44|timeout|utils.pm:
> run_ssh_command(4906)|
> >> attempt 3/3: executing SSH command on VM7Cent6VCL1:
> >> > |22905|44:44|timeout| /usr/bin/ssh -i /etc/vcl/vcl.key  -o
> >> > |StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o
> >> > |ConnectionAttempts=1 -o ConnectTimeout=3 -l root -p 22 -x
> VM7Cent6VCL1
> >> > |'sed -i -e "/^AllowUsers .*/d" /etc/ssh/external_sshd_config' 2>&1
> >> > |22905|44:44|timeout| ---- WARNING ----
> >> > |22905|44:44|timeout| 2015-05-17
> >> > |08:35:52|22905|44:44|timeout|utils.pm:run_ssh_command(5006)|attempt
> >> > 3/3:
> >> > |failed to execute SSH command on VM7Cent6VCL1: 'sed -i -e
> "/^AllowUsers
> >> > |.*/d" /etc/ssh/external_sshd_config', exit status: 255, output:
> >> > |22905|44:44|timeout| ssh output (sed -i -e ...): ssh: connect to host
> >> > |VM7Cent6VCL1 port 22: No route to host 22905|44:44|timeout| ( 0)
> >> > utils.pm,
> >> > |run_ssh_command (line: 5006)
> >> > |22905|44:44|timeout| (-1) Linux.pm, delete_user (line: 890)
> >> > |22905|44:44|timeout| (-2) Linux.pm, sanitize (line: 1177)
> >> > |22905|44:44|timeout| (-3) reclaim.pm, call_os_sanitize (line: 271)
> >> > |22905|44:44|timeout| (-4) reclaim.pm, process (line: 162)
> >> > |22905|44:44|timeout| (-5) vcld, make_new_child (line: 571)
> >> > |22905|44:44|timeout| ---- WARNING ----
> >> > |22905|44:44|timeout| 2015-05-17
> >> > |08:35:52|22905|44:44|timeout|utils.pm:run_ssh_command(5043)|failed
> to
> >> > run
> >> > |SSH command after 3 attempts, command: /usr/bin/ssh -i
> /etc/vcl/vcl.key
> >> > |-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o
> >> > |ConnectionAttempts=1 -o ConnectTimeout=3 -l root -p 22 -x
> VM7Cent6VCL1
> >> > |'sed -i -e "/^AllowUsers .*/d" /etc/ssh/external_sshd_config' 2>&1,
> >> > exit
> >> > |status: 255, output: 22905|44:44|timeout| ssh output (sed -i -e ...):
> >> > ssh:
> >> > |connect to host VM7Cent6VCL1 port 22: No route to host
> >> > |22905|44:44|timeout| ( 0) utils.pm, run_ssh_command (line: 5043)
> >> > |22905|44:44|timeout| (-1) Linux.pm, delete_user (line: 890)
> >> > |22905|44:44|timeout| (-2) Linux.pm, sanitize (line: 1177)
> >> > |22905|44:44|timeout| (-3) reclaim.pm, call_os_sanitize (line: 271)
> >> > |22905|44:44|timeout| (-4) reclaim.pm, process (line: 162)
> >> > |22905|44:44|timeout| (-5) vcld, make_new_child (line: 571)
> >> >
> >> > 2015-05-17
> >> >
> >> >
> 08:35:52|22905|44:44|timeout|DataStructure.pm:get_computer_private_ip_addre
> >> > ss(1641)|attempting to retrieve private IP address for computer:
> >> > VM7Cent6VCL1 2015-05-17
> >> > 08:35:52|22905|44:44|timeout|OS.pm:get_file_contents(1897)|retrieved
> 15
> >> > lines from file on localhost: '/etc/hosts' 2015-05-17
> >> >
> >> >
> 08:35:52|22905|44:44|timeout|DataStructure.pm:get_computer_private_ip_addre
> >> > ss(1697)|returning IP address from /etc/hosts file: 192.168.10.48
> >> > 2015-05-17
> >> > 08:35:52|22905|44:44|timeout|utils.pm:is_inblockrequest(5760)|zero
> rows
> >> > were returned from database select 2015-05-17
> >> >
> >> >
> 08:35:52|22905|44:44|timeout|DataStructure.pm:get_image_affiliation_name(21
> >> > 21)|image owner id: 1 2015-05-17
> >> >
> >> >
> 08:35:52|22905|44:44|timeout|DataStructure.pm:retrieve_user_data(1412)|atte
> >> > mpting to retrieve and store data for user: user.id = '1' 2015-05-17
> >> >
> >> >
> 08:35:52|22905|44:44|timeout|DataStructure.pm:retrieve_user_data(1475)|data
> >> > has been retrieved for user: admin (id: 1)
> >> - --
> >> - -------------------------------
> >> Josh Thompson
> >> VCL Developer
> >> North Carolina State University
> >>
> >> my GPG/PGP key can be found at pgp.mit.edu
> >>
> >> All electronic mail messages in connection with State business which
> >> are sent to or received by this account are subject to the NC Public
> >> Records Law and may be disclosed to third parties.
> >> -----BEGIN PGP SIGNATURE-----
> >> Version: GnuPG v2
> >>
> >> iD8DBQFVWcsOV/LQcNdtPQMRAtezAJ0Sw6V2YJ9msiQ8o50Rwbe21fgkQACeKeme
> >> gQWeyHN1fzwbqDZ0Vw5aW+o=
> >> =I6Ta
> >> -----END PGP SIGNATURE-----
> >>
> >
>
>
>
> --
> Aaron Peeler
> Program Manager
> Virtual Computing Lab
> NC State University
>
> All electronic mail messages in connection with State business which
> are sent to or received by this account are subject to the NC Public
> Records Law and may be disclosed to third parties.
>

Re: Fw: PROBLEM -- localhost|44:44|timeout|Linux.pm|VM7Cent6VCL1>VM7|vmwarelinux-vm7c2testvcl212-v0|admin

Posted by Aaron Peeler <aa...@ncsu.edu>.
Hello Lewis,

Sorry we don't have a good VCL starter or VCL for dummies sort of documentation.

On Tue, May 19, 2015 at 7:44 AM, Elwin Litchfield <le...@mail.usf.edu> wrote:
> Thanks, your explanation clears up a lot of things for me.  I am still
> looking for some  good documentation/book that explains Apache VCL from
> beginning to end, IE Apache VCL for Dummies.
>
> I will setup some more test & examine the vcld.log file more closely, it is
> Greek to me.  But the more I dig into Apache VCL the clearer things become.
>
> It looks like I need to rethink my plans about how to use Apache VCL.  I was
> under the impression that the virtual machine would be powered on for the
> reservation & powered off after the reservation was over.  If I understand
> correctly, using 2.3.2, a user making a reservation may or may not receive
> the same virtual machine each time, but the functionality is the same?


In 2.4.2 (the latest release) there is a new reload action which
powers off the virtual machine(or node) after a reservation.  See this
link for the details:
https://cwiki.apache.org/confluence/display/VCL/Computer+Predictive+Loading+Modules+and+Power+Off

Correct, the user may or may not get the same machine, but the image
is generally the same with different IP address and hostname.


>
> Thanks Again.
>
> Lewis
>
> On Mon, May 18, 2015 at 7:20 AM, Josh Thompson <jo...@ncsu.edu>
> wrote:
>>
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> Lewis,
>>
>> Looking through vcld.log will allow you to see every ssh command that vcld
>> sends to a node.
>>
>> Also, let me explain the process VCL uses for provisioning.  We use
>> something
>> we call "hot standby" to help minimize the amount of time it takes for
>> reservations to become ready for users to connect to their reservations.
>> At
>> the end of each reservation, the node is reloaded so that it is almost
>> ready
>> for another user.  Then, when another user makes a reservation for it,
>> only a
>> few steps need to be done to finish provisioning it.  The image selected
>> to be
>> reloaded on the node depends on what Predictive Reload module is assigned.
>> Previous to 2.4.2, Predictive Reload modules were assigned per management
>> node, meaning all computers managed by a specific management node would be
>> handled the same way.  With 2.4.2, it is assigned per computer, meaning
>> each
>> computer can be handled differently.  Also, 2.4.2 introduced a new module
>> that
>> doesn't actually reload it, but instead powers it off, and for VMs,
>> unregisters it.
>>
>> All that to say, the behavior you are seeing may be that the node has been
>> reloaded at the end of your reservation to prep for another one.
>>
>> One more point to note is that VCL randomizes the root/admin password when
>> it
>> provisions each node.  You can find the password it selects for each
>> reservation in the vcld.log file.
>>
>> Josh
>>
>> On Sun May 17 2015 1:09:04 PM Litchfield, Elwin L. wrote:
>> > The VM was powered on as scheduled, I logged on to the VM using admin &
>> > PW
>> > supplied.  Logged off & the VM stayed powered on after the reservation
>> > expired.  Ping to the VM works, ICMP Type 0 Code 0.   ICMP Type 3 Code
>> > 10
>> > to Port 22 reply is Host Administratively prohibited.
>> >
>> > From the VM console, on the VMware host,  no administrator user is able
>> > to
>> > logon.  I have added 2 more admin users & they will not work either.
>> > The
>> > admin logon supplied from the reservation worked, but ssh was not
>> > active/displayed as a service.
>> >
>> > I have a Wireshark 1.8.10 trace of the private network conversation.  I
>> > am
>> > researching how to upgrade to the current release of Wireshark, but I
>> > have
>> > not found any information that Wireshark will decode SSH.
>> >
>> > Is there any trace to run to determine what is taking place between the
>> > Management Node & the VM?
>> >
>> > Thanks
>> >
>> > Lewis Litchfield
>> > Technology Services – Viking Center Lab
>> > University of South Florida Sarasota-Manatee
>> > 941-359-4231
>> >
>> > ________________________________________
>> > From: root@localhost.localdomain <ro...@localhost.localdomain>
>> > Sent: Sunday, May 17, 2015 8:35 AM
>> > To: lewis15@mail.usf.edu
>> > Subject: PROBLEM --
>> >
>> > localhost|44:44|timeout|Linux.pm|VM7Cent6VCL1>VM7|vmwarelinux-vm7c2testvcl2
>> > 12-v0|admin
>> >
>> > failed to add AllowUsers admin to external_sshd_config
>> > ------------------------------------------------------------------------
>> > time: 2015-05-17 08:35:52
>> > caller: Linux.pm:delete_user(894)
>> > ( 0) Linux.pm, delete_user (line: 894)
>> > (-1) Linux.pm, sanitize (line: 1177)
>> > (-2) reclaim.pm, call_os_sanitize (line: 271)
>> > (-3) reclaim.pm, process (line: 162)
>> > (-4) vcld, make_new_child (line: 571)
>> > (-5) vcld, main (line: 350)
>> > ------------------------------------------------------------------------
>> > management node: localhost
>> > reservation PID: 22905
>> > parent vcld PID: 26123
>> >
>> > request ID: 44
>> > reservation ID: 44
>> > request state/laststate: timeout/reserved
>> > request start time: 2015-05-17 08:15:00
>> > request end time: 2015-05-17 09:00:00
>> > for imaging: no
>> > log ID: 51
>> >
>> > computer: VM7Cent6VCL1
>> > computer id: 6
>> > computer type: virtualmachine
>> > computer eth0 MAC address: 00:50:56:3F:40:E3
>> > computer eth1 MAC address: 00:50:56:3F:40:ED
>> > computer private IP address: 192.168.10.48
>> > computer public IP address: 131.247.29.132
>> > computer in block allocation: no
>> > provisioning module: VCL::Module::Provisioning::VMware::VMware
>> >
>> > vm host: VM7
>> > vm host ID: 2
>> > vm host computer ID: 5
>> > vm profile: VMware ESXi - local storage
>> > vm profile VM path: datastore1
>> > vm profile repository path: <undefined>
>> > vm profile datastore path: datastore1
>> > vm profile disk type: shared
>> >
>> > image: vmwarelinux-vm7c2testvcl212-v0
>> > image display name: vm7c2testvcl2
>> > image ID: 12
>> > image revision ID: 12
>> > image size: 5762 MB
>> > use Sysprep: yes
>> > root access: yes
>> > image owner ID: 1
>> > image owner affiliation: Local
>> > image revision date created: 2015-05-01 09:43:57
>> > image revision production: yes
>> > OS module: VCL::Module::OS::Linux
>> >
>> > user: admin
>> > user name: vcl admin
>> > user ID: 1
>> > user affiliation: Local
>> > ------------------------------------------------------------------------
>> >
>> > RECENT LOG ENTRIES FOR THIS PROCESS:
>> > |22905|44:44|timeout| (-1) Linux.pm, delete_user (line: 871)
>> > |22905|44:44|timeout| (-2) Linux.pm, sanitize (line: 1177)
>> > |22905|44:44|timeout| (-3) reclaim.pm, call_os_sanitize (line: 271)
>> > |22905|44:44|timeout| (-4) reclaim.pm, process (line: 162)
>> > |22905|44:44|timeout| (-5) vcld, make_new_child (line: 571)
>> >
>> > 2015-05-17 08:35:40|22905|44:44|timeout|utils.pm:run_ssh_command(4902)|
>> executing SSH command on VM7Cent6VCL1:
>> > |22905|44:44|timeout| /usr/bin/ssh -i /etc/vcl/vcl.key  -o
>> > |StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o
>> > |ConnectionAttempts=1 -o ConnectTimeout=3 -l root -p 22 -x VM7Cent6VCL1
>> > |'/usr/sbin/userdel admin' 2>&1 22905|44:44|timeout| ---- WARNING ----
>> > |22905|44:44|timeout| 2015-05-17
>> > |08:35:40|22905|44:44|timeout|utils.pm:run_ssh_command(5006)|attempt
>> > 1/3:
>> > |failed to execute SSH command on VM7Cent6VCL1: '/usr/sbin/userdel
>> > admin',
>> > |exit status: 255, output: 22905|44:44|timeout| ssh output
>> > (/usr/sbin/...):
>> > |ssh: connect to host VM7Cent6VCL1 port 22: No route to host
>> > |22905|44:44|timeout| ( 0) utils.pm, run_ssh_command (line: 5006)
>> > |22905|44:44|timeout| (-1) Linux.pm, delete_user (line: 881)
>> > |22905|44:44|timeout| (-2) Linux.pm, sanitize (line: 1177)
>> > |22905|44:44|timeout| (-3) reclaim.pm, call_os_sanitize (line: 271)
>> > |22905|44:44|timeout| (-4) reclaim.pm, process (line: 162)
>> > |22905|44:44|timeout| (-5) vcld, make_new_child (line: 571)
>> >
>> > 2015-05-17
>> > 08:35:40|22905|44:44|timeout|utils.pm:run_ssh_command(4894)|sleeping for
>> > 2
>> > seconds before making next SSH attempt
>> > 2015-05-17 08:35:42|22905|44:44|timeout|utils.pm:run_ssh_command(4906)|
>> attempt 2/3: executing SSH command on VM7Cent6VCL1:
>> > |22905|44:44|timeout| /usr/bin/ssh -i /etc/vcl/vcl.key  -o
>> > |StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o
>> > |ConnectionAttempts=1 -o ConnectTimeout=3 -l root -p 22 -x VM7Cent6VCL1
>> > |'/usr/sbin/userdel admin' 2>&1 22905|44:44|timeout| ---- WARNING ----
>> > |22905|44:44|timeout| 2015-05-17
>> > |08:35:42|22905|44:44|timeout|utils.pm:run_ssh_command(5006)|attempt
>> > 2/3:
>> > |failed to execute SSH command on VM7Cent6VCL1: '/usr/sbin/userdel
>> > admin',
>> > |exit status: 255, output: 22905|44:44|timeout| ssh output
>> > (/usr/sbin/...):
>> > |ssh: connect to host VM7Cent6VCL1 port 22: No route to host
>> > |22905|44:44|timeout| ( 0) utils.pm, run_ssh_command (line: 5006)
>> > |22905|44:44|timeout| (-1) Linux.pm, delete_user (line: 881)
>> > |22905|44:44|timeout| (-2) Linux.pm, sanitize (line: 1177)
>> > |22905|44:44|timeout| (-3) reclaim.pm, call_os_sanitize (line: 271)
>> > |22905|44:44|timeout| (-4) reclaim.pm, process (line: 162)
>> > |22905|44:44|timeout| (-5) vcld, make_new_child (line: 571)
>> >
>> > 2015-05-17
>> > 08:35:42|22905|44:44|timeout|utils.pm:run_ssh_command(4894)|sleeping for
>> > 4
>> > seconds before making next SSH attempt
>> > 2015-05-17 08:35:46|22905|44:44|timeout|utils.pm:run_ssh_command(4906)|
>> attempt 3/3: executing SSH command on VM7Cent6VCL1:
>> > |22905|44:44|timeout| /usr/bin/ssh -i /etc/vcl/vcl.key  -o
>> > |StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o
>> > |ConnectionAttempts=1 -o ConnectTimeout=3 -l root -p 22 -x VM7Cent6VCL1
>> > |'/usr/sbin/userdel admin' 2>&1 22905|44:44|timeout| ---- WARNING ----
>> > |22905|44:44|timeout| 2015-05-17
>> > |08:35:46|22905|44:44|timeout|utils.pm:run_ssh_command(5006)|attempt
>> > 3/3:
>> > |failed to execute SSH command on VM7Cent6VCL1: '/usr/sbin/userdel
>> > admin',
>> > |exit status: 255, output: 22905|44:44|timeout| ssh output
>> > (/usr/sbin/...):
>> > |ssh: connect to host VM7Cent6VCL1 port 22: No route to host
>> > |22905|44:44|timeout| ( 0) utils.pm, run_ssh_command (line: 5006)
>> > |22905|44:44|timeout| (-1) Linux.pm, delete_user (line: 881)
>> > |22905|44:44|timeout| (-2) Linux.pm, sanitize (line: 1177)
>> > |22905|44:44|timeout| (-3) reclaim.pm, call_os_sanitize (line: 271)
>> > |22905|44:44|timeout| (-4) reclaim.pm, process (line: 162)
>> > |22905|44:44|timeout| (-5) vcld, make_new_child (line: 571)
>> > |22905|44:44|timeout| ---- WARNING ----
>> > |22905|44:44|timeout| 2015-05-17
>> > |08:35:46|22905|44:44|timeout|utils.pm:run_ssh_command(5043)|failed to
>> > run
>> > |SSH command after 3 attempts, command: /usr/bin/ssh -i /etc/vcl/vcl.key
>> > |-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o
>> > |ConnectionAttempts=1 -o ConnectTimeout=3 -l root -p 22 -x VM7Cent6VCL1
>> > |'/usr/sbin/userdel admin' 2>&1, exit status: 255, output:
>> > |22905|44:44|timeout| ssh output (/usr/sbin/...): ssh: connect to host
>> > |VM7Cent6VCL1 port 22: No route to host 22905|44:44|timeout| ( 0)
>> > utils.pm,
>> > |run_ssh_command (line: 5043)
>> > |22905|44:44|timeout| (-1) Linux.pm, delete_user (line: 881)
>> > |22905|44:44|timeout| (-2) Linux.pm, sanitize (line: 1177)
>> > |22905|44:44|timeout| (-3) reclaim.pm, call_os_sanitize (line: 271)
>> > |22905|44:44|timeout| (-4) reclaim.pm, process (line: 162)
>> > |22905|44:44|timeout| (-5) vcld, make_new_child (line: 571)
>> >
>> > 2015-05-17 08:35:46|22905|44:44|timeout|utils.pm:run_ssh_command(4902)|
>> executing SSH command on VM7Cent6VCL1:
>> > |22905|44:44|timeout| /usr/bin/ssh -i /etc/vcl/vcl.key  -o
>> > |StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o
>> > |ConnectionAttempts=1 -o ConnectTimeout=3 -l root -p 22 -x VM7Cent6VCL1
>> > |'sed -i -e "/^AllowUsers .*/d" /etc/ssh/external_sshd_config' 2>&1
>> > |22905|44:44|timeout| ---- WARNING ----
>> > |22905|44:44|timeout| 2015-05-17
>> > |08:35:46|22905|44:44|timeout|utils.pm:run_ssh_command(5006)|attempt
>> > 1/3:
>> > |failed to execute SSH command on VM7Cent6VCL1: 'sed -i -e "/^AllowUsers
>> > |.*/d" /etc/ssh/external_sshd_config', exit status: 255, output:
>> > |22905|44:44|timeout| ssh output (sed -i -e ...): ssh: connect to host
>> > |VM7Cent6VCL1 port 22: No route to host 22905|44:44|timeout| ( 0)
>> > utils.pm,
>> > |run_ssh_command (line: 5006)
>> > |22905|44:44|timeout| (-1) Linux.pm, delete_user (line: 890)
>> > |22905|44:44|timeout| (-2) Linux.pm, sanitize (line: 1177)
>> > |22905|44:44|timeout| (-3) reclaim.pm, call_os_sanitize (line: 271)
>> > |22905|44:44|timeout| (-4) reclaim.pm, process (line: 162)
>> > |22905|44:44|timeout| (-5) vcld, make_new_child (line: 571)
>> >
>> > 2015-05-17
>> > 08:35:46|22905|44:44|timeout|utils.pm:run_ssh_command(4894)|sleeping for
>> > 2
>> > seconds before making next SSH attempt
>> > 2015-05-17 08:35:48|22905|44:44|timeout|utils.pm:run_ssh_command(4906)|
>> attempt 2/3: executing SSH command on VM7Cent6VCL1:
>> > |22905|44:44|timeout| /usr/bin/ssh -i /etc/vcl/vcl.key  -o
>> > |StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o
>> > |ConnectionAttempts=1 -o ConnectTimeout=3 -l root -p 22 -x VM7Cent6VCL1
>> > |'sed -i -e "/^AllowUsers .*/d" /etc/ssh/external_sshd_config' 2>&1
>> > |22905|44:44|timeout| ---- WARNING ----
>> > |22905|44:44|timeout| 2015-05-17
>> > |08:35:48|22905|44:44|timeout|utils.pm:run_ssh_command(5006)|attempt
>> > 2/3:
>> > |failed to execute SSH command on VM7Cent6VCL1: 'sed -i -e "/^AllowUsers
>> > |.*/d" /etc/ssh/external_sshd_config', exit status: 255, output:
>> > |22905|44:44|timeout| ssh output (sed -i -e ...): ssh: connect to host
>> > |VM7Cent6VCL1 port 22: No route to host 22905|44:44|timeout| ( 0)
>> > utils.pm,
>> > |run_ssh_command (line: 5006)
>> > |22905|44:44|timeout| (-1) Linux.pm, delete_user (line: 890)
>> > |22905|44:44|timeout| (-2) Linux.pm, sanitize (line: 1177)
>> > |22905|44:44|timeout| (-3) reclaim.pm, call_os_sanitize (line: 271)
>> > |22905|44:44|timeout| (-4) reclaim.pm, process (line: 162)
>> > |22905|44:44|timeout| (-5) vcld, make_new_child (line: 571)
>> >
>> > 2015-05-17
>> > 08:35:48|22905|44:44|timeout|utils.pm:run_ssh_command(4894)|sleeping for
>> > 4
>> > seconds before making next SSH attempt
>> > 2015-05-17 08:35:52|22905|44:44|timeout|utils.pm:run_ssh_command(4906)|
>> attempt 3/3: executing SSH command on VM7Cent6VCL1:
>> > |22905|44:44|timeout| /usr/bin/ssh -i /etc/vcl/vcl.key  -o
>> > |StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o
>> > |ConnectionAttempts=1 -o ConnectTimeout=3 -l root -p 22 -x VM7Cent6VCL1
>> > |'sed -i -e "/^AllowUsers .*/d" /etc/ssh/external_sshd_config' 2>&1
>> > |22905|44:44|timeout| ---- WARNING ----
>> > |22905|44:44|timeout| 2015-05-17
>> > |08:35:52|22905|44:44|timeout|utils.pm:run_ssh_command(5006)|attempt
>> > 3/3:
>> > |failed to execute SSH command on VM7Cent6VCL1: 'sed -i -e "/^AllowUsers
>> > |.*/d" /etc/ssh/external_sshd_config', exit status: 255, output:
>> > |22905|44:44|timeout| ssh output (sed -i -e ...): ssh: connect to host
>> > |VM7Cent6VCL1 port 22: No route to host 22905|44:44|timeout| ( 0)
>> > utils.pm,
>> > |run_ssh_command (line: 5006)
>> > |22905|44:44|timeout| (-1) Linux.pm, delete_user (line: 890)
>> > |22905|44:44|timeout| (-2) Linux.pm, sanitize (line: 1177)
>> > |22905|44:44|timeout| (-3) reclaim.pm, call_os_sanitize (line: 271)
>> > |22905|44:44|timeout| (-4) reclaim.pm, process (line: 162)
>> > |22905|44:44|timeout| (-5) vcld, make_new_child (line: 571)
>> > |22905|44:44|timeout| ---- WARNING ----
>> > |22905|44:44|timeout| 2015-05-17
>> > |08:35:52|22905|44:44|timeout|utils.pm:run_ssh_command(5043)|failed to
>> > run
>> > |SSH command after 3 attempts, command: /usr/bin/ssh -i /etc/vcl/vcl.key
>> > |-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o
>> > |ConnectionAttempts=1 -o ConnectTimeout=3 -l root -p 22 -x VM7Cent6VCL1
>> > |'sed -i -e "/^AllowUsers .*/d" /etc/ssh/external_sshd_config' 2>&1,
>> > exit
>> > |status: 255, output: 22905|44:44|timeout| ssh output (sed -i -e ...):
>> > ssh:
>> > |connect to host VM7Cent6VCL1 port 22: No route to host
>> > |22905|44:44|timeout| ( 0) utils.pm, run_ssh_command (line: 5043)
>> > |22905|44:44|timeout| (-1) Linux.pm, delete_user (line: 890)
>> > |22905|44:44|timeout| (-2) Linux.pm, sanitize (line: 1177)
>> > |22905|44:44|timeout| (-3) reclaim.pm, call_os_sanitize (line: 271)
>> > |22905|44:44|timeout| (-4) reclaim.pm, process (line: 162)
>> > |22905|44:44|timeout| (-5) vcld, make_new_child (line: 571)
>> >
>> > 2015-05-17
>> >
>> > 08:35:52|22905|44:44|timeout|DataStructure.pm:get_computer_private_ip_addre
>> > ss(1641)|attempting to retrieve private IP address for computer:
>> > VM7Cent6VCL1 2015-05-17
>> > 08:35:52|22905|44:44|timeout|OS.pm:get_file_contents(1897)|retrieved 15
>> > lines from file on localhost: '/etc/hosts' 2015-05-17
>> >
>> > 08:35:52|22905|44:44|timeout|DataStructure.pm:get_computer_private_ip_addre
>> > ss(1697)|returning IP address from /etc/hosts file: 192.168.10.48
>> > 2015-05-17
>> > 08:35:52|22905|44:44|timeout|utils.pm:is_inblockrequest(5760)|zero rows
>> > were returned from database select 2015-05-17
>> >
>> > 08:35:52|22905|44:44|timeout|DataStructure.pm:get_image_affiliation_name(21
>> > 21)|image owner id: 1 2015-05-17
>> >
>> > 08:35:52|22905|44:44|timeout|DataStructure.pm:retrieve_user_data(1412)|atte
>> > mpting to retrieve and store data for user: user.id = '1' 2015-05-17
>> >
>> > 08:35:52|22905|44:44|timeout|DataStructure.pm:retrieve_user_data(1475)|data
>> > has been retrieved for user: admin (id: 1)
>> - --
>> - -------------------------------
>> Josh Thompson
>> VCL Developer
>> North Carolina State University
>>
>> my GPG/PGP key can be found at pgp.mit.edu
>>
>> All electronic mail messages in connection with State business which
>> are sent to or received by this account are subject to the NC Public
>> Records Law and may be disclosed to third parties.
>> -----BEGIN PGP SIGNATURE-----
>> Version: GnuPG v2
>>
>> iD8DBQFVWcsOV/LQcNdtPQMRAtezAJ0Sw6V2YJ9msiQ8o50Rwbe21fgkQACeKeme
>> gQWeyHN1fzwbqDZ0Vw5aW+o=
>> =I6Ta
>> -----END PGP SIGNATURE-----
>>
>



-- 
Aaron Peeler
Program Manager
Virtual Computing Lab
NC State University

All electronic mail messages in connection with State business which
are sent to or received by this account are subject to the NC Public
Records Law and may be disclosed to third parties.

Re: Fw: PROBLEM -- localhost|44:44|timeout|Linux.pm|VM7Cent6VCL1>VM7|vmwarelinux-vm7c2testvcl212-v0|admin

Posted by Elwin Litchfield <le...@mail.usf.edu>.
Thanks, your explanation clears up a lot of things for me.  I am still
looking for some  good documentation/book that explains Apache VCL from
beginning to end, IE Apache VCL for Dummies.

I will setup some more test & examine the vcld.log file more closely, it is
Greek to me.  But the more I dig into Apache VCL the clearer things become.


It looks like I need to rethink my plans about how to use Apache VCL.  I
was under the impression that the virtual machine would be powered on for
the reservation & powered off after the reservation was over.  If I
understand correctly, using 2.3.2, a user making a reservation may or may
not receive the same virtual machine each time, but the functionality is
the same?

Thanks Again.

Lewis

On Mon, May 18, 2015 at 7:20 AM, Josh Thompson <jo...@ncsu.edu>
wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Lewis,
>
> Looking through vcld.log will allow you to see every ssh command that vcld
> sends to a node.
>
> Also, let me explain the process VCL uses for provisioning.  We use
> something
> we call "hot standby" to help minimize the amount of time it takes for
> reservations to become ready for users to connect to their reservations.
> At
> the end of each reservation, the node is reloaded so that it is almost
> ready
> for another user.  Then, when another user makes a reservation for it,
> only a
> few steps need to be done to finish provisioning it.  The image selected
> to be
> reloaded on the node depends on what Predictive Reload module is assigned.
> Previous to 2.4.2, Predictive Reload modules were assigned per management
> node, meaning all computers managed by a specific management node would be
> handled the same way.  With 2.4.2, it is assigned per computer, meaning
> each
> computer can be handled differently.  Also, 2.4.2 introduced a new module
> that
> doesn't actually reload it, but instead powers it off, and for VMs,
> unregisters it.
>
> All that to say, the behavior you are seeing may be that the node has been
> reloaded at the end of your reservation to prep for another one.
>
> One more point to note is that VCL randomizes the root/admin password when
> it
> provisions each node.  You can find the password it selects for each
> reservation in the vcld.log file.
>
> Josh
>
> On Sun May 17 2015 1:09:04 PM Litchfield, Elwin L. wrote:
> > The VM was powered on as scheduled, I logged on to the VM using admin &
> PW
> > supplied.  Logged off & the VM stayed powered on after the reservation
> > expired.  Ping to the VM works, ICMP Type 0 Code 0.   ICMP Type 3 Code 10
> > to Port 22 reply is Host Administratively prohibited.
> >
> > From the VM console, on the VMware host,  no administrator user is able
> to
> > logon.  I have added 2 more admin users & they will not work either.  The
> > admin logon supplied from the reservation worked, but ssh was not
> > active/displayed as a service.
> >
> > I have a Wireshark 1.8.10 trace of the private network conversation.  I
> am
> > researching how to upgrade to the current release of Wireshark, but I
> have
> > not found any information that Wireshark will decode SSH.
> >
> > Is there any trace to run to determine what is taking place between the
> > Management Node & the VM?
> >
> > Thanks
> >
> > Lewis Litchfield
> > Technology Services – Viking Center Lab
> > University of South Florida Sarasota-Manatee
> > 941-359-4231
> >
> > ________________________________________
> > From: root@localhost.localdomain <ro...@localhost.localdomain>
> > Sent: Sunday, May 17, 2015 8:35 AM
> > To: lewis15@mail.usf.edu
> > Subject: PROBLEM --
> >
> localhost|44:44|timeout|Linux.pm|VM7Cent6VCL1>VM7|vmwarelinux-vm7c2testvcl2
> > 12-v0|admin
> >
> > failed to add AllowUsers admin to external_sshd_config
> > ------------------------------------------------------------------------
> > time: 2015-05-17 08:35:52
> > caller: Linux.pm:delete_user(894)
> > ( 0) Linux.pm, delete_user (line: 894)
> > (-1) Linux.pm, sanitize (line: 1177)
> > (-2) reclaim.pm, call_os_sanitize (line: 271)
> > (-3) reclaim.pm, process (line: 162)
> > (-4) vcld, make_new_child (line: 571)
> > (-5) vcld, main (line: 350)
> > ------------------------------------------------------------------------
> > management node: localhost
> > reservation PID: 22905
> > parent vcld PID: 26123
> >
> > request ID: 44
> > reservation ID: 44
> > request state/laststate: timeout/reserved
> > request start time: 2015-05-17 08:15:00
> > request end time: 2015-05-17 09:00:00
> > for imaging: no
> > log ID: 51
> >
> > computer: VM7Cent6VCL1
> > computer id: 6
> > computer type: virtualmachine
> > computer eth0 MAC address: 00:50:56:3F:40:E3
> > computer eth1 MAC address: 00:50:56:3F:40:ED
> > computer private IP address: 192.168.10.48
> > computer public IP address: 131.247.29.132
> > computer in block allocation: no
> > provisioning module: VCL::Module::Provisioning::VMware::VMware
> >
> > vm host: VM7
> > vm host ID: 2
> > vm host computer ID: 5
> > vm profile: VMware ESXi - local storage
> > vm profile VM path: datastore1
> > vm profile repository path: <undefined>
> > vm profile datastore path: datastore1
> > vm profile disk type: shared
> >
> > image: vmwarelinux-vm7c2testvcl212-v0
> > image display name: vm7c2testvcl2
> > image ID: 12
> > image revision ID: 12
> > image size: 5762 MB
> > use Sysprep: yes
> > root access: yes
> > image owner ID: 1
> > image owner affiliation: Local
> > image revision date created: 2015-05-01 09:43:57
> > image revision production: yes
> > OS module: VCL::Module::OS::Linux
> >
> > user: admin
> > user name: vcl admin
> > user ID: 1
> > user affiliation: Local
> > ------------------------------------------------------------------------
> >
> > RECENT LOG ENTRIES FOR THIS PROCESS:
> > |22905|44:44|timeout| (-1) Linux.pm, delete_user (line: 871)
> > |22905|44:44|timeout| (-2) Linux.pm, sanitize (line: 1177)
> > |22905|44:44|timeout| (-3) reclaim.pm, call_os_sanitize (line: 271)
> > |22905|44:44|timeout| (-4) reclaim.pm, process (line: 162)
> > |22905|44:44|timeout| (-5) vcld, make_new_child (line: 571)
> >
> > 2015-05-17 08:35:40|22905|44:44|timeout|utils.pm:run_ssh_command(4902)|
> executing SSH command on VM7Cent6VCL1:
> > |22905|44:44|timeout| /usr/bin/ssh -i /etc/vcl/vcl.key  -o
> > |StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o
> > |ConnectionAttempts=1 -o ConnectTimeout=3 -l root -p 22 -x VM7Cent6VCL1
> > |'/usr/sbin/userdel admin' 2>&1 22905|44:44|timeout| ---- WARNING ----
> > |22905|44:44|timeout| 2015-05-17
> > |08:35:40|22905|44:44|timeout|utils.pm:run_ssh_command(5006)|attempt
> 1/3:
> > |failed to execute SSH command on VM7Cent6VCL1: '/usr/sbin/userdel
> admin',
> > |exit status: 255, output: 22905|44:44|timeout| ssh output
> (/usr/sbin/...):
> > |ssh: connect to host VM7Cent6VCL1 port 22: No route to host
> > |22905|44:44|timeout| ( 0) utils.pm, run_ssh_command (line: 5006)
> > |22905|44:44|timeout| (-1) Linux.pm, delete_user (line: 881)
> > |22905|44:44|timeout| (-2) Linux.pm, sanitize (line: 1177)
> > |22905|44:44|timeout| (-3) reclaim.pm, call_os_sanitize (line: 271)
> > |22905|44:44|timeout| (-4) reclaim.pm, process (line: 162)
> > |22905|44:44|timeout| (-5) vcld, make_new_child (line: 571)
> >
> > 2015-05-17
> > 08:35:40|22905|44:44|timeout|utils.pm:run_ssh_command(4894)|sleeping
> for 2
> > seconds before making next SSH attempt
> > 2015-05-17 08:35:42|22905|44:44|timeout|utils.pm:run_ssh_command(4906)|
> attempt 2/3: executing SSH command on VM7Cent6VCL1:
> > |22905|44:44|timeout| /usr/bin/ssh -i /etc/vcl/vcl.key  -o
> > |StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o
> > |ConnectionAttempts=1 -o ConnectTimeout=3 -l root -p 22 -x VM7Cent6VCL1
> > |'/usr/sbin/userdel admin' 2>&1 22905|44:44|timeout| ---- WARNING ----
> > |22905|44:44|timeout| 2015-05-17
> > |08:35:42|22905|44:44|timeout|utils.pm:run_ssh_command(5006)|attempt
> 2/3:
> > |failed to execute SSH command on VM7Cent6VCL1: '/usr/sbin/userdel
> admin',
> > |exit status: 255, output: 22905|44:44|timeout| ssh output
> (/usr/sbin/...):
> > |ssh: connect to host VM7Cent6VCL1 port 22: No route to host
> > |22905|44:44|timeout| ( 0) utils.pm, run_ssh_command (line: 5006)
> > |22905|44:44|timeout| (-1) Linux.pm, delete_user (line: 881)
> > |22905|44:44|timeout| (-2) Linux.pm, sanitize (line: 1177)
> > |22905|44:44|timeout| (-3) reclaim.pm, call_os_sanitize (line: 271)
> > |22905|44:44|timeout| (-4) reclaim.pm, process (line: 162)
> > |22905|44:44|timeout| (-5) vcld, make_new_child (line: 571)
> >
> > 2015-05-17
> > 08:35:42|22905|44:44|timeout|utils.pm:run_ssh_command(4894)|sleeping
> for 4
> > seconds before making next SSH attempt
> > 2015-05-17 08:35:46|22905|44:44|timeout|utils.pm:run_ssh_command(4906)|
> attempt 3/3: executing SSH command on VM7Cent6VCL1:
> > |22905|44:44|timeout| /usr/bin/ssh -i /etc/vcl/vcl.key  -o
> > |StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o
> > |ConnectionAttempts=1 -o ConnectTimeout=3 -l root -p 22 -x VM7Cent6VCL1
> > |'/usr/sbin/userdel admin' 2>&1 22905|44:44|timeout| ---- WARNING ----
> > |22905|44:44|timeout| 2015-05-17
> > |08:35:46|22905|44:44|timeout|utils.pm:run_ssh_command(5006)|attempt
> 3/3:
> > |failed to execute SSH command on VM7Cent6VCL1: '/usr/sbin/userdel
> admin',
> > |exit status: 255, output: 22905|44:44|timeout| ssh output
> (/usr/sbin/...):
> > |ssh: connect to host VM7Cent6VCL1 port 22: No route to host
> > |22905|44:44|timeout| ( 0) utils.pm, run_ssh_command (line: 5006)
> > |22905|44:44|timeout| (-1) Linux.pm, delete_user (line: 881)
> > |22905|44:44|timeout| (-2) Linux.pm, sanitize (line: 1177)
> > |22905|44:44|timeout| (-3) reclaim.pm, call_os_sanitize (line: 271)
> > |22905|44:44|timeout| (-4) reclaim.pm, process (line: 162)
> > |22905|44:44|timeout| (-5) vcld, make_new_child (line: 571)
> > |22905|44:44|timeout| ---- WARNING ----
> > |22905|44:44|timeout| 2015-05-17
> > |08:35:46|22905|44:44|timeout|utils.pm:run_ssh_command(5043)|failed to
> run
> > |SSH command after 3 attempts, command: /usr/bin/ssh -i /etc/vcl/vcl.key
> > |-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o
> > |ConnectionAttempts=1 -o ConnectTimeout=3 -l root -p 22 -x VM7Cent6VCL1
> > |'/usr/sbin/userdel admin' 2>&1, exit status: 255, output:
> > |22905|44:44|timeout| ssh output (/usr/sbin/...): ssh: connect to host
> > |VM7Cent6VCL1 port 22: No route to host 22905|44:44|timeout| ( 0)
> utils.pm,
> > |run_ssh_command (line: 5043)
> > |22905|44:44|timeout| (-1) Linux.pm, delete_user (line: 881)
> > |22905|44:44|timeout| (-2) Linux.pm, sanitize (line: 1177)
> > |22905|44:44|timeout| (-3) reclaim.pm, call_os_sanitize (line: 271)
> > |22905|44:44|timeout| (-4) reclaim.pm, process (line: 162)
> > |22905|44:44|timeout| (-5) vcld, make_new_child (line: 571)
> >
> > 2015-05-17 08:35:46|22905|44:44|timeout|utils.pm:run_ssh_command(4902)|
> executing SSH command on VM7Cent6VCL1:
> > |22905|44:44|timeout| /usr/bin/ssh -i /etc/vcl/vcl.key  -o
> > |StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o
> > |ConnectionAttempts=1 -o ConnectTimeout=3 -l root -p 22 -x VM7Cent6VCL1
> > |'sed -i -e "/^AllowUsers .*/d" /etc/ssh/external_sshd_config' 2>&1
> > |22905|44:44|timeout| ---- WARNING ----
> > |22905|44:44|timeout| 2015-05-17
> > |08:35:46|22905|44:44|timeout|utils.pm:run_ssh_command(5006)|attempt
> 1/3:
> > |failed to execute SSH command on VM7Cent6VCL1: 'sed -i -e "/^AllowUsers
> > |.*/d" /etc/ssh/external_sshd_config', exit status: 255, output:
> > |22905|44:44|timeout| ssh output (sed -i -e ...): ssh: connect to host
> > |VM7Cent6VCL1 port 22: No route to host 22905|44:44|timeout| ( 0)
> utils.pm,
> > |run_ssh_command (line: 5006)
> > |22905|44:44|timeout| (-1) Linux.pm, delete_user (line: 890)
> > |22905|44:44|timeout| (-2) Linux.pm, sanitize (line: 1177)
> > |22905|44:44|timeout| (-3) reclaim.pm, call_os_sanitize (line: 271)
> > |22905|44:44|timeout| (-4) reclaim.pm, process (line: 162)
> > |22905|44:44|timeout| (-5) vcld, make_new_child (line: 571)
> >
> > 2015-05-17
> > 08:35:46|22905|44:44|timeout|utils.pm:run_ssh_command(4894)|sleeping
> for 2
> > seconds before making next SSH attempt
> > 2015-05-17 08:35:48|22905|44:44|timeout|utils.pm:run_ssh_command(4906)|
> attempt 2/3: executing SSH command on VM7Cent6VCL1:
> > |22905|44:44|timeout| /usr/bin/ssh -i /etc/vcl/vcl.key  -o
> > |StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o
> > |ConnectionAttempts=1 -o ConnectTimeout=3 -l root -p 22 -x VM7Cent6VCL1
> > |'sed -i -e "/^AllowUsers .*/d" /etc/ssh/external_sshd_config' 2>&1
> > |22905|44:44|timeout| ---- WARNING ----
> > |22905|44:44|timeout| 2015-05-17
> > |08:35:48|22905|44:44|timeout|utils.pm:run_ssh_command(5006)|attempt
> 2/3:
> > |failed to execute SSH command on VM7Cent6VCL1: 'sed -i -e "/^AllowUsers
> > |.*/d" /etc/ssh/external_sshd_config', exit status: 255, output:
> > |22905|44:44|timeout| ssh output (sed -i -e ...): ssh: connect to host
> > |VM7Cent6VCL1 port 22: No route to host 22905|44:44|timeout| ( 0)
> utils.pm,
> > |run_ssh_command (line: 5006)
> > |22905|44:44|timeout| (-1) Linux.pm, delete_user (line: 890)
> > |22905|44:44|timeout| (-2) Linux.pm, sanitize (line: 1177)
> > |22905|44:44|timeout| (-3) reclaim.pm, call_os_sanitize (line: 271)
> > |22905|44:44|timeout| (-4) reclaim.pm, process (line: 162)
> > |22905|44:44|timeout| (-5) vcld, make_new_child (line: 571)
> >
> > 2015-05-17
> > 08:35:48|22905|44:44|timeout|utils.pm:run_ssh_command(4894)|sleeping
> for 4
> > seconds before making next SSH attempt
> > 2015-05-17 08:35:52|22905|44:44|timeout|utils.pm:run_ssh_command(4906)|
> attempt 3/3: executing SSH command on VM7Cent6VCL1:
> > |22905|44:44|timeout| /usr/bin/ssh -i /etc/vcl/vcl.key  -o
> > |StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o
> > |ConnectionAttempts=1 -o ConnectTimeout=3 -l root -p 22 -x VM7Cent6VCL1
> > |'sed -i -e "/^AllowUsers .*/d" /etc/ssh/external_sshd_config' 2>&1
> > |22905|44:44|timeout| ---- WARNING ----
> > |22905|44:44|timeout| 2015-05-17
> > |08:35:52|22905|44:44|timeout|utils.pm:run_ssh_command(5006)|attempt
> 3/3:
> > |failed to execute SSH command on VM7Cent6VCL1: 'sed -i -e "/^AllowUsers
> > |.*/d" /etc/ssh/external_sshd_config', exit status: 255, output:
> > |22905|44:44|timeout| ssh output (sed -i -e ...): ssh: connect to host
> > |VM7Cent6VCL1 port 22: No route to host 22905|44:44|timeout| ( 0)
> utils.pm,
> > |run_ssh_command (line: 5006)
> > |22905|44:44|timeout| (-1) Linux.pm, delete_user (line: 890)
> > |22905|44:44|timeout| (-2) Linux.pm, sanitize (line: 1177)
> > |22905|44:44|timeout| (-3) reclaim.pm, call_os_sanitize (line: 271)
> > |22905|44:44|timeout| (-4) reclaim.pm, process (line: 162)
> > |22905|44:44|timeout| (-5) vcld, make_new_child (line: 571)
> > |22905|44:44|timeout| ---- WARNING ----
> > |22905|44:44|timeout| 2015-05-17
> > |08:35:52|22905|44:44|timeout|utils.pm:run_ssh_command(5043)|failed to
> run
> > |SSH command after 3 attempts, command: /usr/bin/ssh -i /etc/vcl/vcl.key
> > |-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o
> > |ConnectionAttempts=1 -o ConnectTimeout=3 -l root -p 22 -x VM7Cent6VCL1
> > |'sed -i -e "/^AllowUsers .*/d" /etc/ssh/external_sshd_config' 2>&1, exit
> > |status: 255, output: 22905|44:44|timeout| ssh output (sed -i -e ...):
> ssh:
> > |connect to host VM7Cent6VCL1 port 22: No route to host
> > |22905|44:44|timeout| ( 0) utils.pm, run_ssh_command (line: 5043)
> > |22905|44:44|timeout| (-1) Linux.pm, delete_user (line: 890)
> > |22905|44:44|timeout| (-2) Linux.pm, sanitize (line: 1177)
> > |22905|44:44|timeout| (-3) reclaim.pm, call_os_sanitize (line: 271)
> > |22905|44:44|timeout| (-4) reclaim.pm, process (line: 162)
> > |22905|44:44|timeout| (-5) vcld, make_new_child (line: 571)
> >
> > 2015-05-17
> >
> 08:35:52|22905|44:44|timeout|DataStructure.pm:get_computer_private_ip_addre
> > ss(1641)|attempting to retrieve private IP address for computer:
> > VM7Cent6VCL1 2015-05-17
> > 08:35:52|22905|44:44|timeout|OS.pm:get_file_contents(1897)|retrieved 15
> > lines from file on localhost: '/etc/hosts' 2015-05-17
> >
> 08:35:52|22905|44:44|timeout|DataStructure.pm:get_computer_private_ip_addre
> > ss(1697)|returning IP address from /etc/hosts file: 192.168.10.48
> 2015-05-17
> > 08:35:52|22905|44:44|timeout|utils.pm:is_inblockrequest(5760)|zero rows
> > were returned from database select 2015-05-17
> >
> 08:35:52|22905|44:44|timeout|DataStructure.pm:get_image_affiliation_name(21
> > 21)|image owner id: 1 2015-05-17
> >
> 08:35:52|22905|44:44|timeout|DataStructure.pm:retrieve_user_data(1412)|atte
> > mpting to retrieve and store data for user: user.id = '1' 2015-05-17
> >
> 08:35:52|22905|44:44|timeout|DataStructure.pm:retrieve_user_data(1475)|data
> > has been retrieved for user: admin (id: 1)
> - --
> - -------------------------------
> Josh Thompson
> VCL Developer
> North Carolina State University
>
> my GPG/PGP key can be found at pgp.mit.edu
>
> All electronic mail messages in connection with State business which
> are sent to or received by this account are subject to the NC Public
> Records Law and may be disclosed to third parties.
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2
>
> iD8DBQFVWcsOV/LQcNdtPQMRAtezAJ0Sw6V2YJ9msiQ8o50Rwbe21fgkQACeKeme
> gQWeyHN1fzwbqDZ0Vw5aW+o=
> =I6Ta
> -----END PGP SIGNATURE-----
>
>

Re: Fw: PROBLEM -- localhost|44:44|timeout|Linux.pm|VM7Cent6VCL1>VM7|vmwarelinux-vm7c2testvcl212-v0|admin

Posted by Josh Thompson <jo...@ncsu.edu>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Lewis,

Looking through vcld.log will allow you to see every ssh command that vcld 
sends to a node.

Also, let me explain the process VCL uses for provisioning.  We use something 
we call "hot standby" to help minimize the amount of time it takes for 
reservations to become ready for users to connect to their reservations.  At 
the end of each reservation, the node is reloaded so that it is almost ready 
for another user.  Then, when another user makes a reservation for it, only a 
few steps need to be done to finish provisioning it.  The image selected to be 
reloaded on the node depends on what Predictive Reload module is assigned.  
Previous to 2.4.2, Predictive Reload modules were assigned per management 
node, meaning all computers managed by a specific management node would be 
handled the same way.  With 2.4.2, it is assigned per computer, meaning each 
computer can be handled differently.  Also, 2.4.2 introduced a new module that 
doesn't actually reload it, but instead powers it off, and for VMs, 
unregisters it.

All that to say, the behavior you are seeing may be that the node has been 
reloaded at the end of your reservation to prep for another one.

One more point to note is that VCL randomizes the root/admin password when it 
provisions each node.  You can find the password it selects for each 
reservation in the vcld.log file.

Josh

On Sun May 17 2015 1:09:04 PM Litchfield, Elwin L. wrote:
> The VM was powered on as scheduled, I logged on to the VM using admin & PW
> supplied.  Logged off & the VM stayed powered on after the reservation
> expired.  Ping to the VM works, ICMP Type 0 Code 0.   ICMP Type 3 Code 10
> to Port 22 reply is Host Administratively prohibited.
> 
> From the VM console, on the VMware host,  no administrator user is able to
> logon.  I have added 2 more admin users & they will not work either.  The
> admin logon supplied from the reservation worked, but ssh was not
> active/displayed as a service.
> 
> I have a Wireshark 1.8.10 trace of the private network conversation.  I am
> researching how to upgrade to the current release of Wireshark, but I have
> not found any information that Wireshark will decode SSH.
> 
> Is there any trace to run to determine what is taking place between the
> Management Node & the VM?
> 
> Thanks
> 
> Lewis Litchfield
> Technology Services – Viking Center Lab
> University of South Florida Sarasota-Manatee
> 941-359-4231
> 
> ________________________________________
> From: root@localhost.localdomain <ro...@localhost.localdomain>
> Sent: Sunday, May 17, 2015 8:35 AM
> To: lewis15@mail.usf.edu
> Subject: PROBLEM --
> localhost|44:44|timeout|Linux.pm|VM7Cent6VCL1>VM7|vmwarelinux-vm7c2testvcl2
> 12-v0|admin
> 
> failed to add AllowUsers admin to external_sshd_config
> ------------------------------------------------------------------------
> time: 2015-05-17 08:35:52
> caller: Linux.pm:delete_user(894)
> ( 0) Linux.pm, delete_user (line: 894)
> (-1) Linux.pm, sanitize (line: 1177)
> (-2) reclaim.pm, call_os_sanitize (line: 271)
> (-3) reclaim.pm, process (line: 162)
> (-4) vcld, make_new_child (line: 571)
> (-5) vcld, main (line: 350)
> ------------------------------------------------------------------------
> management node: localhost
> reservation PID: 22905
> parent vcld PID: 26123
> 
> request ID: 44
> reservation ID: 44
> request state/laststate: timeout/reserved
> request start time: 2015-05-17 08:15:00
> request end time: 2015-05-17 09:00:00
> for imaging: no
> log ID: 51
> 
> computer: VM7Cent6VCL1
> computer id: 6
> computer type: virtualmachine
> computer eth0 MAC address: 00:50:56:3F:40:E3
> computer eth1 MAC address: 00:50:56:3F:40:ED
> computer private IP address: 192.168.10.48
> computer public IP address: 131.247.29.132
> computer in block allocation: no
> provisioning module: VCL::Module::Provisioning::VMware::VMware
> 
> vm host: VM7
> vm host ID: 2
> vm host computer ID: 5
> vm profile: VMware ESXi - local storage
> vm profile VM path: datastore1
> vm profile repository path: <undefined>
> vm profile datastore path: datastore1
> vm profile disk type: shared
> 
> image: vmwarelinux-vm7c2testvcl212-v0
> image display name: vm7c2testvcl2
> image ID: 12
> image revision ID: 12
> image size: 5762 MB
> use Sysprep: yes
> root access: yes
> image owner ID: 1
> image owner affiliation: Local
> image revision date created: 2015-05-01 09:43:57
> image revision production: yes
> OS module: VCL::Module::OS::Linux
> 
> user: admin
> user name: vcl admin
> user ID: 1
> user affiliation: Local
> ------------------------------------------------------------------------
> 
> RECENT LOG ENTRIES FOR THIS PROCESS:
> |22905|44:44|timeout| (-1) Linux.pm, delete_user (line: 871)
> |22905|44:44|timeout| (-2) Linux.pm, sanitize (line: 1177)
> |22905|44:44|timeout| (-3) reclaim.pm, call_os_sanitize (line: 271)
> |22905|44:44|timeout| (-4) reclaim.pm, process (line: 162)
> |22905|44:44|timeout| (-5) vcld, make_new_child (line: 571)
> 
> 2015-05-17 08:35:40|22905|44:44|timeout|utils.pm:run_ssh_command(4902)|
executing SSH command on VM7Cent6VCL1:
> |22905|44:44|timeout| /usr/bin/ssh -i /etc/vcl/vcl.key  -o
> |StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o
> |ConnectionAttempts=1 -o ConnectTimeout=3 -l root -p 22 -x VM7Cent6VCL1
> |'/usr/sbin/userdel admin' 2>&1 22905|44:44|timeout| ---- WARNING ----
> |22905|44:44|timeout| 2015-05-17
> |08:35:40|22905|44:44|timeout|utils.pm:run_ssh_command(5006)|attempt 1/3:
> |failed to execute SSH command on VM7Cent6VCL1: '/usr/sbin/userdel admin',
> |exit status: 255, output: 22905|44:44|timeout| ssh output (/usr/sbin/...):
> |ssh: connect to host VM7Cent6VCL1 port 22: No route to host
> |22905|44:44|timeout| ( 0) utils.pm, run_ssh_command (line: 5006)
> |22905|44:44|timeout| (-1) Linux.pm, delete_user (line: 881)
> |22905|44:44|timeout| (-2) Linux.pm, sanitize (line: 1177)
> |22905|44:44|timeout| (-3) reclaim.pm, call_os_sanitize (line: 271)
> |22905|44:44|timeout| (-4) reclaim.pm, process (line: 162)
> |22905|44:44|timeout| (-5) vcld, make_new_child (line: 571)
> 
> 2015-05-17
> 08:35:40|22905|44:44|timeout|utils.pm:run_ssh_command(4894)|sleeping for 2
> seconds before making next SSH attempt
> 2015-05-17 08:35:42|22905|44:44|timeout|utils.pm:run_ssh_command(4906)|
attempt 2/3: executing SSH command on VM7Cent6VCL1:
> |22905|44:44|timeout| /usr/bin/ssh -i /etc/vcl/vcl.key  -o
> |StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o
> |ConnectionAttempts=1 -o ConnectTimeout=3 -l root -p 22 -x VM7Cent6VCL1
> |'/usr/sbin/userdel admin' 2>&1 22905|44:44|timeout| ---- WARNING ----
> |22905|44:44|timeout| 2015-05-17
> |08:35:42|22905|44:44|timeout|utils.pm:run_ssh_command(5006)|attempt 2/3:
> |failed to execute SSH command on VM7Cent6VCL1: '/usr/sbin/userdel admin',
> |exit status: 255, output: 22905|44:44|timeout| ssh output (/usr/sbin/...):
> |ssh: connect to host VM7Cent6VCL1 port 22: No route to host
> |22905|44:44|timeout| ( 0) utils.pm, run_ssh_command (line: 5006)
> |22905|44:44|timeout| (-1) Linux.pm, delete_user (line: 881)
> |22905|44:44|timeout| (-2) Linux.pm, sanitize (line: 1177)
> |22905|44:44|timeout| (-3) reclaim.pm, call_os_sanitize (line: 271)
> |22905|44:44|timeout| (-4) reclaim.pm, process (line: 162)
> |22905|44:44|timeout| (-5) vcld, make_new_child (line: 571)
> 
> 2015-05-17
> 08:35:42|22905|44:44|timeout|utils.pm:run_ssh_command(4894)|sleeping for 4
> seconds before making next SSH attempt
> 2015-05-17 08:35:46|22905|44:44|timeout|utils.pm:run_ssh_command(4906)|
attempt 3/3: executing SSH command on VM7Cent6VCL1:
> |22905|44:44|timeout| /usr/bin/ssh -i /etc/vcl/vcl.key  -o
> |StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o
> |ConnectionAttempts=1 -o ConnectTimeout=3 -l root -p 22 -x VM7Cent6VCL1
> |'/usr/sbin/userdel admin' 2>&1 22905|44:44|timeout| ---- WARNING ----
> |22905|44:44|timeout| 2015-05-17
> |08:35:46|22905|44:44|timeout|utils.pm:run_ssh_command(5006)|attempt 3/3:
> |failed to execute SSH command on VM7Cent6VCL1: '/usr/sbin/userdel admin',
> |exit status: 255, output: 22905|44:44|timeout| ssh output (/usr/sbin/...):
> |ssh: connect to host VM7Cent6VCL1 port 22: No route to host
> |22905|44:44|timeout| ( 0) utils.pm, run_ssh_command (line: 5006)
> |22905|44:44|timeout| (-1) Linux.pm, delete_user (line: 881)
> |22905|44:44|timeout| (-2) Linux.pm, sanitize (line: 1177)
> |22905|44:44|timeout| (-3) reclaim.pm, call_os_sanitize (line: 271)
> |22905|44:44|timeout| (-4) reclaim.pm, process (line: 162)
> |22905|44:44|timeout| (-5) vcld, make_new_child (line: 571)
> |22905|44:44|timeout| ---- WARNING ----
> |22905|44:44|timeout| 2015-05-17
> |08:35:46|22905|44:44|timeout|utils.pm:run_ssh_command(5043)|failed to run
> |SSH command after 3 attempts, command: /usr/bin/ssh -i /etc/vcl/vcl.key 
> |-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o
> |ConnectionAttempts=1 -o ConnectTimeout=3 -l root -p 22 -x VM7Cent6VCL1
> |'/usr/sbin/userdel admin' 2>&1, exit status: 255, output:
> |22905|44:44|timeout| ssh output (/usr/sbin/...): ssh: connect to host
> |VM7Cent6VCL1 port 22: No route to host 22905|44:44|timeout| ( 0) utils.pm,
> |run_ssh_command (line: 5043)
> |22905|44:44|timeout| (-1) Linux.pm, delete_user (line: 881)
> |22905|44:44|timeout| (-2) Linux.pm, sanitize (line: 1177)
> |22905|44:44|timeout| (-3) reclaim.pm, call_os_sanitize (line: 271)
> |22905|44:44|timeout| (-4) reclaim.pm, process (line: 162)
> |22905|44:44|timeout| (-5) vcld, make_new_child (line: 571)
> 
> 2015-05-17 08:35:46|22905|44:44|timeout|utils.pm:run_ssh_command(4902)|
executing SSH command on VM7Cent6VCL1:
> |22905|44:44|timeout| /usr/bin/ssh -i /etc/vcl/vcl.key  -o
> |StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o
> |ConnectionAttempts=1 -o ConnectTimeout=3 -l root -p 22 -x VM7Cent6VCL1
> |'sed -i -e "/^AllowUsers .*/d" /etc/ssh/external_sshd_config' 2>&1
> |22905|44:44|timeout| ---- WARNING ----
> |22905|44:44|timeout| 2015-05-17
> |08:35:46|22905|44:44|timeout|utils.pm:run_ssh_command(5006)|attempt 1/3:
> |failed to execute SSH command on VM7Cent6VCL1: 'sed -i -e "/^AllowUsers
> |.*/d" /etc/ssh/external_sshd_config', exit status: 255, output:
> |22905|44:44|timeout| ssh output (sed -i -e ...): ssh: connect to host
> |VM7Cent6VCL1 port 22: No route to host 22905|44:44|timeout| ( 0) utils.pm,
> |run_ssh_command (line: 5006)
> |22905|44:44|timeout| (-1) Linux.pm, delete_user (line: 890)
> |22905|44:44|timeout| (-2) Linux.pm, sanitize (line: 1177)
> |22905|44:44|timeout| (-3) reclaim.pm, call_os_sanitize (line: 271)
> |22905|44:44|timeout| (-4) reclaim.pm, process (line: 162)
> |22905|44:44|timeout| (-5) vcld, make_new_child (line: 571)
> 
> 2015-05-17
> 08:35:46|22905|44:44|timeout|utils.pm:run_ssh_command(4894)|sleeping for 2
> seconds before making next SSH attempt
> 2015-05-17 08:35:48|22905|44:44|timeout|utils.pm:run_ssh_command(4906)|
attempt 2/3: executing SSH command on VM7Cent6VCL1:
> |22905|44:44|timeout| /usr/bin/ssh -i /etc/vcl/vcl.key  -o
> |StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o
> |ConnectionAttempts=1 -o ConnectTimeout=3 -l root -p 22 -x VM7Cent6VCL1
> |'sed -i -e "/^AllowUsers .*/d" /etc/ssh/external_sshd_config' 2>&1
> |22905|44:44|timeout| ---- WARNING ----
> |22905|44:44|timeout| 2015-05-17
> |08:35:48|22905|44:44|timeout|utils.pm:run_ssh_command(5006)|attempt 2/3:
> |failed to execute SSH command on VM7Cent6VCL1: 'sed -i -e "/^AllowUsers
> |.*/d" /etc/ssh/external_sshd_config', exit status: 255, output:
> |22905|44:44|timeout| ssh output (sed -i -e ...): ssh: connect to host
> |VM7Cent6VCL1 port 22: No route to host 22905|44:44|timeout| ( 0) utils.pm,
> |run_ssh_command (line: 5006)
> |22905|44:44|timeout| (-1) Linux.pm, delete_user (line: 890)
> |22905|44:44|timeout| (-2) Linux.pm, sanitize (line: 1177)
> |22905|44:44|timeout| (-3) reclaim.pm, call_os_sanitize (line: 271)
> |22905|44:44|timeout| (-4) reclaim.pm, process (line: 162)
> |22905|44:44|timeout| (-5) vcld, make_new_child (line: 571)
> 
> 2015-05-17
> 08:35:48|22905|44:44|timeout|utils.pm:run_ssh_command(4894)|sleeping for 4
> seconds before making next SSH attempt
> 2015-05-17 08:35:52|22905|44:44|timeout|utils.pm:run_ssh_command(4906)|
attempt 3/3: executing SSH command on VM7Cent6VCL1:
> |22905|44:44|timeout| /usr/bin/ssh -i /etc/vcl/vcl.key  -o
> |StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o
> |ConnectionAttempts=1 -o ConnectTimeout=3 -l root -p 22 -x VM7Cent6VCL1
> |'sed -i -e "/^AllowUsers .*/d" /etc/ssh/external_sshd_config' 2>&1
> |22905|44:44|timeout| ---- WARNING ----
> |22905|44:44|timeout| 2015-05-17
> |08:35:52|22905|44:44|timeout|utils.pm:run_ssh_command(5006)|attempt 3/3:
> |failed to execute SSH command on VM7Cent6VCL1: 'sed -i -e "/^AllowUsers
> |.*/d" /etc/ssh/external_sshd_config', exit status: 255, output:
> |22905|44:44|timeout| ssh output (sed -i -e ...): ssh: connect to host
> |VM7Cent6VCL1 port 22: No route to host 22905|44:44|timeout| ( 0) utils.pm,
> |run_ssh_command (line: 5006)
> |22905|44:44|timeout| (-1) Linux.pm, delete_user (line: 890)
> |22905|44:44|timeout| (-2) Linux.pm, sanitize (line: 1177)
> |22905|44:44|timeout| (-3) reclaim.pm, call_os_sanitize (line: 271)
> |22905|44:44|timeout| (-4) reclaim.pm, process (line: 162)
> |22905|44:44|timeout| (-5) vcld, make_new_child (line: 571)
> |22905|44:44|timeout| ---- WARNING ----
> |22905|44:44|timeout| 2015-05-17
> |08:35:52|22905|44:44|timeout|utils.pm:run_ssh_command(5043)|failed to run
> |SSH command after 3 attempts, command: /usr/bin/ssh -i /etc/vcl/vcl.key 
> |-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o
> |ConnectionAttempts=1 -o ConnectTimeout=3 -l root -p 22 -x VM7Cent6VCL1
> |'sed -i -e "/^AllowUsers .*/d" /etc/ssh/external_sshd_config' 2>&1, exit
> |status: 255, output: 22905|44:44|timeout| ssh output (sed -i -e ...): ssh:
> |connect to host VM7Cent6VCL1 port 22: No route to host
> |22905|44:44|timeout| ( 0) utils.pm, run_ssh_command (line: 5043)
> |22905|44:44|timeout| (-1) Linux.pm, delete_user (line: 890)
> |22905|44:44|timeout| (-2) Linux.pm, sanitize (line: 1177)
> |22905|44:44|timeout| (-3) reclaim.pm, call_os_sanitize (line: 271)
> |22905|44:44|timeout| (-4) reclaim.pm, process (line: 162)
> |22905|44:44|timeout| (-5) vcld, make_new_child (line: 571)
> 
> 2015-05-17
> 08:35:52|22905|44:44|timeout|DataStructure.pm:get_computer_private_ip_addre
> ss(1641)|attempting to retrieve private IP address for computer:
> VM7Cent6VCL1 2015-05-17
> 08:35:52|22905|44:44|timeout|OS.pm:get_file_contents(1897)|retrieved 15
> lines from file on localhost: '/etc/hosts' 2015-05-17
> 08:35:52|22905|44:44|timeout|DataStructure.pm:get_computer_private_ip_addre
> ss(1697)|returning IP address from /etc/hosts file: 192.168.10.48 2015-05-17
> 08:35:52|22905|44:44|timeout|utils.pm:is_inblockrequest(5760)|zero rows
> were returned from database select 2015-05-17
> 08:35:52|22905|44:44|timeout|DataStructure.pm:get_image_affiliation_name(21
> 21)|image owner id: 1 2015-05-17
> 08:35:52|22905|44:44|timeout|DataStructure.pm:retrieve_user_data(1412)|atte
> mpting to retrieve and store data for user: user.id = '1' 2015-05-17
> 08:35:52|22905|44:44|timeout|DataStructure.pm:retrieve_user_data(1475)|data
> has been retrieved for user: admin (id: 1)
- -- 
- -------------------------------
Josh Thompson
VCL Developer
North Carolina State University

my GPG/PGP key can be found at pgp.mit.edu

All electronic mail messages in connection with State business which
are sent to or received by this account are subject to the NC Public
Records Law and may be disclosed to third parties.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iD8DBQFVWcsOV/LQcNdtPQMRAtezAJ0Sw6V2YJ9msiQ8o50Rwbe21fgkQACeKeme
gQWeyHN1fzwbqDZ0Vw5aW+o=
=I6Ta
-----END PGP SIGNATURE-----