You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Nezar Madbouh <ne...@ipscape.com.au> on 2016/04/14 02:16:21 UTC
Re: Unable to create a New VM - Error message: Unable to start
instance due to Unable to get answer that is of class com.cloud.agent.api.StartAnswer
Hi Simon,
The following work was done but unfortunately the problem is not fixed:
on management server:
service cloudstack-management stop
on the host:
service cloudstack-agent stop
service libvirtd restart
service cloudstack-agent start
on management server:
service cloudstack-management start
Regards,
Nezar.
On 24 March 2016 at 05:06, Simon Weller <sw...@ena.com> wrote:
> Yes, I think that order is fine. Make sure the processes are indeed
> shutdown before moving on to the next step.
>
>
>
>
>
> ________________________________________
> From: Nezar Madbouh <ne...@ipscape.com.au>
> Sent: Tuesday, March 22, 2016 5:33 PM
> To: users@cloudstack.apache.org
> Subject: Re: Unable to create a New VM - Error message: Unable to start
> instance due to Unable to get answer that is of class
> com.cloud.agent.api.StartAnswer
>
> Hi Simon,
>
> Can you please check if the following sequence of action is correct:
>
> on management server:
>
> service cloudstack-management stop
>
>
> on the host:
>
> service cloudstack-agent stop
>
> service libvirtd restart
>
> service cloudstack-agent start
>
>
> on management server:
>
> service cloudstack-management start
>
> Thanks very much for your help.
> Nezar.
>
> On 23 March 2016 at 08:39, Simon Weller <sw...@ena.com> wrote:
>
> > Shutting down all the management services should clear the jobs.
> >
> >
> > ________________________________________
> > From: Nezar Madbouh <ne...@ipscape.com.au>
> > Sent: Tuesday, March 22, 2016 4:28 PM
> > To: users@cloudstack.apache.org
> > Subject: Re: Unable to create a New VM - Error message: Unable to start
> > instance due to Unable to get answer that is of class
> > com.cloud.agent.api.StartAnswer
> >
> > Hi Simon,
> >
> > I will plan to restart agents, libvirtd and management services.
> >
> > How do I clear hung jobs.
> >
> > Your help is much appreciated,
> > Nezar.
> >
> > On 22 March 2016 at 23:03, Simon Weller <sw...@ena.com> wrote:
> >
> > > Nezar,
> > >
> > > I've looked through the code related to your release and it seems that
> in
> > > order to throw this exception, 10 attempts have been made to retrieve
> the
> > > disk information prior to the exception being thrown.
> > >
> > > It appears there is something really odd going on here. I'm starting
> to
> > > wonder whether it might be worth while for you to stop all of your
> > agents,
> > > plus management server processes, restart libvirtd on each of the hosts
> > and
> > > then bring the agents back up prior to starting the management servers.
> > >
> > > There are a bunch of logs the pastebin you posted that seems to
> indicate
> > > you have some jobs hung, for example:
> > >
> > > 2016-03-14 21:45:31,828 WARN [o.a.c.f.j.i.AsyncJobMonitor]
> > > (Timer-1:ctx-0c090bd7) Task (job-9066) has been pending for 1660851
> > seconds
> > > 2016-03-14 21:45:31,828 WARN [o.a.c.f.j.i.AsyncJobMonitor]
> > > (Timer-1:ctx-0c090bd7) Task (job-8133) has been pending for 4253003
> > seconds
> > >
> > > Restarting the agents, libvirtd and management will not cause an
> outage
> > > to your cloud, but I would suggest, if you decided to do this, you
> should
> > > complete it in a maintenance window.
> > >
> > > - Si
> > >
> > > ________________________________________
> > > From: Nezar Madbouh <ne...@ipscape.com.au>
> > > Sent: Sunday, March 20, 2016 10:27 PM
> > > To: users@cloudstack.apache.org
> > > Subject: Re: Unable to create a New VM - Error message: Unable to start
> > > instance due to Unable to get answer that is of class
> > > com.cloud.agent.api.StartAnswer
> > >
> > > Hi Simon,
> > >
> > > Please see answers to your questions below. I am not sure if I provided
> > the
> > > correct answer for the primary and secondary storage configuration.
> > Please
> > > let me know if this is not what you are after:
> > >
> > >
> > > Can you screenshot and post somewhere your secondary storage
> > configuration
> > > and your primary storage configuration from the UI with the details?
> > >
> > > mysql> select * from configuration where name like '%secondary%'\G
> > > *************************** 1. row ***************************
> > > category: Account Defaults
> > > instance: DEFAULT
> > > component: management-server
> > > name: max.account.secondary.storage
> > > value: 400
> > > description: The default maximum secondary storage space (in GiB)
> that
> > > can be used for an account
> > > default_value: 400
> > > updated: NULL
> > > scope: NULL
> > > is_dynamic: 0
> > > *************************** 2. row ***************************
> > > category: Project Defaults
> > > instance: DEFAULT
> > > component: management-server
> > > name: max.project.secondary.storage
> > > value: 400
> > > description: The default maximum secondary storage space (in GiB)
> that
> > > can be used for a project
> > > default_value: 400
> > > updated: NULL
> > > scope: NULL
> > > is_dynamic: 0
> > > *************************** 3. row ***************************
> > > category: Hidden
> > > instance: DEFAULT
> > > component: management-server
> > > name: secondary.storage.vm
> > > value: +rwRhJOXPymbQOuAMLnhiw==
> > > description: Deploys a VM per zone to manage secondary storage if
> true,
> > > otherwise secondary storage is mounted on management server
> > > default_value: NULL
> > > updated: NULL
> > > scope: NULL
> > > is_dynamic: 0
> > > *************************** 4. row ***************************
> > > category: Advanced
> > > instance: DEFAULT
> > > component: ExtensionRegistry
> > > name: secondary.storage.vm.allocators.exclude
> > > value: NULL
> > > description: Extensions to exclude from being registered
> > > default_value: NULL
> > > updated: 2014-12-04 00:18:15
> > > scope: Global
> > > is_dynamic: 0
> > > *************************** 5. row ***************************
> > > category: NetworkManager
> > > instance: DEFAULT
> > > component: management-server
> > > name: vm.network.nic.max.secondary.ipaddresses
> > > value: NULL
> > > description: Specify the number of secondary ip addresses per nic per
> > vm
> > > default_value: 256
> > > updated: NULL
> > > scope: NULL
> > > is_dynamic: 0
> > > 5 rows in set (0.00 sec)
> > >
> > >
> > > mysql> select * from configuration where name like '%primary%'\G
> > > *************************** 1. row ***************************
> > > category: Account Defaults
> > > instance: DEFAULT
> > > component: management-server
> > > name: max.account.primary.storage
> > > value: 200
> > > description: The default maximum primary storage space (in GiB) that
> > can
> > > be used for an account
> > > default_value: 200
> > > updated: NULL
> > > scope: NULL
> > > is_dynamic: 0
> > > *************************** 2. row ***************************
> > > category: Project Defaults
> > > instance: DEFAULT
> > > component: management-server
> > > name: max.project.primary.storage
> > > value: 200
> > > description: The default maximum primary storage space (in GiB) that
> > can
> > > be used for a project
> > > default_value: 200
> > > updated: NULL
> > > scope: NULL
> > > is_dynamic: 0
> > > *************************** 3. row ***************************
> > > category: Storage
> > > instance: DEFAULT
> > > component: TemplateManager
> > > name: primary.storage.download.wait
> > > value: 10800
> > > description: In second, timeout for download template to primary
> > storage
> > > default_value: NULL
> > > updated: NULL
> > > scope: NULL
> > > is_dynamic: 0
> > > 3 rows in set (0.00 sec)
> > >
> > >
> > >
> > > Also, what is the Cloudstack version you are running and what is the OS
> > > you're running it on?
> > >
> > > Cloudstack 4.4.1
> > >
> > > Centos 6.6
> > >
> > > Are you using actimeo=0 in your NFS mount line?
> > >
> > > Primary storage on Hypervisors has the following parameters
> > > 10.61.154.10:/Cloudstack/primary on
> > > /mnt/b3647480-5709-36eb-9ddd-e2c666f30b57 type nfs
> > > (rw,noatime,nfsvers=3,proto=tcp,hard,nolock,intr,addr=10.61.154.10)
> > >
> > >
> > > Secondary storage on SSVM has the following parameters
> > > 10.61.154.10:/Cloudstack/secondary on
> > > /mnt/SecStorage/d11d5c92-6950-30e8-a0b5-3cd46a17ec14 type nfs4
> > >
> > >
> >
> (rw,relatime,vers=4,rsize=262144,wsize=262144,namlen=255,acdirmin=0,acdirmax=0,soft,proto=tcp,port=0,timeo=133,retrans=2147483647,sec=sys,clientaddr=10.61.154.70,minorversion=0,local_lock=none,addr=10.61.154.10)
> > >
> > >
> > > Thanks & Regards,
> > > Nezar.
> > >
> > > On 18 March 2016 at 23:50, Simon Weller <sw...@ena.com> wrote:
> > >
> > > > Also, what is the Cloudstack version you are running and what is the
> OS
> > > > you're running it on?
> > > >
> > > > Are you using actimeo=0 in your NFS mount line?
> > > >
> > > > ________________________________________
> > > > From: Simon Weller <sw...@ena.com>
> > > > Sent: Friday, March 18, 2016 7:40 AM
> > > > To: users@cloudstack.apache.org
> > > > Subject: Re: Unable to create a New VM - Error message: Unable to
> start
> > > > instance due to Unable to get answer that is of class
> > > > com.cloud.agent.api.StartAnswer
> > > >
> > > > Nexar,
> > > >
> > > > That's normal behaviour. You can only ssh to the link local on the
> host
> > > > where the VM resides.
> > > >
> > > > Can you screenshot and post somewhere your secondary storage
> > > configuration
> > > > and your primary storage configuration from the UI with the details?
> > > >
> > > > - Si
> > > >
> > > > ________________________________________
> > > > From: Nezar Madbouh <ne...@ipscape.com.au>
> > > > Sent: Thursday, March 17, 2016 8:19 PM
> > > > To: users@cloudstack.apache.org
> > > > Subject: Re: Unable to create a New VM - Error message: Unable to
> start
> > > > instance due to Unable to get answer that is of class
> > > > com.cloud.agent.api.StartAnswer
> > > >
> > > > Hi Simon,
> > > >
> > > > I just noticed that from other Hosts I can't ssh into the SSVM:
> > > >
> > > > [root@eqx-cs-cmp-01 ~]# ssh -i /root/.ssh/id_rsa.cloud -p 3922
> > > > root@169.254.3.161
> > > > ssh: connect to host 169.254.3.161 port 3922: No route to host
> > > >
> > > >
> > > > I can only ssh from one Host:
> > > > [root@eqx-cs-cmp-06 ~]# ssh -i /root/.ssh/id_rsa.cloud -p 3922
> > > > root@169.254.3.161
> > > > Linux s-11-VM 3.2.0-4-amd64 #1 SMP Debian 3.2.57-3 x86_64
> > > >
> > > > Thanks & Regards,
> > > > Nezar.
> > > >
> > > > On 18 March 2016 at 10:36, Nezar Madbouh <
> nezar.madbouh@ipscape.com.au
> > >
> > > > wrote:
> > > >
> > > > > Hi Simon,
> > > > >
> > > > > Yes existing VMs are successfully running. Yesterday I was able to
> > > create
> > > > > a VM in Dev Environment but I can't create VMs in Prod environment.
> > > > >
> > > > > Thanks,
> > > > > Nezar.
> > > > >
> > > > > On 18 March 2016 at 09:12, Simon Weller <sw...@ena.com> wrote:
> > > > >
> > > > >> I'm assuming you have other VMs successfully running, correct? So
> > you
> > > > can
> > > > >> write to your primary storage.
> > > > >>
> > > > >>
> > > > >>
> > > > >> ________________________________________
> > > > >> From: Nezar Madbouh <ne...@ipscape.com.au>
> > > > >> Sent: Thursday, March 17, 2016 4:43 PM
> > > > >> To: users@cloudstack.apache.org
> > > > >> Subject: Re: Unable to create a New VM - Error message: Unable to
> > > start
> > > > >> instance due to Unable to get answer that is of class
> > > > >> com.cloud.agent.api.StartAnswer
> > > > >>
> > > > >> Hi Simon and Shweta,
> > > > >>
> > > > >> Please see the result below:
> > > > >>
> > > > >> root@s-11-VM:~# /usr/local/cloud/systemvm/ssvm-check.sh
> > > > >> ================================================
> > > > >> First DNS server is 8.8.8.8
> > > > >> PING 8.8.8.8 (8.8.8.8): 48 data bytes
> > > > >> 56 bytes from 8.8.8.8: icmp_seq=0 ttl=57 time=5.209 ms
> > > > >> 56 bytes from 8.8.8.8: icmp_seq=1 ttl=57 time=0.936 ms
> > > > >> --- 8.8.8.8 ping statistics ---
> > > > >> 2 packets transmitted, 2 packets received, 0% packet loss
> > > > >> round-trip min/avg/max/stddev = 0.936/3.072/5.209/2.137 ms
> > > > >> Good: Can ping DNS server
> > > > >> ================================================
> > > > >> Good: DNS resolves download.cloud.com
> > > > >> ================================================
> > > > >> nfs is currently mounted
> > > > >> Mount point is
> /mnt/SecStorage/d11d5c92-6950-30e8-a0b5-3cd46a17ec14
> > > > >> Good: Can write to mount point
> > > > >> ================================================
> > > > >> Management server is 10.61.153.20. Checking connectivity.
> > > > >> Good: Can connect to management server port 8250
> > > > >> ================================================
> > > > >> Good: Java process is running
> > > > >> ================================================
> > > > >> Tests Complete. Look for ERROR or WARNING above.
> > > > >> root@s-11-VM:~#
> > > > >>
> > > > >> Thanks & Regards,
> > > > >> Nezar.
> > > > >>
> > > > >> On 17 March 2016 at 09:42, Simon Weller <sw...@ena.com> wrote:
> > > > >>
> > > > >> > Nezar,
> > > > >> >
> > > > >> > Log into the Cloudstack UI.
> > > > >> >
> > > > >> > Click on "Infrastructure" on the left menu. Click on View All
> > below
> > > > >> > "System VMs". find the item marked "Secondary Storage VM" and
> > then
> > > > >> click
> > > > >> > on the Name of that VM on the left (it will be s-XXXX-VM).
> You'll
> > > see
> > > > a
> > > > >> > bunch of information about the VM. Find the Host line and also
> the
> > > > Link
> > > > >> > Local IP Address line and then follow the original instructions
> in
> > > my
> > > > >> > previous email.
> > > > >> >
> > > > >> > - Si
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > ________________________________________
> > > > >> > From: Shweta Agarwal <sh...@accelerite.com>
> > > > >> > Sent: Wednesday, March 16, 2016 12:16 AM
> > > > >> > To: users@cloudstack.apache.org
> > > > >> > Subject: Re: Unable to create a New VM - Error message: Unable
> to
> > > > start
> > > > >> > instance due to Unable to get answer that is of class
> > > > >> > com.cloud.agent.api.StartAnswer
> > > > >> >
> > > > >> > Hi Nezar,
> > > > >> >
> > > > >> > You can find host name on detail view of SSVM in UI .
> > > > >> > You can also use list api command listSystemVms to get the host
> > > > details
> > > > >> of
> > > > >> > system vms.
> > > > >> >
> > > > >> > Thanks
> > > > >> > Shweta
> > > > >> >
> > > > >> >
> > > > >> > ________________________________________
> > > > >> > From: Nezar Madbouh <ne...@ipscape.com.au>
> > > > >> > Sent: Wednesday, March 16, 2016 9:58 AM
> > > > >> > To: users@cloudstack.apache.org
> > > > >> > Subject: Re: Unable to create a New VM - Error message: Unable
> to
> > > > start
> > > > >> > instance due to Unable to get answer that is of class
> > > > >> > com.cloud.agent.api.StartAnswer
> > > > >> >
> > > > >> > Hi Simon,
> > > > >> >
> > > > >> > How can I find out which host the SSVM is on?
> > > > >> >
> > > > >> > Thanks & Regards,
> > > > >> > Nezar.
> > > > >> >
> > > > >> > On 16 March 2016 at 12:28, Simon Weller <sw...@ena.com>
> wrote:
> > > > >> >
> > > > >> > > Can you ssh into the Secondary Storage VM?
> > > > >> > >
> > > > >> > > This can be done by getting the link-local-ip address from the
> > > > >> > > infrastructure menu in the UI and then sshing into the host
> the
> > > SSVM
> > > > >> is
> > > > >> > on
> > > > >> > > and issuing this command as root: ssh -i
> /root/.ssh/id_rsa.cloud
> > > -p
> > > > >> 3922
> > > > >> > > root@link-local-ip (replace link-local-ip with the link
> local
> > > ip).
> > > > >> > >
> > > > >> > > Try running the SSVM diangnostic script:
> > > > >> > > usr/local/cloud/systemvm/ssvm-check.sh
> > > > >> > >
> > > > >> > > - Si
> > > > >> > > ________________________________________
> > > > >> > > From: Nezar Madbouh <ne...@ipscape.com.au>
> > > > >> > > Sent: Tuesday, March 15, 2016 7:49 PM
> > > > >> > > To: users@cloudstack.apache.org
> > > > >> > > Subject: Re: Unable to create a New VM - Error message: Unable
> > to
> > > > >> start
> > > > >> > > instance due to Unable to get answer that is of class
> > > > >> > > com.cloud.agent.api.StartAnswer
> > > > >> > >
> > > > >> > > Hi Simon,
> > > > >> > >
> > > > >> > > Please see details below:
> > > > >> > >
> > > > >> > >
> > > > >> > > mysql> use cloud; select id, name, status, type from host
> where
> > > id =
> > > > >> 5;
> > > > >> > > Reading table information for completion of table and column
> > names
> > > > >> > > You can turn off this feature to get a quicker startup with -A
> > > > >> > >
> > > > >> > > Database changed
> > > > >> > > +----+------------------------------+--------+---------+
> > > > >> > > | id | name | status | type |
> > > > >> > > +----+------------------------------+--------+---------+
> > > > >> > > | 5 | eqx-cs-cmp-02.ipscape.com.au | up | routing |
> > > > >> > > +----+------------------------------+--------+---------+
> > > > >> > > 1 row in set (0.00 sec)
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > > cat /var/log/cloudstack/agent/agent.log
> > > > >> > > com.cloud.utils.exception.CloudRuntimeException:
> > > > >> > > com.cloud.utils.exception.CloudRuntimeException: Can't find
> > > > >> > > volume:089ff3ca-9bff-402c-a14e-ea7a35aaad5c
> > > > >> > > at
> > > > >> > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.getPhysicalDisk(KVMStoragePoolManager.java:262)
> > > > >> > > at
> > > > >> > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.createVbd(LibvirtComputingResource.java:3921)
> > > > >> > > at
> > > > >> > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(LibvirtComputingResource.java:3789)
> > > > >> > > at
> > > > >> > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1333)
> > > > >> > > at
> com.cloud.agent.Agent.processRequest(Agent.java:501)
> > > > >> > > at
> > > > >> > com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:808)
> > > > >> > > at com.cloud.utils.nio.Task.run(Task.java:84)
> > > > >> > > at
> > > > >> > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > > > >> > > at
> > > > >> > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > > > >> > > at java.lang.Thread.run(Thread.java:745)
> > > > >> > > agent.log.2016-03-03.gz (END)
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > > [root@eqx-cs-cmp-02 agent]# virsh pool-list
> > > > >> > > Name State Autostart
> > > > >> > > -----------------------------------------
> > > > >> > > 2d0967f8-50d6-4d6d-87b7-4da0596f6924 active no
> > > > >> > > b3647480-5709-36eb-9ddd-e2c666f30b57 active no
> > > > >> > > e3831246-ff55-4af4-9964-a17565edaccc active no
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > > [Root @ eqx-cs-cmp-02 agent] # df -h
> > > > >> > > Filesystem Size Used Avail Use% Mounted on
> > > > >> > > /dev/mapper/vg_root-lv_root
> > > > >> > > 203G 215G 1.3G 1% /
> > > > >> > > tmpfs 48g 48g 0 0% / dev / shm
> > > > >> > > / Dev / sda1 477m 427m 25M 6% / boot
> > > > >> > > xx.xx.xx.xx:/Cloudstack/primary
> > > > >> > > 7.6T 4.1T 3.5T 54%
> > > > >> > > /mnt/b3647480-5709-36eb-9ddd-e2c666f30b57
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > > I also noticed the following in cloudstack-agent.err
> > > > >> > >
> > > > >> > > libvirt: QEMU Driver error : Domain not found: no domain with
> > > > matching
> > > > >> > name
> > > > >> > > 'i-11-600-VM'
> > > > >> > > libvirt: Storage Driver error : Storage volume not found: no
> > > storage
> > > > >> vol
> > > > >> > > with matching name '089ff3ca-9bff-402c-a14e-ea7a35aaad5c'
> > > > >> > >
> > > > >> > > Your help is much appreciated,
> > > > >> > > Nezar
> > > > >> > >
> > > > >> > >
> > > > >> > > On 16 March 2016 at 11:03, Simon Weller <sw...@ena.com>
> > wrote:
> > > > >> > >
> > > > >> > > > Nezar,
> > > > >> > > >
> > > > >> > > > ok, I just took a look at a pastebin you posted.
> > > > >> > > >
> > > > >> > > > I see it complaining about the inability to find a volume
> > prior
> > > to
> > > > >> > > > throwing the host exception.
> > > > >> > > >
> > > > >> > > > Firstly, let's find out who host 5 is. Can you log into your
> > > MySQL
> > > > >> > server
> > > > >> > > > and run this:
> > > > >> > > >
> > > > >> > > > use cloud;select id,name,status,type from host where id=5;
> > > > >> > > >
> > > > >> > > > Can you post the results?
> > > > >> > > >
> > > > >> > > > If it's a host, can you post the
> > > > /var/log/cloudstack/agent/agent.log
> > > > >> > from
> > > > >> > > > that host?
> > > > >> > > >
> > > > >> > > > And also run virsh pool-list on the host in question?
> > > > >> > > >
> > > > >> > > > - Si
> > > > >> > > >
> > > > >> > > >
> > > > >> > > >
> > > > >> > > >
> > > > >> > > >
> > > > >> > > > ________________________________________
> > > > >> > > > From: Nezar Madbouh <ne...@ipscape.com.au>
> > > > >> > > > Sent: Tuesday, March 15, 2016 6:11 PM
> > > > >> > > > To: users@cloudstack.apache.org
> > > > >> > > > Subject: Re: Unable to create a New VM - Error message:
> Unable
> > > to
> > > > >> start
> > > > >> > > > instance due to Unable to get answer that is of class
> > > > >> > > > com.cloud.agent.api.StartAnswer
> > > > >> > > >
> > > > >> > > > Hi Simon,
> > > > >> > > >
> > > > >> > > > When you say "double check your routers tied to the
> network",
> > do
> > > > you
> > > > >> > mean
> > > > >> > > > the physical routers or the virtual routers?
> > > > >> > > >
> > > > >> > > > When I try to create a VM in Prod Environment it fails. I
> > can't
> > > > ping
> > > > >> > the
> > > > >> > > > Prod virtual router from management server or Host.
> > > > >> > > >
> > > > >> > > > But I can ping Dev virtual router from management server and
> > > Host
> > > > >> and I
> > > > >> > > was
> > > > >> > > > able to create a VM in Dev Environment.
> > > > >> > > >
> > > > >> > > > How can I troubleshoot the Prod Virtual router. I launched
> the
> > > > >> console
> > > > >> > > but
> > > > >> > > > couldn't login.
> > > > >> > > >
> > > > >> > > >
> > > > >> > > > Please note that we need to take into consideration that the
> > > > >> affected
> > > > >> > > > network is the Production network and we need to minimise
> > > service
> > > > >> > outage
> > > > >> > > as
> > > > >> > > > much as possible.
> > > > >> > > >
> > > > >> > > > Your help is much appreciated,
> > > > >> > > > Nezar
> > > > >> > > >
> > > > >> > > >
> > > > >> > > >
> > > > >> > > > On 16 March 2016 at 08:37, Simon Weller <sw...@ena.com>
> > > wrote:
> > > > >> > > >
> > > > >> > > > > I've seen this before where a Virtual Router has crashed
> and
> > > the
> > > > >> > > > > management server can't communicate with it.
> > > > >> > > > > Are you using an advanced network setup?
> > > > >> > > > >
> > > > >> > > > > If so, double check your routers tied to the network where
> > you
> > > > are
> > > > >> > > > > attempting to start an instance.
> > > > >> > > > >
> > > > >> > > > > - Si
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > > ________________________________________
> > > > >> > > > > From: Nezar Madbouh <ne...@ipscape.com.au>
> > > > >> > > > > Sent: Tuesday, March 15, 2016 4:02 PM
> > > > >> > > > > To: users@cloudstack.apache.org
> > > > >> > > > > Subject: Re: Unable to create a New VM - Error message:
> > Unable
> > > > to
> > > > >> > start
> > > > >> > > > > instance due to Unable to get answer that is of class
> > > > >> > > > > com.cloud.agent.api.StartAnswer
> > > > >> > > > >
> > > > >> > > > > Hi Glenn,
> > > > >> > > > >
> > > > >> > > > > We have 16 hosts, all KVM.
> > > > >> > > > >
> > > > >> > > > > The error is not specific to Host 5. Every time I try to
> > > create
> > > > a
> > > > >> VM
> > > > >> > it
> > > > >> > > > > gives me the same error message but on different host.
> > > > >> > > > >
> > > > >> > > > > How do I restart the agent on the host. Is it service
> > > affecting?
> > > > >> > > > >
> > > > >> > > > > Thanks very much for your help.
> > > > >> > > > > Neza
> > > > >> > > > >
> > > > >> > > > > On 15 March 2016 at 17:37, Glenn Wagner <
> > > > >> glenn.wagner@shapeblue.com>
> > > > >> > > > > wrote:
> > > > >> > > > >
> > > > >> > > > > > Hi,
> > > > >> > > > > >
> > > > >> > > > > > Can you explain your environment first , how many hosts
> ,
> > > what
> > > > >> is
> > > > >> > > your
> > > > >> > > > > > hypervisors (KVM,Xenserver,VMware)
> > > > >> > > > > >
> > > > >> > > > > > The error seems to be with your Host 5 Hypervisor
> > > > >> > > > > > Resource [Host:5] is unreachable:
> > > > >> > > > > >
> > > > >> > > > > > Go to the infrastructure section and see what host5 is
> > > > >> reporting,
> > > > >> > > > > > Have you tried to restart the agent on the Host?
> > > > >> > > > > >
> > > > >> > > > > > Thanks
> > > > >> > > > > > Glenn
> > > > >> > > > > >
> > > > >> > > > > >
> > > > >> > > > > >
> > > > >> > > > > >
> > > > >> > > > > > [image: ShapeBlue] <http://www.shapeblue.com>
> > > > >> > > > > > Glenn Wagner
> > > > >> > > > > > Senior Consultant , ShapeBlue
> > > > >> > > > > > d: * | s: +27 21 527 0091*
> > > > >> <%7C%20s:%20+27%2021%20527%200091> |
> > > > >> > > m:
> > > > >> > > > > > *+27 73 917 4111* <+27%2073%20917%204111>
> > > > >> > > > > > e: *glenn.wagner@shapeblue.com | t: *
> > > > >> > > > > > <glenn.wagner@shapeblue.com%20%7C%20t:> | w: *
> > > > >> www.shapeblue.com
> > > > >> > *
> > > > >> > > > > > <http://www.shapeblue.com>
> > > > >> > > > > > a:
> > > > >> > > > > > 2nd Floor, Oudehuis Centre, 122 Main Rd, Somerset West
> > Cape
> > > > Town
> > > > >> > 7130
> > > > >> > > > > South Africa
> > > > >> > > > > > Shape Blue Ltd is a company incorporated in England &
> > Wales.
> > > > >> > > ShapeBlue
> > > > >> > > > > > Services India LLP is a company incorporated in India
> and
> > is
> > > > >> > operated
> > > > >> > > > > under
> > > > >> > > > > > license from Shape Blue Ltd. Shape Blue Brasil
> Consultoria
> > > > Ltda
> > > > >> is
> > > > >> > a
> > > > >> > > > > > company incorporated in Brasil and is operated under
> > license
> > > > >> from
> > > > >> > > Shape
> > > > >> > > > > > Blue Ltd. ShapeBlue SA Pty Ltd is a company registered
> by
> > > The
> > > > >> > > Republic
> > > > >> > > > of
> > > > >> > > > > > South Africa and is traded under license from Shape Blue
> > > Ltd.
> > > > >> > > ShapeBlue
> > > > >> > > > > is
> > > > >> > > > > > a registered trademark.
> > > > >> > > > > > This email and any attachments to it may be confidential
> > and
> > > > are
> > > > >> > > > intended
> > > > >> > > > > > solely for the use of the individual to whom it is
> > > addressed.
> > > > >> Any
> > > > >> > > views
> > > > >> > > > > or
> > > > >> > > > > > opinions expressed are solely those of the author and do
> > not
> > > > >> > > > necessarily
> > > > >> > > > > > represent those of Shape Blue Ltd or related companies.
> If
> > > you
> > > > >> are
> > > > >> > > not
> > > > >> > > > > the
> > > > >> > > > > > intended recipient of this email, you must neither take
> > any
> > > > >> action
> > > > >> > > > based
> > > > >> > > > > > upon its contents, nor copy or show it to anyone. Please
> > > > contact
> > > > >> > the
> > > > >> > > > > sender
> > > > >> > > > > > if you believe you have received this email in error.
> > > > >> > > > > >
> > > > >> > > > > >
> > > > >> > > > > > -----Original Message-----
> > > > >> > > > > > From: Nezar Madbouh [mailto:
> nezar.madbouh@ipscape.com.au]
> > > > >> > > > > > Sent: Tuesday, 15 March 2016 5:45 AM
> > > > >> > > > > > To: users@cloudstack.apache.org
> > > > >> > > > > > Subject: Re: Unable to create a New VM - Error message:
> > > Unable
> > > > >> to
> > > > >> > > start
> > > > >> > > > > > instance due to Unable to get answer that is of class
> > > > >> > > > > > com.cloud.agent.api.StartAnswer
> > > > >> > > > > >
> > > > >> > > > > > Hi Jeremy,
> > > > >> > > > > >
> > > > >> > > > > > I have uploaded the contents to
> > > http://pastebin.com/1J3mmgTh
> > > > >> > > > > >
> > > > >> > > > > > your help is much appreciated,
> > > > >> > > > > > Nezar.
> > > > >> > > > > >
> > > > >> > > > > > On 15 March 2016 at 13:46, Jeremy Peterson <
> > > > >> jpeterson@acentek.net>
> > > > >> > > > > wrote:
> > > > >> > > > > >
> > > > >> > > > > > > I don't see any attachments can you create a pastebin.
> > > > That's
> > > > >> > > easier
> > > > >> > > > > > > to look through as I think attachments are removed
> > before
> > > > >> being
> > > > >> > > sent
> > > > >> > > > > > > to user group.
> > > > >> > > > > > >
> > > > >> > > > > > > Jeremy
> > > > >> > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > > > -------- Original message --------
> > > > >> > > > > > > From: Nezar Madbouh <ne...@ipscape.com.au>
> > > > >> > > > > > > Date: 3/14/2016 6:38 PM (GMT-06:00)
> > > > >> > > > > > > To: users@cloudstack.apache.org
> > > > >> > > > > > > Subject: Unable to create a New VM - Error message:
> > Unable
> > > > to
> > > > >> > start
> > > > >> > > > > > > instance due to Unable to get answer that is of class
> > > > >> > > > > > > com.cloud.agent.api.StartAnswer
> > > > >> > > > > > >
> > > > >> > > > > > > Hi Shweta and Sadhu,
> > > > >> > > > > > >
> > > > >> > > > > > > Please find attached managerment-server.log which
> > contains
> > > > the
> > > > >> > logs
> > > > >> > > > > > > since I started creating a new VM (VM id 600).
> > > > >> > > > > > >
> > > > >> > > > > > > Please note that we don't have agent.log files
> > > > >> > > > > > >
> > > > >> > > > > > > Your help is much appreciated.
> > > > >> > > > > > >
> > > > >> > > > > > > Regards,
> > > > >> > > > > > > Nezar.
> > > > >> > > > > > >
> > > > >> > > > > > Find out more about ShapeBlue and our range of
> CloudStack
> > > > >> related
> > > > >> > > > > services:
> > > > >> > > > > > IaaS Cloud Design & Build
> > > > >> > > > > > <http://shapeblue.com/iaas-cloud-design-and-build//> |
> > > > CSForge
> > > > >> -
> > > > >> > > rapid
> > > > >> > > > > > IaaS deployment framework <
> http://shapeblue.com/csforge/>
> > > > >> > > > > > CloudStack Consulting <
> > > > >> > http://shapeblue.com/cloudstack-consultancy/>
> > > > >> > > |
> > > > >> > > > > CloudStack
> > > > >> > > > > > Software Engineering
> > > > >> > > > > > <http://shapeblue.com/cloudstack-software-engineering/>
> > > > >> > > > > > CloudStack Infrastructure Support
> > > > >> > > > > > <
> http://shapeblue.com/cloudstack-infrastructure-support/>
> > |
> > > > >> > > CloudStack
> > > > >> > > > > > Bootcamp Training Courses <
> > > > >> > http://shapeblue.com/cloudstack-training/
> > > > >> > > >
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > DISCLAIMER
> > > > >> > ==========
> > > > >> > This e-mail may contain privileged and confidential information
> > > which
> > > > is
> > > > >> > the property of Accelerite, a Persistent Systems business. It is
> > > > >> intended
> > > > >> > only for the use of the individual or entity to which it is
> > > addressed.
> > > > >> If
> > > > >> > you are not the intended recipient, you are not authorized to
> > read,
> > > > >> retain,
> > > > >> > copy, print, distribute or use this message. If you have
> received
> > > this
> > > > >> > communication in error, please notify the sender and delete all
> > > copies
> > > > >> of
> > > > >> > this message. Accelerite, a Persistent Systems business does not
> > > > accept
> > > > >> any
> > > > >> > liability for virus infected mails.
> > > > >> >
> > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>
>
Re: Unable to create a New VM - Error message: Unable to start
instance due to Unable to get answer that is of class com.cloud.agent.api.StartAnswer
Posted by Nezar Madbouh <ne...@ipscape.com.au>.
Hi Ilya,
Please see answers to your questions below:
Primary Storage is configured to use local disk on the Hosts:
Below are examples of 2 entries in "Infrastructure --> Primary Storage" in
UI.
Name: eqx-cs-cmp-01.ipscape.com.au Local storage
Server: 10.61.153.22
Path: /var/lib/libvirtd/images (this is local disk on the host)
cluster: eqx-cs-cluster-01
Scope: Host
Name: eqx-cs-cmp-02.ipscape.com.au Local storage
Server: 10.61.153.23
Path: /var/lib/libvirtd/images (this is local disk on the host)
cluster: eqx-cs-cluster-01
Scope: Host
and so on for all 16 Hosts
Agent logs for eqx-cs-cmp-12
-------------------------------------------
logs are pasted in
http://pastebin.com/tiMMhEzW
Mount points:
--------------------
Below are mount points from 2 Hosts:
[root@eqx-cs-cmp-12 agent]# mount
/dev/mapper/vg_root-lv_root on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
/dev/sda1 on /boot type ext4 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
nfsd on /proc/fs/nfsd type nfsd (rw)
10.61.154.10:/Cloudstack/primary on
/mnt/b3647480-5709-36eb-9ddd-e2c666f30b57 type nfs
(rw,noatime,nfsvers=3,proto=tcp,hard,nolock,intr,noac,actimeo=0,addr=10.61.154.10)
[root@eqx-cs-cmp-01 ~]# mount
/dev/mapper/vg_root-lv_root on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
/dev/sda1 on /boot type ext4 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
nfsd on /proc/fs/nfsd type nfsd (rw)
10.61.154.10:/Cloudstack/primary on
/mnt/b3647480-5709-36eb-9ddd-e2c666f30b57 type nfs
(rw,noatime,nfsvers=3,proto=tcp,hard,nolock,intr,addr=10.61.154.10)
Your help is much appreciated,
Nezar.
On 16 April 2016 at 03:32, ilya <il...@gmail.com> wrote:
> Here is what i need to understand, if you are using SharedMountPoint,
> then /mnt/storage must be preset on all KVM hosts in a persistent manner
> (i.e. fstab).
>
> However, if you are using NFS, you can just let cloudstack manage the
> NFS mount points by adding the storage as NFS type.
>
> On 4/15/16 12:46 AM, ilya wrote:
> > Can you goto primary storage in cloudstack UI, is this what you have
> > defined for your NFS Server?
> >
> > "SharedMountPoint","host":"localhost","path":"/mnt/storage"
> >
> > If so, please use IP or FQDN instead..
> >
> >
> > On 4/14/16 10:39 PM, Nezar Madbouh wrote:
> >> Hi Ilya,
> >>
> >> Here is the output of mount command:
> >>
> >> [root@eqx-cs-cmp-01 ~]# mount
> >> /dev/mapper/vg_root-lv_root on / type ext4 (rw)
> >> proc on /proc type proc (rw)
> >> sysfs on /sys type sysfs (rw)
> >> devpts on /dev/pts type devpts (rw,gid=5,mode=620)
> >> tmpfs on /dev/shm type tmpfs (rw)
> >> /dev/sda1 on /boot type ext4 (rw)
> >> none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
> >> sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
> >> nfsd on /proc/fs/nfsd type nfsd (rw)
> >> 10.61.154.10:/Cloudstack/primary on
> >> /mnt/b3647480-5709-36eb-9ddd-e2c666f30b57 type nfs
> >> (rw,noatime,nfsvers=3,proto=tcp,hard,nolock,intr,addr=10.61.154.10)
> >> [root@eqx-cs-cmp-01 ~]#
> >>
> >>
> >> On 15 April 2016 at 15:22, Nezar Madbouh <ne...@ipscape.com.au>
> >> wrote:
> >>
> >>> Hi Ilya,
> >>>
> >>> Thanks for the instructions. I followed them and pasted the
> information on
> >>> pastebin:
> >>>
> >>> management-server logs job-14938
> >>> http://pastebin.com/GAiy8u5H
> >>>
> >>> Agent logs
> >>> http://pastebin.com/mfiADsJc
> >>>
> >>>
> >>> Your help is much appreciated.
> >>>
> >>> Nezar.
> >>>
> >>> On 15 April 2016 at 13:58, ilya <il...@gmail.com> wrote:
> >>>
> >>>> Here is how i would troubleshoot this:
> >>>>
> >>>> 1) ssh to MS host, run
> >>>>
> >>>> tail -f /var/cloudstack/management/management-server.log | grep -i
> >>>> StartVMCmdByAdmin
> >>>>
> >>>> 2) on cloudstack UI, press start on the VM and watch the tail log
> >>>>
> >>>> 3) on the tail log, you should see a StartVirtualMachine api call show
> >>>> up with job ID. For example:
> >>>>
> >>>>> 2016-04-15 03:41:57,193 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>>> (API-Job-Executor-24:ctx-7892387a job-291575) Executing AsyncJobVO
> >>>> {id:291575, userId: 157, accountId: 2, instanceType: VirtualMachine,
> >>>> instanceId: 7648, cmd:
> >>>> org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin, cmdInfo:
> >>>>
> {"id":"7648","response":"json","sessionkey":"mMJsduS9KLKApqH4dPpIlGZdt1Y\u003d","ctxDetails":"{\"com.cloud.vm.VirtualMachine\":7648,\"com.cloud.uservm.UserVm\":7648,\"com.cloud.network.router.VirtualRouter\":7648}","cmdEventType":"VM.START","expires":"2016-04-15T03:51:57+0000","ctxUserId":"157","signatureversion":"3","httpmethod":"GET","uuid":"7648","ctxAccountId":"2","ctxStartEventId":"314416"},
> >>>> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
> >>>> result: null, initMsid: 345051337894, completeMsid: null, lastUpdated:
> >>>> null, lastPolled: null, created: null}
> >>>>
> >>>> Each task in cloudstack spins up its own job, in this case its
> >>>> *job-291575*
> >>>>
> >>>> 4) now execute
> >>>>> grep -A3 -B3 job-291575
> >>>> /var/cloudstack/management/management-server.log | tee job-291575.log
> >>>>> grep job-291575 /var/cloudstack/management/management-server.log |
> grep
> >>>> via | head -1
> >>>>
> >>>> you should see a response like so,
> >>>>
> >>>>> 2016-04-15 03:41:59,259 DEBUG [c.c.a.m.ClusteredAgentAttache]
> >>>> (Work-Job-Executor-42:ctx-b2c4af6b job-291575/job-291576
> ctx-0988152e) Seq
> >>>> 441-4181310779036795138: Forwarding Seq 441-4181310779036795138: {
> Cmd ,
> >>>> MgmtId: 345051337894, via: 441(my-kvm-host01.example.com), Ver: v1,
> >>>> Flags: 100011,
> >>>>
> [{"com.cloud.agent.api.routing.DhcpEntryCommand":{"vmMac":"06:fe:4e:00:12:ad","vmIpAddress":"10.10.27.178","vmName":"ol6-test-ilya1","defaultRouter":"10.10.27.1","defaultDns":"10.17.27.166","duid":"00:03:00:01:06:fe:4e:00:12:ad","isDefault":true,"executeInSequence":false,"accessDetails":{"zone.network.type":"Advanced","router.guest.ip":"10.10.27.166","router.ip":"169.254.0.238","
> >>>> router.name":"r-7301-VM"},"wait":0}}] } to 345051337893
> >>>>
> >>>> We are looking for this line "via: 441(my-kvm-host01.example.com)"
> >>>>
> >>>> SSH to my-kvm-host01.example.com, and copy the
> >>>> /var/log/cloudstack/agent.log to pastebin
> >>>>
> >>>> also copy job-291575.log to pastebin
> >>>>
> >>>> I'll try to have a look, though when i get chance.
> >>>>
> >>>>
> >>>> Also paste the output of mount command on kvm host in question. I'm
> >>>> suspecting one of your storage pools maybe bad.
> >>>>
> >>>> Lastly, what distro are you running for KVM?
> >>>>
> >>>> Regards
> >>>> ilya
> >>>>
> >>>> On 4/14/16 5:05 PM, Nezar Madbouh wrote:
> >>>>> Hi Ilya,
> >>>>>
> >>>>> I shutdown all the VMs in Pre-prod Environmnet expecting the Pre-prod
> >>>>> Virtual router to be deleted automatically because no VM is running
> on
> >>>> this
> >>>>> vlan but it wasn't deleted.
> >>>>>
> >>>>> Am I right to expect the VR to be deleted automatically if no VM is
> >>>> running
> >>>>> in that vlan?
> >>>>>
> >>>>> Thanks,
> >>>>> Nezar.
> >>>>>
> >>>>> On 15 April 2016 at 09:53, Nezar Madbouh <
> nezar.madbouh@ipscape.com.au>
> >>>>> wrote:
> >>>>>
> >>>>>> Hi Ilya,
> >>>>>>
> >>>>>> I have run the script and it didn't return any results:
> >>>>>>
> >>>>>> [root@eqx-cs-mgt-01 ~]# ./async_jobs.sh
> >>>>>> [root@eqx-cs-mgt-01 ~]#
> >>>>>>
> >>>>>>
> >>>>>> I also ran the command inside mysql:
> >>>>>>
> >>>>>> mysql> SELECT id, user_id, account_id, instance_type, instance_id,
> >>>>>> job_cmd, job_cmd_info, job_dispatcher, job_pending_signals FROM
> >>>>>> cloud.async_job where job_status=0 and job_dispatcher not like
> >>>>>> "pseudoJobDispatcher";
> >>>>>> Empty set (0.00 sec)
> >>>>>>
> >>>>>> your help is much appreciated,
> >>>>>> Nezar.
> >>>>>>
> >>>>>>
> >>>>>> On 14 April 2016 at 16:49, ilya <il...@gmail.com>
> wrote:
> >>>>>>
> >>>>>>> It appears you have pending jobs that are not letting you go
> through.
> >>>>>>>
> >>>>>>> Usually stop and starting management service should be enough.
> >>>>>>>
> >>>>>>> However, try running this script from your management server to
> show
> >>>> us
> >>>>>>> what you have running in async_job table.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>> #!/bin/bash
> >>>>>>>> DATESTAMP=$(date +%m%d%y-%H%M%S)
> >>>>>>>> DBPASS=$(java -classpath
> >>>>>>> /usr/share/cloudstack-common/lib/jasypt-1.9.0.jar
> >>>>>>> org.jasypt.intf.cli.JasyptPBEStringDecryptionCLI input="$(cat
> >>>>>>> /etc/cloudstack/management/db.properties | grep db.cloud.password |
> >>>> awk
> >>>>>>> -F'(' '{print $2}' | sed 's/)//g')" password="$(cat
> >>>>>>> /etc/cloudstack/management/key)" | grep -A2 OUTPUT | tail -1)
> >>>>>>>> DBHOST=$(cat /etc/cloudstack/management/db.properties | grep
> >>>>>>> db.cloud.host | awk -F'=' '{print $2}' | tail -1 )
> >>>>>>>> DBUSER=$(cat /etc/cloudstack/management/db.properties | grep
> >>>>>>> db.cloud.username | awk -F'=' '{print $2}')
> >>>>>>>> DB=$(cat /etc/cloudstack/management/db.properties | grep
> >>>> db.cloud.name
> >>>>>>> | awk -F'=' '{print $2}')
> >>>>>>>> DBPORT=$(cat /etc/cloudstack/management/db.properties | grep
> >>>>>>> db.cloud.port | awk -F'=' '{print $2}')
> >>>>>>>> MYSQLCMD="mysql -h $DBHOST -u $DBUSER -P $DBPORT -p$DBPASS $DB"
> >>>>>>>> echo 'SELECT id, user_id, account_id, instance_type, instance_id,
> >>>>>>> job_cmd, job_cmd_info, job_dispatcher, job_pending_signals FROM
> >>>>>>> cloud.async_job where job_status=0 and job_dispatcher not like
> >>>>>>> "pseudoJobDispatcher"' | $MYSQLCMD
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On 4/13/16 5:16 PM, Nezar Madbouh wrote:
> >>>>>>>> Hi Simon,
> >>>>>>>>
> >>>>>>>> The following work was done but unfortunately the problem is not
> >>>> fixed:
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> on management server:
> >>>>>>>>
> >>>>>>>> service cloudstack-management stop
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> on the host:
> >>>>>>>>
> >>>>>>>> service cloudstack-agent stop
> >>>>>>>>
> >>>>>>>> service libvirtd restart
> >>>>>>>>
> >>>>>>>> service cloudstack-agent start
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> on management server:
> >>>>>>>>
> >>>>>>>> service cloudstack-management start
> >>>>>>>>
> >>>>>>>> Regards,
> >>>>>>>> Nezar.
> >>>>>>>>
> >>>>>>>> On 24 March 2016 at 05:06, Simon Weller <sw...@ena.com> wrote:
> >>>>>>>>
> >>>>>>>>> Yes, I think that order is fine. Make sure the processes are
> indeed
> >>>>>>>>> shutdown before moving on to the next step.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> ________________________________________
> >>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
> >>>>>>>>> Sent: Tuesday, March 22, 2016 5:33 PM
> >>>>>>>>> To: users@cloudstack.apache.org
> >>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable to
> >>>> start
> >>>>>>>>> instance due to Unable to get answer that is of class
> >>>>>>>>> com.cloud.agent.api.StartAnswer
> >>>>>>>>>
> >>>>>>>>> Hi Simon,
> >>>>>>>>>
> >>>>>>>>> Can you please check if the following sequence of action is
> correct:
> >>>>>>>>>
> >>>>>>>>> on management server:
> >>>>>>>>>
> >>>>>>>>> service cloudstack-management stop
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> on the host:
> >>>>>>>>>
> >>>>>>>>> service cloudstack-agent stop
> >>>>>>>>>
> >>>>>>>>> service libvirtd restart
> >>>>>>>>>
> >>>>>>>>> service cloudstack-agent start
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> on management server:
> >>>>>>>>>
> >>>>>>>>> service cloudstack-management start
> >>>>>>>>>
> >>>>>>>>> Thanks very much for your help.
> >>>>>>>>> Nezar.
> >>>>>>>>>
> >>>>>>>>> On 23 March 2016 at 08:39, Simon Weller <sw...@ena.com> wrote:
> >>>>>>>>>
> >>>>>>>>>> Shutting down all the management services should clear the jobs.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> ________________________________________
> >>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
> >>>>>>>>>> Sent: Tuesday, March 22, 2016 4:28 PM
> >>>>>>>>>> To: users@cloudstack.apache.org
> >>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable
> to
> >>>>>>> start
> >>>>>>>>>> instance due to Unable to get answer that is of class
> >>>>>>>>>> com.cloud.agent.api.StartAnswer
> >>>>>>>>>>
> >>>>>>>>>> Hi Simon,
> >>>>>>>>>>
> >>>>>>>>>> I will plan to restart agents, libvirtd and management services.
> >>>>>>>>>>
> >>>>>>>>>> How do I clear hung jobs.
> >>>>>>>>>>
> >>>>>>>>>> Your help is much appreciated,
> >>>>>>>>>> Nezar.
> >>>>>>>>>>
> >>>>>>>>>> On 22 March 2016 at 23:03, Simon Weller <sw...@ena.com>
> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Nezar,
> >>>>>>>>>>>
> >>>>>>>>>>> I've looked through the code related to your release and it
> seems
> >>>>>>> that
> >>>>>>>>> in
> >>>>>>>>>>> order to throw this exception, 10 attempts have been made to
> >>>> retrieve
> >>>>>>>>> the
> >>>>>>>>>>> disk information prior to the exception being thrown.
> >>>>>>>>>>>
> >>>>>>>>>>> It appears there is something really odd going on here. I'm
> >>>> starting
> >>>>>>>>> to
> >>>>>>>>>>> wonder whether it might be worth while for you to stop all of
> your
> >>>>>>>>>> agents,
> >>>>>>>>>>> plus management server processes, restart libvirtd on each of
> the
> >>>>>>> hosts
> >>>>>>>>>> and
> >>>>>>>>>>> then bring the agents back up prior to starting the management
> >>>>>>> servers.
> >>>>>>>>>>>
> >>>>>>>>>>> There are a bunch of logs the pastebin you posted that seems to
> >>>>>>>>> indicate
> >>>>>>>>>>> you have some jobs hung, for example:
> >>>>>>>>>>>
> >>>>>>>>>>> 2016-03-14 21:45:31,828 WARN [o.a.c.f.j.i.AsyncJobMonitor]
> >>>>>>>>>>> (Timer-1:ctx-0c090bd7) Task (job-9066) has been pending for
> >>>> 1660851
> >>>>>>>>>> seconds
> >>>>>>>>>>> 2016-03-14 21:45:31,828 WARN [o.a.c.f.j.i.AsyncJobMonitor]
> >>>>>>>>>>> (Timer-1:ctx-0c090bd7) Task (job-8133) has been pending for
> >>>> 4253003
> >>>>>>>>>> seconds
> >>>>>>>>>>>
> >>>>>>>>>>> Restarting the agents, libvirtd and management will not cause
> an
> >>>>>>>>> outage
> >>>>>>>>>>> to your cloud, but I would suggest, if you decided to do this,
> you
> >>>>>>>>> should
> >>>>>>>>>>> complete it in a maintenance window.
> >>>>>>>>>>>
> >>>>>>>>>>> - Si
> >>>>>>>>>>>
> >>>>>>>>>>> ________________________________________
> >>>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
> >>>>>>>>>>> Sent: Sunday, March 20, 2016 10:27 PM
> >>>>>>>>>>> To: users@cloudstack.apache.org
> >>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable
> to
> >>>>>>> start
> >>>>>>>>>>> instance due to Unable to get answer that is of class
> >>>>>>>>>>> com.cloud.agent.api.StartAnswer
> >>>>>>>>>>>
> >>>>>>>>>>> Hi Simon,
> >>>>>>>>>>>
> >>>>>>>>>>> Please see answers to your questions below. I am not sure if I
> >>>>>>> provided
> >>>>>>>>>> the
> >>>>>>>>>>> correct answer for the primary and secondary storage
> >>>> configuration.
> >>>>>>>>>> Please
> >>>>>>>>>>> let me know if this is not what you are after:
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Can you screenshot and post somewhere your secondary storage
> >>>>>>>>>> configuration
> >>>>>>>>>>> and your primary storage configuration from the UI with the
> >>>> details?
> >>>>>>>>>>>
> >>>>>>>>>>> mysql> select * from configuration where name like
> '%secondary%'\G
> >>>>>>>>>>> *************************** 1. row ***************************
> >>>>>>>>>>> category: Account Defaults
> >>>>>>>>>>> instance: DEFAULT
> >>>>>>>>>>> component: management-server
> >>>>>>>>>>> name: max.account.secondary.storage
> >>>>>>>>>>> value: 400
> >>>>>>>>>>> description: The default maximum secondary storage space (in
> >>>> GiB)
> >>>>>>>>> that
> >>>>>>>>>>> can be used for an account
> >>>>>>>>>>> default_value: 400
> >>>>>>>>>>> updated: NULL
> >>>>>>>>>>> scope: NULL
> >>>>>>>>>>> is_dynamic: 0
> >>>>>>>>>>> *************************** 2. row ***************************
> >>>>>>>>>>> category: Project Defaults
> >>>>>>>>>>> instance: DEFAULT
> >>>>>>>>>>> component: management-server
> >>>>>>>>>>> name: max.project.secondary.storage
> >>>>>>>>>>> value: 400
> >>>>>>>>>>> description: The default maximum secondary storage space (in
> >>>> GiB)
> >>>>>>>>> that
> >>>>>>>>>>> can be used for a project
> >>>>>>>>>>> default_value: 400
> >>>>>>>>>>> updated: NULL
> >>>>>>>>>>> scope: NULL
> >>>>>>>>>>> is_dynamic: 0
> >>>>>>>>>>> *************************** 3. row ***************************
> >>>>>>>>>>> category: Hidden
> >>>>>>>>>>> instance: DEFAULT
> >>>>>>>>>>> component: management-server
> >>>>>>>>>>> name: secondary.storage.vm
> >>>>>>>>>>> value: +rwRhJOXPymbQOuAMLnhiw==
> >>>>>>>>>>> description: Deploys a VM per zone to manage secondary
> storage
> >>>> if
> >>>>>>>>> true,
> >>>>>>>>>>> otherwise secondary storage is mounted on management server
> >>>>>>>>>>> default_value: NULL
> >>>>>>>>>>> updated: NULL
> >>>>>>>>>>> scope: NULL
> >>>>>>>>>>> is_dynamic: 0
> >>>>>>>>>>> *************************** 4. row ***************************
> >>>>>>>>>>> category: Advanced
> >>>>>>>>>>> instance: DEFAULT
> >>>>>>>>>>> component: ExtensionRegistry
> >>>>>>>>>>> name: secondary.storage.vm.allocators.exclude
> >>>>>>>>>>> value: NULL
> >>>>>>>>>>> description: Extensions to exclude from being registered
> >>>>>>>>>>> default_value: NULL
> >>>>>>>>>>> updated: 2014-12-04 00:18:15
> >>>>>>>>>>> scope: Global
> >>>>>>>>>>> is_dynamic: 0
> >>>>>>>>>>> *************************** 5. row ***************************
> >>>>>>>>>>> category: NetworkManager
> >>>>>>>>>>> instance: DEFAULT
> >>>>>>>>>>> component: management-server
> >>>>>>>>>>> name: vm.network.nic.max.secondary.ipaddresses
> >>>>>>>>>>> value: NULL
> >>>>>>>>>>> description: Specify the number of secondary ip addresses per
> >>>> nic
> >>>>>>> per
> >>>>>>>>>> vm
> >>>>>>>>>>> default_value: 256
> >>>>>>>>>>> updated: NULL
> >>>>>>>>>>> scope: NULL
> >>>>>>>>>>> is_dynamic: 0
> >>>>>>>>>>> 5 rows in set (0.00 sec)
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> mysql> select * from configuration where name like
> '%primary%'\G
> >>>>>>>>>>> *************************** 1. row ***************************
> >>>>>>>>>>> category: Account Defaults
> >>>>>>>>>>> instance: DEFAULT
> >>>>>>>>>>> component: management-server
> >>>>>>>>>>> name: max.account.primary.storage
> >>>>>>>>>>> value: 200
> >>>>>>>>>>> description: The default maximum primary storage space (in
> GiB)
> >>>>>>> that
> >>>>>>>>>> can
> >>>>>>>>>>> be used for an account
> >>>>>>>>>>> default_value: 200
> >>>>>>>>>>> updated: NULL
> >>>>>>>>>>> scope: NULL
> >>>>>>>>>>> is_dynamic: 0
> >>>>>>>>>>> *************************** 2. row ***************************
> >>>>>>>>>>> category: Project Defaults
> >>>>>>>>>>> instance: DEFAULT
> >>>>>>>>>>> component: management-server
> >>>>>>>>>>> name: max.project.primary.storage
> >>>>>>>>>>> value: 200
> >>>>>>>>>>> description: The default maximum primary storage space (in
> GiB)
> >>>>>>> that
> >>>>>>>>>> can
> >>>>>>>>>>> be used for a project
> >>>>>>>>>>> default_value: 200
> >>>>>>>>>>> updated: NULL
> >>>>>>>>>>> scope: NULL
> >>>>>>>>>>> is_dynamic: 0
> >>>>>>>>>>> *************************** 3. row ***************************
> >>>>>>>>>>> category: Storage
> >>>>>>>>>>> instance: DEFAULT
> >>>>>>>>>>> component: TemplateManager
> >>>>>>>>>>> name: primary.storage.download.wait
> >>>>>>>>>>> value: 10800
> >>>>>>>>>>> description: In second, timeout for download template to
> primary
> >>>>>>>>>> storage
> >>>>>>>>>>> default_value: NULL
> >>>>>>>>>>> updated: NULL
> >>>>>>>>>>> scope: NULL
> >>>>>>>>>>> is_dynamic: 0
> >>>>>>>>>>> 3 rows in set (0.00 sec)
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Also, what is the Cloudstack version you are running and what
> is
> >>>> the
> >>>>>>> OS
> >>>>>>>>>>> you're running it on?
> >>>>>>>>>>>
> >>>>>>>>>>> Cloudstack 4.4.1
> >>>>>>>>>>>
> >>>>>>>>>>> Centos 6.6
> >>>>>>>>>>>
> >>>>>>>>>>> Are you using actimeo=0 in your NFS mount line?
> >>>>>>>>>>>
> >>>>>>>>>>> Primary storage on Hypervisors has the following parameters
> >>>>>>>>>>> 10.61.154.10:/Cloudstack/primary on
> >>>>>>>>>>> /mnt/b3647480-5709-36eb-9ddd-e2c666f30b57 type nfs
> >>>>>>>>>>>
> >>>> (rw,noatime,nfsvers=3,proto=tcp,hard,nolock,intr,addr=10.61.154.10)
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Secondary storage on SSVM has the following parameters
> >>>>>>>>>>> 10.61.154.10:/Cloudstack/secondary on
> >>>>>>>>>>> /mnt/SecStorage/d11d5c92-6950-30e8-a0b5-3cd46a17ec14 type nfs4
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>
> (rw,relatime,vers=4,rsize=262144,wsize=262144,namlen=255,acdirmin=0,acdirmax=0,soft,proto=tcp,port=0,timeo=133,retrans=
> >>>>>>> 2147483647
> >>>>>>>
> >>>>
> ,sec=sys,clientaddr=10.61.154.70,minorversion=0,local_lock=none,addr=10.61.154.10)
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks & Regards,
> >>>>>>>>>>> Nezar.
> >>>>>>>>>>>
> >>>>>>>>>>> On 18 March 2016 at 23:50, Simon Weller <sw...@ena.com>
> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Also, what is the Cloudstack version you are running and what
> is
> >>>> the
> >>>>>>>>> OS
> >>>>>>>>>>>> you're running it on?
> >>>>>>>>>>>>
> >>>>>>>>>>>> Are you using actimeo=0 in your NFS mount line?
> >>>>>>>>>>>>
> >>>>>>>>>>>> ________________________________________
> >>>>>>>>>>>> From: Simon Weller <sw...@ena.com>
> >>>>>>>>>>>> Sent: Friday, March 18, 2016 7:40 AM
> >>>>>>>>>>>> To: users@cloudstack.apache.org
> >>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message:
> Unable to
> >>>>>>>>> start
> >>>>>>>>>>>> instance due to Unable to get answer that is of class
> >>>>>>>>>>>> com.cloud.agent.api.StartAnswer
> >>>>>>>>>>>>
> >>>>>>>>>>>> Nexar,
> >>>>>>>>>>>>
> >>>>>>>>>>>> That's normal behaviour. You can only ssh to the link local on
> >>>> the
> >>>>>>>>> host
> >>>>>>>>>>>> where the VM resides.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Can you screenshot and post somewhere your secondary storage
> >>>>>>>>>>> configuration
> >>>>>>>>>>>> and your primary storage configuration from the UI with the
> >>>> details?
> >>>>>>>>>>>>
> >>>>>>>>>>>> - Si
> >>>>>>>>>>>>
> >>>>>>>>>>>> ________________________________________
> >>>>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
> >>>>>>>>>>>> Sent: Thursday, March 17, 2016 8:19 PM
> >>>>>>>>>>>> To: users@cloudstack.apache.org
> >>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message:
> Unable to
> >>>>>>>>> start
> >>>>>>>>>>>> instance due to Unable to get answer that is of class
> >>>>>>>>>>>> com.cloud.agent.api.StartAnswer
> >>>>>>>>>>>>
> >>>>>>>>>>>> Hi Simon,
> >>>>>>>>>>>>
> >>>>>>>>>>>> I just noticed that from other Hosts I can't ssh into the
> SSVM:
> >>>>>>>>>>>>
> >>>>>>>>>>>> [root@eqx-cs-cmp-01 ~]# ssh -i /root/.ssh/id_rsa.cloud -p
> 3922
> >>>>>>>>>>>> root@169.254.3.161
> >>>>>>>>>>>> ssh: connect to host 169.254.3.161 port 3922: No route to host
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> I can only ssh from one Host:
> >>>>>>>>>>>> [root@eqx-cs-cmp-06 ~]# ssh -i /root/.ssh/id_rsa.cloud -p
> 3922
> >>>>>>>>>>>> root@169.254.3.161
> >>>>>>>>>>>> Linux s-11-VM 3.2.0-4-amd64 #1 SMP Debian 3.2.57-3 x86_64
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks & Regards,
> >>>>>>>>>>>> Nezar.
> >>>>>>>>>>>>
> >>>>>>>>>>>> On 18 March 2016 at 10:36, Nezar Madbouh <
> >>>>>>>>> nezar.madbouh@ipscape.com.au
> >>>>>>>>>>>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Hi Simon,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Yes existing VMs are successfully running. Yesterday I was
> able
> >>>> to
> >>>>>>>>>>> create
> >>>>>>>>>>>>> a VM in Dev Environment but I can't create VMs in Prod
> >>>> environment.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>> Nezar.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On 18 March 2016 at 09:12, Simon Weller <sw...@ena.com>
> >>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> I'm assuming you have other VMs successfully running,
> correct?
> >>>> So
> >>>>>>>>>> you
> >>>>>>>>>>>> can
> >>>>>>>>>>>>>> write to your primary storage.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> ________________________________________
> >>>>>>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
> >>>>>>>>>>>>>> Sent: Thursday, March 17, 2016 4:43 PM
> >>>>>>>>>>>>>> To: users@cloudstack.apache.org
> >>>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message:
> Unable
> >>>> to
> >>>>>>>>>>> start
> >>>>>>>>>>>>>> instance due to Unable to get answer that is of class
> >>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Hi Simon and Shweta,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Please see the result below:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> root@s-11-VM:~# /usr/local/cloud/systemvm/ssvm-check.sh
> >>>>>>>>>>>>>> ================================================
> >>>>>>>>>>>>>> First DNS server is 8.8.8.8
> >>>>>>>>>>>>>> PING 8.8.8.8 (8.8.8.8): 48 data bytes
> >>>>>>>>>>>>>> 56 bytes from 8.8.8.8: icmp_seq=0 ttl=57 time=5.209 ms
> >>>>>>>>>>>>>> 56 bytes from 8.8.8.8: icmp_seq=1 ttl=57 time=0.936 ms
> >>>>>>>>>>>>>> --- 8.8.8.8 ping statistics ---
> >>>>>>>>>>>>>> 2 packets transmitted, 2 packets received, 0% packet loss
> >>>>>>>>>>>>>> round-trip min/avg/max/stddev = 0.936/3.072/5.209/2.137 ms
> >>>>>>>>>>>>>> Good: Can ping DNS server
> >>>>>>>>>>>>>> ================================================
> >>>>>>>>>>>>>> Good: DNS resolves download.cloud.com
> >>>>>>>>>>>>>> ================================================
> >>>>>>>>>>>>>> nfs is currently mounted
> >>>>>>>>>>>>>> Mount point is
> >>>>>>>>> /mnt/SecStorage/d11d5c92-6950-30e8-a0b5-3cd46a17ec14
> >>>>>>>>>>>>>> Good: Can write to mount point
> >>>>>>>>>>>>>> ================================================
> >>>>>>>>>>>>>> Management server is 10.61.153.20. Checking connectivity.
> >>>>>>>>>>>>>> Good: Can connect to management server port 8250
> >>>>>>>>>>>>>> ================================================
> >>>>>>>>>>>>>> Good: Java process is running
> >>>>>>>>>>>>>> ================================================
> >>>>>>>>>>>>>> Tests Complete. Look for ERROR or WARNING above.
> >>>>>>>>>>>>>> root@s-11-VM:~#
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks & Regards,
> >>>>>>>>>>>>>> Nezar.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On 17 March 2016 at 09:42, Simon Weller <sw...@ena.com>
> >>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Nezar,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Log into the Cloudstack UI.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Click on "Infrastructure" on the left menu. Click on View
> All
> >>>>>>>>>> below
> >>>>>>>>>>>>>>> "System VMs". find the item marked "Secondary Storage VM"
> and
> >>>>>>>>>> then
> >>>>>>>>>>>>>> click
> >>>>>>>>>>>>>>> on the Name of that VM on the left (it will be s-XXXX-VM).
> >>>>>>>>> You'll
> >>>>>>>>>>> see
> >>>>>>>>>>>> a
> >>>>>>>>>>>>>>> bunch of information about the VM. Find the Host line and
> also
> >>>>>>>>> the
> >>>>>>>>>>>> Link
> >>>>>>>>>>>>>>> Local IP Address line and then follow the original
> >>>> instructions
> >>>>>>>>> in
> >>>>>>>>>>> my
> >>>>>>>>>>>>>>> previous email.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> - Si
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> ________________________________________
> >>>>>>>>>>>>>>> From: Shweta Agarwal <sh...@accelerite.com>
> >>>>>>>>>>>>>>> Sent: Wednesday, March 16, 2016 12:16 AM
> >>>>>>>>>>>>>>> To: users@cloudstack.apache.org
> >>>>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message:
> Unable
> >>>>>>>>> to
> >>>>>>>>>>>> start
> >>>>>>>>>>>>>>> instance due to Unable to get answer that is of class
> >>>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hi Nezar,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> You can find host name on detail view of SSVM in UI .
> >>>>>>>>>>>>>>> You can also use list api command listSystemVms to get the
> >>>> host
> >>>>>>>>>>>> details
> >>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>> system vms.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks
> >>>>>>>>>>>>>>> Shweta
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> ________________________________________
> >>>>>>>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
> >>>>>>>>>>>>>>> Sent: Wednesday, March 16, 2016 9:58 AM
> >>>>>>>>>>>>>>> To: users@cloudstack.apache.org
> >>>>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message:
> Unable
> >>>>>>>>> to
> >>>>>>>>>>>> start
> >>>>>>>>>>>>>>> instance due to Unable to get answer that is of class
> >>>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hi Simon,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> How can I find out which host the SSVM is on?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks & Regards,
> >>>>>>>>>>>>>>> Nezar.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On 16 March 2016 at 12:28, Simon Weller <sw...@ena.com>
> >>>>>>>>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Can you ssh into the Secondary Storage VM?
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> This can be done by getting the link-local-ip address from
> >>>> the
> >>>>>>>>>>>>>>>> infrastructure menu in the UI and then sshing into the
> host
> >>>>>>>>> the
> >>>>>>>>>>> SSVM
> >>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>> on
> >>>>>>>>>>>>>>>> and issuing this command as root: ssh -i
> >>>>>>>>> /root/.ssh/id_rsa.cloud
> >>>>>>>>>>> -p
> >>>>>>>>>>>>>> 3922
> >>>>>>>>>>>>>>>> root@link-local-ip (replace link-local-ip with the link
> >>>>>>>>> local
> >>>>>>>>>>> ip).
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Try running the SSVM diangnostic script:
> >>>>>>>>>>>>>>>> usr/local/cloud/systemvm/ssvm-check.sh
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> - Si
> >>>>>>>>>>>>>>>> ________________________________________
> >>>>>>>>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
> >>>>>>>>>>>>>>>> Sent: Tuesday, March 15, 2016 7:49 PM
> >>>>>>>>>>>>>>>> To: users@cloudstack.apache.org
> >>>>>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message:
> >>>> Unable
> >>>>>>>>>> to
> >>>>>>>>>>>>>> start
> >>>>>>>>>>>>>>>> instance due to Unable to get answer that is of class
> >>>>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Hi Simon,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Please see details below:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> mysql> use cloud; select id, name, status, type from host
> >>>>>>>>> where
> >>>>>>>>>>> id =
> >>>>>>>>>>>>>> 5;
> >>>>>>>>>>>>>>>> Reading table information for completion of table and
> column
> >>>>>>>>>> names
> >>>>>>>>>>>>>>>> You can turn off this feature to get a quicker startup
> with
> >>>> -A
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Database changed
> >>>>>>>>>>>>>>>> +----+------------------------------+--------+---------+
> >>>>>>>>>>>>>>>> | id | name | status | type |
> >>>>>>>>>>>>>>>> +----+------------------------------+--------+---------+
> >>>>>>>>>>>>>>>> | 5 | eqx-cs-cmp-02.ipscape.com.au | up | routing |
> >>>>>>>>>>>>>>>> +----+------------------------------+--------+---------+
> >>>>>>>>>>>>>>>> 1 row in set (0.00 sec)
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> cat /var/log/cloudstack/agent/agent.log
> >>>>>>>>>>>>>>>> com.cloud.utils.exception.CloudRuntimeException:
> >>>>>>>>>>>>>>>> com.cloud.utils.exception.CloudRuntimeException: Can't
> find
> >>>>>>>>>>>>>>>> volume:089ff3ca-9bff-402c-a14e-ea7a35aaad5c
> >>>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>
> com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.getPhysicalDisk(KVMStoragePoolManager.java:262)
> >>>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>
> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.createVbd(LibvirtComputingResource.java:3921)
> >>>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>
> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(LibvirtComputingResource.java:3789)
> >>>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>
> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1333)
> >>>>>>>>>>>>>>>> at
> >>>>>>>>> com.cloud.agent.Agent.processRequest(Agent.java:501)
> >>>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>>
> >>>> com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:808)
> >>>>>>>>>>>>>>>> at com.cloud.utils.nio.Task.run(Task.java:84)
> >>>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> >>>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> >>>>>>>>>>>>>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>>>>>>>>>>>> agent.log.2016-03-03.gz (END)
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> [root@eqx-cs-cmp-02 agent]# virsh pool-list
> >>>>>>>>>>>>>>>> Name State Autostart
> >>>>>>>>>>>>>>>> -----------------------------------------
> >>>>>>>>>>>>>>>> 2d0967f8-50d6-4d6d-87b7-4da0596f6924 active no
> >>>>>>>>>>>>>>>> b3647480-5709-36eb-9ddd-e2c666f30b57 active no
> >>>>>>>>>>>>>>>> e3831246-ff55-4af4-9964-a17565edaccc active no
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> [Root @ eqx-cs-cmp-02 agent] # df -h
> >>>>>>>>>>>>>>>> Filesystem Size Used Avail Use% Mounted on
> >>>>>>>>>>>>>>>> /dev/mapper/vg_root-lv_root
> >>>>>>>>>>>>>>>> 203G 215G 1.3G 1% /
> >>>>>>>>>>>>>>>> tmpfs 48g 48g 0 0% / dev / shm
> >>>>>>>>>>>>>>>> / Dev / sda1 477m 427m 25M 6% / boot
> >>>>>>>>>>>>>>>> xx.xx.xx.xx:/Cloudstack/primary
> >>>>>>>>>>>>>>>> 7.6T 4.1T 3.5T 54%
> >>>>>>>>>>>>>>>> /mnt/b3647480-5709-36eb-9ddd-e2c666f30b57
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> I also noticed the following in cloudstack-agent.err
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> libvirt: QEMU Driver error : Domain not found: no domain
> with
> >>>>>>>>>>>> matching
> >>>>>>>>>>>>>>> name
> >>>>>>>>>>>>>>>> 'i-11-600-VM'
> >>>>>>>>>>>>>>>> libvirt: Storage Driver error : Storage volume not found:
> no
> >>>>>>>>>>> storage
> >>>>>>>>>>>>>> vol
> >>>>>>>>>>>>>>>> with matching name '089ff3ca-9bff-402c-a14e-ea7a35aaad5c'
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Your help is much appreciated,
> >>>>>>>>>>>>>>>> Nezar
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On 16 March 2016 at 11:03, Simon Weller <sw...@ena.com>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Nezar,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> ok, I just took a look at a pastebin you posted.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> I see it complaining about the inability to find a volume
> >>>>>>>>>> prior
> >>>>>>>>>>> to
> >>>>>>>>>>>>>>>>> throwing the host exception.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Firstly, let's find out who host 5 is. Can you log into
> your
> >>>>>>>>>>> MySQL
> >>>>>>>>>>>>>>> server
> >>>>>>>>>>>>>>>>> and run this:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> use cloud;select id,name,status,type from host where
> id=5;
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Can you post the results?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> If it's a host, can you post the
> >>>>>>>>>>>> /var/log/cloudstack/agent/agent.log
> >>>>>>>>>>>>>>> from
> >>>>>>>>>>>>>>>>> that host?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> And also run virsh pool-list on the host in question?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> - Si
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> ________________________________________
> >>>>>>>>>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
> >>>>>>>>>>>>>>>>> Sent: Tuesday, March 15, 2016 6:11 PM
> >>>>>>>>>>>>>>>>> To: users@cloudstack.apache.org
> >>>>>>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message:
> >>>>>>>>> Unable
> >>>>>>>>>>> to
> >>>>>>>>>>>>>> start
> >>>>>>>>>>>>>>>>> instance due to Unable to get answer that is of class
> >>>>>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Hi Simon,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> When you say "double check your routers tied to the
> >>>>>>>>> network",
> >>>>>>>>>> do
> >>>>>>>>>>>> you
> >>>>>>>>>>>>>>> mean
> >>>>>>>>>>>>>>>>> the physical routers or the virtual routers?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> When I try to create a VM in Prod Environment it fails. I
> >>>>>>>>>> can't
> >>>>>>>>>>>> ping
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>> Prod virtual router from management server or Host.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> But I can ping Dev virtual router from management server
> and
> >>>>>>>>>>> Host
> >>>>>>>>>>>>>> and I
> >>>>>>>>>>>>>>>> was
> >>>>>>>>>>>>>>>>> able to create a VM in Dev Environment.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> How can I troubleshoot the Prod Virtual router. I
> launched
> >>>>>>>>> the
> >>>>>>>>>>>>>> console
> >>>>>>>>>>>>>>>> but
> >>>>>>>>>>>>>>>>> couldn't login.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Please note that we need to take into consideration that
> the
> >>>>>>>>>>>>>> affected
> >>>>>>>>>>>>>>>>> network is the Production network and we need to minimise
> >>>>>>>>>>> service
> >>>>>>>>>>>>>>> outage
> >>>>>>>>>>>>>>>> as
> >>>>>>>>>>>>>>>>> much as possible.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Your help is much appreciated,
> >>>>>>>>>>>>>>>>> Nezar
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On 16 March 2016 at 08:37, Simon Weller <sweller@ena.com
> >
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> I've seen this before where a Virtual Router has crashed
> >>>>>>>>> and
> >>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> management server can't communicate with it.
> >>>>>>>>>>>>>>>>>> Are you using an advanced network setup?
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> If so, double check your routers tied to the network
> where
> >>>>>>>>>> you
> >>>>>>>>>>>> are
> >>>>>>>>>>>>>>>>>> attempting to start an instance.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> - Si
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> ________________________________________
> >>>>>>>>>>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
> >>>>>>>>>>>>>>>>>> Sent: Tuesday, March 15, 2016 4:02 PM
> >>>>>>>>>>>>>>>>>> To: users@cloudstack.apache.org
> >>>>>>>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message:
> >>>>>>>>>> Unable
> >>>>>>>>>>>> to
> >>>>>>>>>>>>>>> start
> >>>>>>>>>>>>>>>>>> instance due to Unable to get answer that is of class
> >>>>>>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Hi Glenn,
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> We have 16 hosts, all KVM.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> The error is not specific to Host 5. Every time I try to
> >>>>>>>>>>> create
> >>>>>>>>>>>> a
> >>>>>>>>>>>>>> VM
> >>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>> gives me the same error message but on different host.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> How do I restart the agent on the host. Is it service
> >>>>>>>>>>> affecting?
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Thanks very much for your help.
> >>>>>>>>>>>>>>>>>> Neza
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On 15 March 2016 at 17:37, Glenn Wagner <
> >>>>>>>>>>>>>> glenn.wagner@shapeblue.com>
> >>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Hi,
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Can you explain your environment first , how many hosts
> >>>>>>>>> ,
> >>>>>>>>>>> what
> >>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>> your
> >>>>>>>>>>>>>>>>>>> hypervisors (KVM,Xenserver,VMware)
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> The error seems to be with your Host 5 Hypervisor
> >>>>>>>>>>>>>>>>>>> Resource [Host:5] is unreachable:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Go to the infrastructure section and see what host5 is
> >>>>>>>>>>>>>> reporting,
> >>>>>>>>>>>>>>>>>>> Have you tried to restart the agent on the Host?
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Thanks
> >>>>>>>>>>>>>>>>>>> Glenn
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> [image: ShapeBlue] <http://www.shapeblue.com>
> >>>>>>>>>>>>>>>>>>> Glenn Wagner
> >>>>>>>>>>>>>>>>>>> Senior Consultant , ShapeBlue
> >>>>>>>>>>>>>>>>>>> d: * | s: +27 21 527 0091*
> >>>>>>>>>>>>>> <%7C%20s:%20+27%2021%20527%200091> |
> >>>>>>>>>>>>>>>> m:
> >>>>>>>>>>>>>>>>>>> *+27 73 917 4111* <+27%2073%20917%204111>
> >>>>>>>>>>>>>>>>>>> e: *glenn.wagner@shapeblue.com | t: *
> >>>>>>>>>>>>>>>>>>> <glenn.wagner@shapeblue.com%20%7C%20t:> | w: *
> >>>>>>>>>>>>>> www.shapeblue.com
> >>>>>>>>>>>>>>> *
> >>>>>>>>>>>>>>>>>>> <http://www.shapeblue.com>
> >>>>>>>>>>>>>>>>>>> a:
> >>>>>>>>>>>>>>>>>>> 2nd Floor, Oudehuis Centre, 122 Main Rd, Somerset West
> >>>>>>>>>> Cape
> >>>>>>>>>>>> Town
> >>>>>>>>>>>>>>> 7130
> >>>>>>>>>>>>>>>>>> South Africa
> >>>>>>>>>>>>>>>>>>> Shape Blue Ltd is a company incorporated in England &
> >>>>>>>>>> Wales.
> >>>>>>>>>>>>>>>> ShapeBlue
> >>>>>>>>>>>>>>>>>>> Services India LLP is a company incorporated in India
> >>>>>>>>> and
> >>>>>>>>>> is
> >>>>>>>>>>>>>>> operated
> >>>>>>>>>>>>>>>>>> under
> >>>>>>>>>>>>>>>>>>> license from Shape Blue Ltd. Shape Blue Brasil
> >>>>>>>>> Consultoria
> >>>>>>>>>>>> Ltda
> >>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>> company incorporated in Brasil and is operated under
> >>>>>>>>>> license
> >>>>>>>>>>>>>> from
> >>>>>>>>>>>>>>>> Shape
> >>>>>>>>>>>>>>>>>>> Blue Ltd. ShapeBlue SA Pty Ltd is a company registered
> >>>>>>>>> by
> >>>>>>>>>>> The
> >>>>>>>>>>>>>>>> Republic
> >>>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>> South Africa and is traded under license from Shape
> Blue
> >>>>>>>>>>> Ltd.
> >>>>>>>>>>>>>>>> ShapeBlue
> >>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>> a registered trademark.
> >>>>>>>>>>>>>>>>>>> This email and any attachments to it may be
> confidential
> >>>>>>>>>> and
> >>>>>>>>>>>> are
> >>>>>>>>>>>>>>>>> intended
> >>>>>>>>>>>>>>>>>>> solely for the use of the individual to whom it is
> >>>>>>>>>>> addressed.
> >>>>>>>>>>>>>> Any
> >>>>>>>>>>>>>>>> views
> >>>>>>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>>>> opinions expressed are solely those of the author and
> do
> >>>>>>>>>> not
> >>>>>>>>>>>>>>>>> necessarily
> >>>>>>>>>>>>>>>>>>> represent those of Shape Blue Ltd or related companies.
> >>>>>>>>> If
> >>>>>>>>>>> you
> >>>>>>>>>>>>>> are
> >>>>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> intended recipient of this email, you must neither take
> >>>>>>>>>> any
> >>>>>>>>>>>>>> action
> >>>>>>>>>>>>>>>>> based
> >>>>>>>>>>>>>>>>>>> upon its contents, nor copy or show it to anyone.
> Please
> >>>>>>>>>>>> contact
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> sender
> >>>>>>>>>>>>>>>>>>> if you believe you have received this email in error.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> -----Original Message-----
> >>>>>>>>>>>>>>>>>>> From: Nezar Madbouh [mailto:
> >>>>>>>>> nezar.madbouh@ipscape.com.au]
> >>>>>>>>>>>>>>>>>>> Sent: Tuesday, 15 March 2016 5:45 AM
> >>>>>>>>>>>>>>>>>>> To: users@cloudstack.apache.org
> >>>>>>>>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message:
> >>>>>>>>>>> Unable
> >>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>> start
> >>>>>>>>>>>>>>>>>>> instance due to Unable to get answer that is of class
> >>>>>>>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Hi Jeremy,
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> I have uploaded the contents to
> >>>>>>>>>>> http://pastebin.com/1J3mmgTh
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> your help is much appreciated,
> >>>>>>>>>>>>>>>>>>> Nezar.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> On 15 March 2016 at 13:46, Jeremy Peterson <
> >>>>>>>>>>>>>> jpeterson@acentek.net>
> >>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> I don't see any attachments can you create a pastebin.
> >>>>>>>>>>>> That's
> >>>>>>>>>>>>>>>> easier
> >>>>>>>>>>>>>>>>>>>> to look through as I think attachments are removed
> >>>>>>>>>> before
> >>>>>>>>>>>>>> being
> >>>>>>>>>>>>>>>> sent
> >>>>>>>>>>>>>>>>>>>> to user group.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Jeremy
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> -------- Original message --------
> >>>>>>>>>>>>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
> >>>>>>>>>>>>>>>>>>>> Date: 3/14/2016 6:38 PM (GMT-06:00)
> >>>>>>>>>>>>>>>>>>>> To: users@cloudstack.apache.org
> >>>>>>>>>>>>>>>>>>>> Subject: Unable to create a New VM - Error message:
> >>>>>>>>>> Unable
> >>>>>>>>>>>> to
> >>>>>>>>>>>>>>> start
> >>>>>>>>>>>>>>>>>>>> instance due to Unable to get answer that is of class
> >>>>>>>>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Hi Shweta and Sadhu,
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Please find attached managerment-server.log which
> >>>>>>>>>> contains
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>>> logs
> >>>>>>>>>>>>>>>>>>>> since I started creating a new VM (VM id 600).
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Please note that we don't have agent.log files
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Your help is much appreciated.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Regards,
> >>>>>>>>>>>>>>>>>>>> Nezar.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Find out more about ShapeBlue and our range of
> >>>>>>>>> CloudStack
> >>>>>>>>>>>>>> related
> >>>>>>>>>>>>>>>>>> services:
> >>>>>>>>>>>>>>>>>>> IaaS Cloud Design & Build
> >>>>>>>>>>>>>>>>>>> <http://shapeblue.com/iaas-cloud-design-and-build//> |
> >>>>>>>>>>>> CSForge
> >>>>>>>>>>>>>> -
> >>>>>>>>>>>>>>>> rapid
> >>>>>>>>>>>>>>>>>>> IaaS deployment framework <
> >>>>>>>>> http://shapeblue.com/csforge/>
> >>>>>>>>>>>>>>>>>>> CloudStack Consulting <
> >>>>>>>>>>>>>>> http://shapeblue.com/cloudstack-consultancy/>
> >>>>>>>>>>>>>>>> |
> >>>>>>>>>>>>>>>>>> CloudStack
> >>>>>>>>>>>>>>>>>>> Software Engineering
> >>>>>>>>>>>>>>>>>>> <http://shapeblue.com/cloudstack-software-engineering/
> >
> >>>>>>>>>>>>>>>>>>> CloudStack Infrastructure Support
> >>>>>>>>>>>>>>>>>>> <
> >>>>>>>>> http://shapeblue.com/cloudstack-infrastructure-support/>
> >>>>>>>>>> |
> >>>>>>>>>>>>>>>> CloudStack
> >>>>>>>>>>>>>>>>>>> Bootcamp Training Courses <
> >>>>>>>>>>>>>>> http://shapeblue.com/cloudstack-training/
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> DISCLAIMER
> >>>>>>>>>>>>>>> ==========
> >>>>>>>>>>>>>>> This e-mail may contain privileged and confidential
> >>>> information
> >>>>>>>>>>> which
> >>>>>>>>>>>> is
> >>>>>>>>>>>>>>> the property of Accelerite, a Persistent Systems business.
> It
> >>>> is
> >>>>>>>>>>>>>> intended
> >>>>>>>>>>>>>>> only for the use of the individual or entity to which it is
> >>>>>>>>>>> addressed.
> >>>>>>>>>>>>>> If
> >>>>>>>>>>>>>>> you are not the intended recipient, you are not authorized
> to
> >>>>>>>>>> read,
> >>>>>>>>>>>>>> retain,
> >>>>>>>>>>>>>>> copy, print, distribute or use this message. If you have
> >>>>>>>>> received
> >>>>>>>>>>> this
> >>>>>>>>>>>>>>> communication in error, please notify the sender and delete
> >>>> all
> >>>>>>>>>>> copies
> >>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>> this message. Accelerite, a Persistent Systems business
> does
> >>>> not
> >>>>>>>>>>>> accept
> >>>>>>>>>>>>>> any
> >>>>>>>>>>>>>>> liability for virus infected mails.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>>
> >>>
> >>
>
>
Re: Unable to create a New VM - Error message: Unable to start
instance due to Unable to get answer that is of class com.cloud.agent.api.StartAnswer
Posted by Nezar Madbouh <ne...@ipscape.com.au>.
Hi Ilya,
Adrian Sender provided us some support today and he managed to find the
problem. The problem is with the compute and disk offerings. We created new
compute offering and created a new VM from it and it worked.
Thank you very much for your help.
Regards,
Nezar.
On 16 April 2016 at 03:32, ilya <il...@gmail.com> wrote:
> Here is what i need to understand, if you are using SharedMountPoint,
> then /mnt/storage must be preset on all KVM hosts in a persistent manner
> (i.e. fstab).
>
> However, if you are using NFS, you can just let cloudstack manage the
> NFS mount points by adding the storage as NFS type.
>
> On 4/15/16 12:46 AM, ilya wrote:
> > Can you goto primary storage in cloudstack UI, is this what you have
> > defined for your NFS Server?
> >
> > "SharedMountPoint","host":"localhost","path":"/mnt/storage"
> >
> > If so, please use IP or FQDN instead..
> >
> >
> > On 4/14/16 10:39 PM, Nezar Madbouh wrote:
> >> Hi Ilya,
> >>
> >> Here is the output of mount command:
> >>
> >> [root@eqx-cs-cmp-01 ~]# mount
> >> /dev/mapper/vg_root-lv_root on / type ext4 (rw)
> >> proc on /proc type proc (rw)
> >> sysfs on /sys type sysfs (rw)
> >> devpts on /dev/pts type devpts (rw,gid=5,mode=620)
> >> tmpfs on /dev/shm type tmpfs (rw)
> >> /dev/sda1 on /boot type ext4 (rw)
> >> none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
> >> sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
> >> nfsd on /proc/fs/nfsd type nfsd (rw)
> >> 10.61.154.10:/Cloudstack/primary on
> >> /mnt/b3647480-5709-36eb-9ddd-e2c666f30b57 type nfs
> >> (rw,noatime,nfsvers=3,proto=tcp,hard,nolock,intr,addr=10.61.154.10)
> >> [root@eqx-cs-cmp-01 ~]#
> >>
> >>
> >> On 15 April 2016 at 15:22, Nezar Madbouh <ne...@ipscape.com.au>
> >> wrote:
> >>
> >>> Hi Ilya,
> >>>
> >>> Thanks for the instructions. I followed them and pasted the
> information on
> >>> pastebin:
> >>>
> >>> management-server logs job-14938
> >>> http://pastebin.com/GAiy8u5H
> >>>
> >>> Agent logs
> >>> http://pastebin.com/mfiADsJc
> >>>
> >>>
> >>> Your help is much appreciated.
> >>>
> >>> Nezar.
> >>>
> >>> On 15 April 2016 at 13:58, ilya <il...@gmail.com> wrote:
> >>>
> >>>> Here is how i would troubleshoot this:
> >>>>
> >>>> 1) ssh to MS host, run
> >>>>
> >>>> tail -f /var/cloudstack/management/management-server.log | grep -i
> >>>> StartVMCmdByAdmin
> >>>>
> >>>> 2) on cloudstack UI, press start on the VM and watch the tail log
> >>>>
> >>>> 3) on the tail log, you should see a StartVirtualMachine api call show
> >>>> up with job ID. For example:
> >>>>
> >>>>> 2016-04-15 03:41:57,193 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>>> (API-Job-Executor-24:ctx-7892387a job-291575) Executing AsyncJobVO
> >>>> {id:291575, userId: 157, accountId: 2, instanceType: VirtualMachine,
> >>>> instanceId: 7648, cmd:
> >>>> org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin, cmdInfo:
> >>>>
> {"id":"7648","response":"json","sessionkey":"mMJsduS9KLKApqH4dPpIlGZdt1Y\u003d","ctxDetails":"{\"com.cloud.vm.VirtualMachine\":7648,\"com.cloud.uservm.UserVm\":7648,\"com.cloud.network.router.VirtualRouter\":7648}","cmdEventType":"VM.START","expires":"2016-04-15T03:51:57+0000","ctxUserId":"157","signatureversion":"3","httpmethod":"GET","uuid":"7648","ctxAccountId":"2","ctxStartEventId":"314416"},
> >>>> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
> >>>> result: null, initMsid: 345051337894, completeMsid: null, lastUpdated:
> >>>> null, lastPolled: null, created: null}
> >>>>
> >>>> Each task in cloudstack spins up its own job, in this case its
> >>>> *job-291575*
> >>>>
> >>>> 4) now execute
> >>>>> grep -A3 -B3 job-291575
> >>>> /var/cloudstack/management/management-server.log | tee job-291575.log
> >>>>> grep job-291575 /var/cloudstack/management/management-server.log |
> grep
> >>>> via | head -1
> >>>>
> >>>> you should see a response like so,
> >>>>
> >>>>> 2016-04-15 03:41:59,259 DEBUG [c.c.a.m.ClusteredAgentAttache]
> >>>> (Work-Job-Executor-42:ctx-b2c4af6b job-291575/job-291576
> ctx-0988152e) Seq
> >>>> 441-4181310779036795138: Forwarding Seq 441-4181310779036795138: {
> Cmd ,
> >>>> MgmtId: 345051337894, via: 441(my-kvm-host01.example.com), Ver: v1,
> >>>> Flags: 100011,
> >>>>
> [{"com.cloud.agent.api.routing.DhcpEntryCommand":{"vmMac":"06:fe:4e:00:12:ad","vmIpAddress":"10.10.27.178","vmName":"ol6-test-ilya1","defaultRouter":"10.10.27.1","defaultDns":"10.17.27.166","duid":"00:03:00:01:06:fe:4e:00:12:ad","isDefault":true,"executeInSequence":false,"accessDetails":{"zone.network.type":"Advanced","router.guest.ip":"10.10.27.166","router.ip":"169.254.0.238","
> >>>> router.name":"r-7301-VM"},"wait":0}}] } to 345051337893
> >>>>
> >>>> We are looking for this line "via: 441(my-kvm-host01.example.com)"
> >>>>
> >>>> SSH to my-kvm-host01.example.com, and copy the
> >>>> /var/log/cloudstack/agent.log to pastebin
> >>>>
> >>>> also copy job-291575.log to pastebin
> >>>>
> >>>> I'll try to have a look, though when i get chance.
> >>>>
> >>>>
> >>>> Also paste the output of mount command on kvm host in question. I'm
> >>>> suspecting one of your storage pools maybe bad.
> >>>>
> >>>> Lastly, what distro are you running for KVM?
> >>>>
> >>>> Regards
> >>>> ilya
> >>>>
> >>>> On 4/14/16 5:05 PM, Nezar Madbouh wrote:
> >>>>> Hi Ilya,
> >>>>>
> >>>>> I shutdown all the VMs in Pre-prod Environmnet expecting the Pre-prod
> >>>>> Virtual router to be deleted automatically because no VM is running
> on
> >>>> this
> >>>>> vlan but it wasn't deleted.
> >>>>>
> >>>>> Am I right to expect the VR to be deleted automatically if no VM is
> >>>> running
> >>>>> in that vlan?
> >>>>>
> >>>>> Thanks,
> >>>>> Nezar.
> >>>>>
> >>>>> On 15 April 2016 at 09:53, Nezar Madbouh <
> nezar.madbouh@ipscape.com.au>
> >>>>> wrote:
> >>>>>
> >>>>>> Hi Ilya,
> >>>>>>
> >>>>>> I have run the script and it didn't return any results:
> >>>>>>
> >>>>>> [root@eqx-cs-mgt-01 ~]# ./async_jobs.sh
> >>>>>> [root@eqx-cs-mgt-01 ~]#
> >>>>>>
> >>>>>>
> >>>>>> I also ran the command inside mysql:
> >>>>>>
> >>>>>> mysql> SELECT id, user_id, account_id, instance_type, instance_id,
> >>>>>> job_cmd, job_cmd_info, job_dispatcher, job_pending_signals FROM
> >>>>>> cloud.async_job where job_status=0 and job_dispatcher not like
> >>>>>> "pseudoJobDispatcher";
> >>>>>> Empty set (0.00 sec)
> >>>>>>
> >>>>>> your help is much appreciated,
> >>>>>> Nezar.
> >>>>>>
> >>>>>>
> >>>>>> On 14 April 2016 at 16:49, ilya <il...@gmail.com>
> wrote:
> >>>>>>
> >>>>>>> It appears you have pending jobs that are not letting you go
> through.
> >>>>>>>
> >>>>>>> Usually stop and starting management service should be enough.
> >>>>>>>
> >>>>>>> However, try running this script from your management server to
> show
> >>>> us
> >>>>>>> what you have running in async_job table.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>> #!/bin/bash
> >>>>>>>> DATESTAMP=$(date +%m%d%y-%H%M%S)
> >>>>>>>> DBPASS=$(java -classpath
> >>>>>>> /usr/share/cloudstack-common/lib/jasypt-1.9.0.jar
> >>>>>>> org.jasypt.intf.cli.JasyptPBEStringDecryptionCLI input="$(cat
> >>>>>>> /etc/cloudstack/management/db.properties | grep db.cloud.password |
> >>>> awk
> >>>>>>> -F'(' '{print $2}' | sed 's/)//g')" password="$(cat
> >>>>>>> /etc/cloudstack/management/key)" | grep -A2 OUTPUT | tail -1)
> >>>>>>>> DBHOST=$(cat /etc/cloudstack/management/db.properties | grep
> >>>>>>> db.cloud.host | awk -F'=' '{print $2}' | tail -1 )
> >>>>>>>> DBUSER=$(cat /etc/cloudstack/management/db.properties | grep
> >>>>>>> db.cloud.username | awk -F'=' '{print $2}')
> >>>>>>>> DB=$(cat /etc/cloudstack/management/db.properties | grep
> >>>> db.cloud.name
> >>>>>>> | awk -F'=' '{print $2}')
> >>>>>>>> DBPORT=$(cat /etc/cloudstack/management/db.properties | grep
> >>>>>>> db.cloud.port | awk -F'=' '{print $2}')
> >>>>>>>> MYSQLCMD="mysql -h $DBHOST -u $DBUSER -P $DBPORT -p$DBPASS $DB"
> >>>>>>>> echo 'SELECT id, user_id, account_id, instance_type, instance_id,
> >>>>>>> job_cmd, job_cmd_info, job_dispatcher, job_pending_signals FROM
> >>>>>>> cloud.async_job where job_status=0 and job_dispatcher not like
> >>>>>>> "pseudoJobDispatcher"' | $MYSQLCMD
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On 4/13/16 5:16 PM, Nezar Madbouh wrote:
> >>>>>>>> Hi Simon,
> >>>>>>>>
> >>>>>>>> The following work was done but unfortunately the problem is not
> >>>> fixed:
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> on management server:
> >>>>>>>>
> >>>>>>>> service cloudstack-management stop
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> on the host:
> >>>>>>>>
> >>>>>>>> service cloudstack-agent stop
> >>>>>>>>
> >>>>>>>> service libvirtd restart
> >>>>>>>>
> >>>>>>>> service cloudstack-agent start
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> on management server:
> >>>>>>>>
> >>>>>>>> service cloudstack-management start
> >>>>>>>>
> >>>>>>>> Regards,
> >>>>>>>> Nezar.
> >>>>>>>>
> >>>>>>>> On 24 March 2016 at 05:06, Simon Weller <sw...@ena.com> wrote:
> >>>>>>>>
> >>>>>>>>> Yes, I think that order is fine. Make sure the processes are
> indeed
> >>>>>>>>> shutdown before moving on to the next step.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> ________________________________________
> >>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
> >>>>>>>>> Sent: Tuesday, March 22, 2016 5:33 PM
> >>>>>>>>> To: users@cloudstack.apache.org
> >>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable to
> >>>> start
> >>>>>>>>> instance due to Unable to get answer that is of class
> >>>>>>>>> com.cloud.agent.api.StartAnswer
> >>>>>>>>>
> >>>>>>>>> Hi Simon,
> >>>>>>>>>
> >>>>>>>>> Can you please check if the following sequence of action is
> correct:
> >>>>>>>>>
> >>>>>>>>> on management server:
> >>>>>>>>>
> >>>>>>>>> service cloudstack-management stop
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> on the host:
> >>>>>>>>>
> >>>>>>>>> service cloudstack-agent stop
> >>>>>>>>>
> >>>>>>>>> service libvirtd restart
> >>>>>>>>>
> >>>>>>>>> service cloudstack-agent start
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> on management server:
> >>>>>>>>>
> >>>>>>>>> service cloudstack-management start
> >>>>>>>>>
> >>>>>>>>> Thanks very much for your help.
> >>>>>>>>> Nezar.
> >>>>>>>>>
> >>>>>>>>> On 23 March 2016 at 08:39, Simon Weller <sw...@ena.com> wrote:
> >>>>>>>>>
> >>>>>>>>>> Shutting down all the management services should clear the jobs.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> ________________________________________
> >>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
> >>>>>>>>>> Sent: Tuesday, March 22, 2016 4:28 PM
> >>>>>>>>>> To: users@cloudstack.apache.org
> >>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable
> to
> >>>>>>> start
> >>>>>>>>>> instance due to Unable to get answer that is of class
> >>>>>>>>>> com.cloud.agent.api.StartAnswer
> >>>>>>>>>>
> >>>>>>>>>> Hi Simon,
> >>>>>>>>>>
> >>>>>>>>>> I will plan to restart agents, libvirtd and management services.
> >>>>>>>>>>
> >>>>>>>>>> How do I clear hung jobs.
> >>>>>>>>>>
> >>>>>>>>>> Your help is much appreciated,
> >>>>>>>>>> Nezar.
> >>>>>>>>>>
> >>>>>>>>>> On 22 March 2016 at 23:03, Simon Weller <sw...@ena.com>
> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Nezar,
> >>>>>>>>>>>
> >>>>>>>>>>> I've looked through the code related to your release and it
> seems
> >>>>>>> that
> >>>>>>>>> in
> >>>>>>>>>>> order to throw this exception, 10 attempts have been made to
> >>>> retrieve
> >>>>>>>>> the
> >>>>>>>>>>> disk information prior to the exception being thrown.
> >>>>>>>>>>>
> >>>>>>>>>>> It appears there is something really odd going on here. I'm
> >>>> starting
> >>>>>>>>> to
> >>>>>>>>>>> wonder whether it might be worth while for you to stop all of
> your
> >>>>>>>>>> agents,
> >>>>>>>>>>> plus management server processes, restart libvirtd on each of
> the
> >>>>>>> hosts
> >>>>>>>>>> and
> >>>>>>>>>>> then bring the agents back up prior to starting the management
> >>>>>>> servers.
> >>>>>>>>>>>
> >>>>>>>>>>> There are a bunch of logs the pastebin you posted that seems to
> >>>>>>>>> indicate
> >>>>>>>>>>> you have some jobs hung, for example:
> >>>>>>>>>>>
> >>>>>>>>>>> 2016-03-14 21:45:31,828 WARN [o.a.c.f.j.i.AsyncJobMonitor]
> >>>>>>>>>>> (Timer-1:ctx-0c090bd7) Task (job-9066) has been pending for
> >>>> 1660851
> >>>>>>>>>> seconds
> >>>>>>>>>>> 2016-03-14 21:45:31,828 WARN [o.a.c.f.j.i.AsyncJobMonitor]
> >>>>>>>>>>> (Timer-1:ctx-0c090bd7) Task (job-8133) has been pending for
> >>>> 4253003
> >>>>>>>>>> seconds
> >>>>>>>>>>>
> >>>>>>>>>>> Restarting the agents, libvirtd and management will not cause
> an
> >>>>>>>>> outage
> >>>>>>>>>>> to your cloud, but I would suggest, if you decided to do this,
> you
> >>>>>>>>> should
> >>>>>>>>>>> complete it in a maintenance window.
> >>>>>>>>>>>
> >>>>>>>>>>> - Si
> >>>>>>>>>>>
> >>>>>>>>>>> ________________________________________
> >>>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
> >>>>>>>>>>> Sent: Sunday, March 20, 2016 10:27 PM
> >>>>>>>>>>> To: users@cloudstack.apache.org
> >>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable
> to
> >>>>>>> start
> >>>>>>>>>>> instance due to Unable to get answer that is of class
> >>>>>>>>>>> com.cloud.agent.api.StartAnswer
> >>>>>>>>>>>
> >>>>>>>>>>> Hi Simon,
> >>>>>>>>>>>
> >>>>>>>>>>> Please see answers to your questions below. I am not sure if I
> >>>>>>> provided
> >>>>>>>>>> the
> >>>>>>>>>>> correct answer for the primary and secondary storage
> >>>> configuration.
> >>>>>>>>>> Please
> >>>>>>>>>>> let me know if this is not what you are after:
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Can you screenshot and post somewhere your secondary storage
> >>>>>>>>>> configuration
> >>>>>>>>>>> and your primary storage configuration from the UI with the
> >>>> details?
> >>>>>>>>>>>
> >>>>>>>>>>> mysql> select * from configuration where name like
> '%secondary%'\G
> >>>>>>>>>>> *************************** 1. row ***************************
> >>>>>>>>>>> category: Account Defaults
> >>>>>>>>>>> instance: DEFAULT
> >>>>>>>>>>> component: management-server
> >>>>>>>>>>> name: max.account.secondary.storage
> >>>>>>>>>>> value: 400
> >>>>>>>>>>> description: The default maximum secondary storage space (in
> >>>> GiB)
> >>>>>>>>> that
> >>>>>>>>>>> can be used for an account
> >>>>>>>>>>> default_value: 400
> >>>>>>>>>>> updated: NULL
> >>>>>>>>>>> scope: NULL
> >>>>>>>>>>> is_dynamic: 0
> >>>>>>>>>>> *************************** 2. row ***************************
> >>>>>>>>>>> category: Project Defaults
> >>>>>>>>>>> instance: DEFAULT
> >>>>>>>>>>> component: management-server
> >>>>>>>>>>> name: max.project.secondary.storage
> >>>>>>>>>>> value: 400
> >>>>>>>>>>> description: The default maximum secondary storage space (in
> >>>> GiB)
> >>>>>>>>> that
> >>>>>>>>>>> can be used for a project
> >>>>>>>>>>> default_value: 400
> >>>>>>>>>>> updated: NULL
> >>>>>>>>>>> scope: NULL
> >>>>>>>>>>> is_dynamic: 0
> >>>>>>>>>>> *************************** 3. row ***************************
> >>>>>>>>>>> category: Hidden
> >>>>>>>>>>> instance: DEFAULT
> >>>>>>>>>>> component: management-server
> >>>>>>>>>>> name: secondary.storage.vm
> >>>>>>>>>>> value: +rwRhJOXPymbQOuAMLnhiw==
> >>>>>>>>>>> description: Deploys a VM per zone to manage secondary
> storage
> >>>> if
> >>>>>>>>> true,
> >>>>>>>>>>> otherwise secondary storage is mounted on management server
> >>>>>>>>>>> default_value: NULL
> >>>>>>>>>>> updated: NULL
> >>>>>>>>>>> scope: NULL
> >>>>>>>>>>> is_dynamic: 0
> >>>>>>>>>>> *************************** 4. row ***************************
> >>>>>>>>>>> category: Advanced
> >>>>>>>>>>> instance: DEFAULT
> >>>>>>>>>>> component: ExtensionRegistry
> >>>>>>>>>>> name: secondary.storage.vm.allocators.exclude
> >>>>>>>>>>> value: NULL
> >>>>>>>>>>> description: Extensions to exclude from being registered
> >>>>>>>>>>> default_value: NULL
> >>>>>>>>>>> updated: 2014-12-04 00:18:15
> >>>>>>>>>>> scope: Global
> >>>>>>>>>>> is_dynamic: 0
> >>>>>>>>>>> *************************** 5. row ***************************
> >>>>>>>>>>> category: NetworkManager
> >>>>>>>>>>> instance: DEFAULT
> >>>>>>>>>>> component: management-server
> >>>>>>>>>>> name: vm.network.nic.max.secondary.ipaddresses
> >>>>>>>>>>> value: NULL
> >>>>>>>>>>> description: Specify the number of secondary ip addresses per
> >>>> nic
> >>>>>>> per
> >>>>>>>>>> vm
> >>>>>>>>>>> default_value: 256
> >>>>>>>>>>> updated: NULL
> >>>>>>>>>>> scope: NULL
> >>>>>>>>>>> is_dynamic: 0
> >>>>>>>>>>> 5 rows in set (0.00 sec)
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> mysql> select * from configuration where name like
> '%primary%'\G
> >>>>>>>>>>> *************************** 1. row ***************************
> >>>>>>>>>>> category: Account Defaults
> >>>>>>>>>>> instance: DEFAULT
> >>>>>>>>>>> component: management-server
> >>>>>>>>>>> name: max.account.primary.storage
> >>>>>>>>>>> value: 200
> >>>>>>>>>>> description: The default maximum primary storage space (in
> GiB)
> >>>>>>> that
> >>>>>>>>>> can
> >>>>>>>>>>> be used for an account
> >>>>>>>>>>> default_value: 200
> >>>>>>>>>>> updated: NULL
> >>>>>>>>>>> scope: NULL
> >>>>>>>>>>> is_dynamic: 0
> >>>>>>>>>>> *************************** 2. row ***************************
> >>>>>>>>>>> category: Project Defaults
> >>>>>>>>>>> instance: DEFAULT
> >>>>>>>>>>> component: management-server
> >>>>>>>>>>> name: max.project.primary.storage
> >>>>>>>>>>> value: 200
> >>>>>>>>>>> description: The default maximum primary storage space (in
> GiB)
> >>>>>>> that
> >>>>>>>>>> can
> >>>>>>>>>>> be used for a project
> >>>>>>>>>>> default_value: 200
> >>>>>>>>>>> updated: NULL
> >>>>>>>>>>> scope: NULL
> >>>>>>>>>>> is_dynamic: 0
> >>>>>>>>>>> *************************** 3. row ***************************
> >>>>>>>>>>> category: Storage
> >>>>>>>>>>> instance: DEFAULT
> >>>>>>>>>>> component: TemplateManager
> >>>>>>>>>>> name: primary.storage.download.wait
> >>>>>>>>>>> value: 10800
> >>>>>>>>>>> description: In second, timeout for download template to
> primary
> >>>>>>>>>> storage
> >>>>>>>>>>> default_value: NULL
> >>>>>>>>>>> updated: NULL
> >>>>>>>>>>> scope: NULL
> >>>>>>>>>>> is_dynamic: 0
> >>>>>>>>>>> 3 rows in set (0.00 sec)
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Also, what is the Cloudstack version you are running and what
> is
> >>>> the
> >>>>>>> OS
> >>>>>>>>>>> you're running it on?
> >>>>>>>>>>>
> >>>>>>>>>>> Cloudstack 4.4.1
> >>>>>>>>>>>
> >>>>>>>>>>> Centos 6.6
> >>>>>>>>>>>
> >>>>>>>>>>> Are you using actimeo=0 in your NFS mount line?
> >>>>>>>>>>>
> >>>>>>>>>>> Primary storage on Hypervisors has the following parameters
> >>>>>>>>>>> 10.61.154.10:/Cloudstack/primary on
> >>>>>>>>>>> /mnt/b3647480-5709-36eb-9ddd-e2c666f30b57 type nfs
> >>>>>>>>>>>
> >>>> (rw,noatime,nfsvers=3,proto=tcp,hard,nolock,intr,addr=10.61.154.10)
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Secondary storage on SSVM has the following parameters
> >>>>>>>>>>> 10.61.154.10:/Cloudstack/secondary on
> >>>>>>>>>>> /mnt/SecStorage/d11d5c92-6950-30e8-a0b5-3cd46a17ec14 type nfs4
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>
> (rw,relatime,vers=4,rsize=262144,wsize=262144,namlen=255,acdirmin=0,acdirmax=0,soft,proto=tcp,port=0,timeo=133,retrans=
> >>>>>>> 2147483647
> >>>>>>>
> >>>>
> ,sec=sys,clientaddr=10.61.154.70,minorversion=0,local_lock=none,addr=10.61.154.10)
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks & Regards,
> >>>>>>>>>>> Nezar.
> >>>>>>>>>>>
> >>>>>>>>>>> On 18 March 2016 at 23:50, Simon Weller <sw...@ena.com>
> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Also, what is the Cloudstack version you are running and what
> is
> >>>> the
> >>>>>>>>> OS
> >>>>>>>>>>>> you're running it on?
> >>>>>>>>>>>>
> >>>>>>>>>>>> Are you using actimeo=0 in your NFS mount line?
> >>>>>>>>>>>>
> >>>>>>>>>>>> ________________________________________
> >>>>>>>>>>>> From: Simon Weller <sw...@ena.com>
> >>>>>>>>>>>> Sent: Friday, March 18, 2016 7:40 AM
> >>>>>>>>>>>> To: users@cloudstack.apache.org
> >>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message:
> Unable to
> >>>>>>>>> start
> >>>>>>>>>>>> instance due to Unable to get answer that is of class
> >>>>>>>>>>>> com.cloud.agent.api.StartAnswer
> >>>>>>>>>>>>
> >>>>>>>>>>>> Nexar,
> >>>>>>>>>>>>
> >>>>>>>>>>>> That's normal behaviour. You can only ssh to the link local on
> >>>> the
> >>>>>>>>> host
> >>>>>>>>>>>> where the VM resides.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Can you screenshot and post somewhere your secondary storage
> >>>>>>>>>>> configuration
> >>>>>>>>>>>> and your primary storage configuration from the UI with the
> >>>> details?
> >>>>>>>>>>>>
> >>>>>>>>>>>> - Si
> >>>>>>>>>>>>
> >>>>>>>>>>>> ________________________________________
> >>>>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
> >>>>>>>>>>>> Sent: Thursday, March 17, 2016 8:19 PM
> >>>>>>>>>>>> To: users@cloudstack.apache.org
> >>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message:
> Unable to
> >>>>>>>>> start
> >>>>>>>>>>>> instance due to Unable to get answer that is of class
> >>>>>>>>>>>> com.cloud.agent.api.StartAnswer
> >>>>>>>>>>>>
> >>>>>>>>>>>> Hi Simon,
> >>>>>>>>>>>>
> >>>>>>>>>>>> I just noticed that from other Hosts I can't ssh into the
> SSVM:
> >>>>>>>>>>>>
> >>>>>>>>>>>> [root@eqx-cs-cmp-01 ~]# ssh -i /root/.ssh/id_rsa.cloud -p
> 3922
> >>>>>>>>>>>> root@169.254.3.161
> >>>>>>>>>>>> ssh: connect to host 169.254.3.161 port 3922: No route to host
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> I can only ssh from one Host:
> >>>>>>>>>>>> [root@eqx-cs-cmp-06 ~]# ssh -i /root/.ssh/id_rsa.cloud -p
> 3922
> >>>>>>>>>>>> root@169.254.3.161
> >>>>>>>>>>>> Linux s-11-VM 3.2.0-4-amd64 #1 SMP Debian 3.2.57-3 x86_64
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks & Regards,
> >>>>>>>>>>>> Nezar.
> >>>>>>>>>>>>
> >>>>>>>>>>>> On 18 March 2016 at 10:36, Nezar Madbouh <
> >>>>>>>>> nezar.madbouh@ipscape.com.au
> >>>>>>>>>>>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Hi Simon,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Yes existing VMs are successfully running. Yesterday I was
> able
> >>>> to
> >>>>>>>>>>> create
> >>>>>>>>>>>>> a VM in Dev Environment but I can't create VMs in Prod
> >>>> environment.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>> Nezar.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On 18 March 2016 at 09:12, Simon Weller <sw...@ena.com>
> >>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> I'm assuming you have other VMs successfully running,
> correct?
> >>>> So
> >>>>>>>>>> you
> >>>>>>>>>>>> can
> >>>>>>>>>>>>>> write to your primary storage.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> ________________________________________
> >>>>>>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
> >>>>>>>>>>>>>> Sent: Thursday, March 17, 2016 4:43 PM
> >>>>>>>>>>>>>> To: users@cloudstack.apache.org
> >>>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message:
> Unable
> >>>> to
> >>>>>>>>>>> start
> >>>>>>>>>>>>>> instance due to Unable to get answer that is of class
> >>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Hi Simon and Shweta,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Please see the result below:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> root@s-11-VM:~# /usr/local/cloud/systemvm/ssvm-check.sh
> >>>>>>>>>>>>>> ================================================
> >>>>>>>>>>>>>> First DNS server is 8.8.8.8
> >>>>>>>>>>>>>> PING 8.8.8.8 (8.8.8.8): 48 data bytes
> >>>>>>>>>>>>>> 56 bytes from 8.8.8.8: icmp_seq=0 ttl=57 time=5.209 ms
> >>>>>>>>>>>>>> 56 bytes from 8.8.8.8: icmp_seq=1 ttl=57 time=0.936 ms
> >>>>>>>>>>>>>> --- 8.8.8.8 ping statistics ---
> >>>>>>>>>>>>>> 2 packets transmitted, 2 packets received, 0% packet loss
> >>>>>>>>>>>>>> round-trip min/avg/max/stddev = 0.936/3.072/5.209/2.137 ms
> >>>>>>>>>>>>>> Good: Can ping DNS server
> >>>>>>>>>>>>>> ================================================
> >>>>>>>>>>>>>> Good: DNS resolves download.cloud.com
> >>>>>>>>>>>>>> ================================================
> >>>>>>>>>>>>>> nfs is currently mounted
> >>>>>>>>>>>>>> Mount point is
> >>>>>>>>> /mnt/SecStorage/d11d5c92-6950-30e8-a0b5-3cd46a17ec14
> >>>>>>>>>>>>>> Good: Can write to mount point
> >>>>>>>>>>>>>> ================================================
> >>>>>>>>>>>>>> Management server is 10.61.153.20. Checking connectivity.
> >>>>>>>>>>>>>> Good: Can connect to management server port 8250
> >>>>>>>>>>>>>> ================================================
> >>>>>>>>>>>>>> Good: Java process is running
> >>>>>>>>>>>>>> ================================================
> >>>>>>>>>>>>>> Tests Complete. Look for ERROR or WARNING above.
> >>>>>>>>>>>>>> root@s-11-VM:~#
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks & Regards,
> >>>>>>>>>>>>>> Nezar.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On 17 March 2016 at 09:42, Simon Weller <sw...@ena.com>
> >>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Nezar,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Log into the Cloudstack UI.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Click on "Infrastructure" on the left menu. Click on View
> All
> >>>>>>>>>> below
> >>>>>>>>>>>>>>> "System VMs". find the item marked "Secondary Storage VM"
> and
> >>>>>>>>>> then
> >>>>>>>>>>>>>> click
> >>>>>>>>>>>>>>> on the Name of that VM on the left (it will be s-XXXX-VM).
> >>>>>>>>> You'll
> >>>>>>>>>>> see
> >>>>>>>>>>>> a
> >>>>>>>>>>>>>>> bunch of information about the VM. Find the Host line and
> also
> >>>>>>>>> the
> >>>>>>>>>>>> Link
> >>>>>>>>>>>>>>> Local IP Address line and then follow the original
> >>>> instructions
> >>>>>>>>> in
> >>>>>>>>>>> my
> >>>>>>>>>>>>>>> previous email.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> - Si
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> ________________________________________
> >>>>>>>>>>>>>>> From: Shweta Agarwal <sh...@accelerite.com>
> >>>>>>>>>>>>>>> Sent: Wednesday, March 16, 2016 12:16 AM
> >>>>>>>>>>>>>>> To: users@cloudstack.apache.org
> >>>>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message:
> Unable
> >>>>>>>>> to
> >>>>>>>>>>>> start
> >>>>>>>>>>>>>>> instance due to Unable to get answer that is of class
> >>>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hi Nezar,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> You can find host name on detail view of SSVM in UI .
> >>>>>>>>>>>>>>> You can also use list api command listSystemVms to get the
> >>>> host
> >>>>>>>>>>>> details
> >>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>> system vms.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks
> >>>>>>>>>>>>>>> Shweta
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> ________________________________________
> >>>>>>>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
> >>>>>>>>>>>>>>> Sent: Wednesday, March 16, 2016 9:58 AM
> >>>>>>>>>>>>>>> To: users@cloudstack.apache.org
> >>>>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message:
> Unable
> >>>>>>>>> to
> >>>>>>>>>>>> start
> >>>>>>>>>>>>>>> instance due to Unable to get answer that is of class
> >>>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hi Simon,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> How can I find out which host the SSVM is on?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks & Regards,
> >>>>>>>>>>>>>>> Nezar.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On 16 March 2016 at 12:28, Simon Weller <sw...@ena.com>
> >>>>>>>>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Can you ssh into the Secondary Storage VM?
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> This can be done by getting the link-local-ip address from
> >>>> the
> >>>>>>>>>>>>>>>> infrastructure menu in the UI and then sshing into the
> host
> >>>>>>>>> the
> >>>>>>>>>>> SSVM
> >>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>> on
> >>>>>>>>>>>>>>>> and issuing this command as root: ssh -i
> >>>>>>>>> /root/.ssh/id_rsa.cloud
> >>>>>>>>>>> -p
> >>>>>>>>>>>>>> 3922
> >>>>>>>>>>>>>>>> root@link-local-ip (replace link-local-ip with the link
> >>>>>>>>> local
> >>>>>>>>>>> ip).
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Try running the SSVM diangnostic script:
> >>>>>>>>>>>>>>>> usr/local/cloud/systemvm/ssvm-check.sh
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> - Si
> >>>>>>>>>>>>>>>> ________________________________________
> >>>>>>>>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
> >>>>>>>>>>>>>>>> Sent: Tuesday, March 15, 2016 7:49 PM
> >>>>>>>>>>>>>>>> To: users@cloudstack.apache.org
> >>>>>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message:
> >>>> Unable
> >>>>>>>>>> to
> >>>>>>>>>>>>>> start
> >>>>>>>>>>>>>>>> instance due to Unable to get answer that is of class
> >>>>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Hi Simon,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Please see details below:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> mysql> use cloud; select id, name, status, type from host
> >>>>>>>>> where
> >>>>>>>>>>> id =
> >>>>>>>>>>>>>> 5;
> >>>>>>>>>>>>>>>> Reading table information for completion of table and
> column
> >>>>>>>>>> names
> >>>>>>>>>>>>>>>> You can turn off this feature to get a quicker startup
> with
> >>>> -A
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Database changed
> >>>>>>>>>>>>>>>> +----+------------------------------+--------+---------+
> >>>>>>>>>>>>>>>> | id | name | status | type |
> >>>>>>>>>>>>>>>> +----+------------------------------+--------+---------+
> >>>>>>>>>>>>>>>> | 5 | eqx-cs-cmp-02.ipscape.com.au | up | routing |
> >>>>>>>>>>>>>>>> +----+------------------------------+--------+---------+
> >>>>>>>>>>>>>>>> 1 row in set (0.00 sec)
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> cat /var/log/cloudstack/agent/agent.log
> >>>>>>>>>>>>>>>> com.cloud.utils.exception.CloudRuntimeException:
> >>>>>>>>>>>>>>>> com.cloud.utils.exception.CloudRuntimeException: Can't
> find
> >>>>>>>>>>>>>>>> volume:089ff3ca-9bff-402c-a14e-ea7a35aaad5c
> >>>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>
> com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.getPhysicalDisk(KVMStoragePoolManager.java:262)
> >>>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>
> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.createVbd(LibvirtComputingResource.java:3921)
> >>>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>
> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(LibvirtComputingResource.java:3789)
> >>>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>
> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1333)
> >>>>>>>>>>>>>>>> at
> >>>>>>>>> com.cloud.agent.Agent.processRequest(Agent.java:501)
> >>>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>>
> >>>> com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:808)
> >>>>>>>>>>>>>>>> at com.cloud.utils.nio.Task.run(Task.java:84)
> >>>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> >>>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> >>>>>>>>>>>>>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>>>>>>>>>>>> agent.log.2016-03-03.gz (END)
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> [root@eqx-cs-cmp-02 agent]# virsh pool-list
> >>>>>>>>>>>>>>>> Name State Autostart
> >>>>>>>>>>>>>>>> -----------------------------------------
> >>>>>>>>>>>>>>>> 2d0967f8-50d6-4d6d-87b7-4da0596f6924 active no
> >>>>>>>>>>>>>>>> b3647480-5709-36eb-9ddd-e2c666f30b57 active no
> >>>>>>>>>>>>>>>> e3831246-ff55-4af4-9964-a17565edaccc active no
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> [Root @ eqx-cs-cmp-02 agent] # df -h
> >>>>>>>>>>>>>>>> Filesystem Size Used Avail Use% Mounted on
> >>>>>>>>>>>>>>>> /dev/mapper/vg_root-lv_root
> >>>>>>>>>>>>>>>> 203G 215G 1.3G 1% /
> >>>>>>>>>>>>>>>> tmpfs 48g 48g 0 0% / dev / shm
> >>>>>>>>>>>>>>>> / Dev / sda1 477m 427m 25M 6% / boot
> >>>>>>>>>>>>>>>> xx.xx.xx.xx:/Cloudstack/primary
> >>>>>>>>>>>>>>>> 7.6T 4.1T 3.5T 54%
> >>>>>>>>>>>>>>>> /mnt/b3647480-5709-36eb-9ddd-e2c666f30b57
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> I also noticed the following in cloudstack-agent.err
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> libvirt: QEMU Driver error : Domain not found: no domain
> with
> >>>>>>>>>>>> matching
> >>>>>>>>>>>>>>> name
> >>>>>>>>>>>>>>>> 'i-11-600-VM'
> >>>>>>>>>>>>>>>> libvirt: Storage Driver error : Storage volume not found:
> no
> >>>>>>>>>>> storage
> >>>>>>>>>>>>>> vol
> >>>>>>>>>>>>>>>> with matching name '089ff3ca-9bff-402c-a14e-ea7a35aaad5c'
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Your help is much appreciated,
> >>>>>>>>>>>>>>>> Nezar
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On 16 March 2016 at 11:03, Simon Weller <sw...@ena.com>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Nezar,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> ok, I just took a look at a pastebin you posted.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> I see it complaining about the inability to find a volume
> >>>>>>>>>> prior
> >>>>>>>>>>> to
> >>>>>>>>>>>>>>>>> throwing the host exception.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Firstly, let's find out who host 5 is. Can you log into
> your
> >>>>>>>>>>> MySQL
> >>>>>>>>>>>>>>> server
> >>>>>>>>>>>>>>>>> and run this:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> use cloud;select id,name,status,type from host where
> id=5;
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Can you post the results?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> If it's a host, can you post the
> >>>>>>>>>>>> /var/log/cloudstack/agent/agent.log
> >>>>>>>>>>>>>>> from
> >>>>>>>>>>>>>>>>> that host?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> And also run virsh pool-list on the host in question?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> - Si
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> ________________________________________
> >>>>>>>>>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
> >>>>>>>>>>>>>>>>> Sent: Tuesday, March 15, 2016 6:11 PM
> >>>>>>>>>>>>>>>>> To: users@cloudstack.apache.org
> >>>>>>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message:
> >>>>>>>>> Unable
> >>>>>>>>>>> to
> >>>>>>>>>>>>>> start
> >>>>>>>>>>>>>>>>> instance due to Unable to get answer that is of class
> >>>>>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Hi Simon,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> When you say "double check your routers tied to the
> >>>>>>>>> network",
> >>>>>>>>>> do
> >>>>>>>>>>>> you
> >>>>>>>>>>>>>>> mean
> >>>>>>>>>>>>>>>>> the physical routers or the virtual routers?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> When I try to create a VM in Prod Environment it fails. I
> >>>>>>>>>> can't
> >>>>>>>>>>>> ping
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>> Prod virtual router from management server or Host.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> But I can ping Dev virtual router from management server
> and
> >>>>>>>>>>> Host
> >>>>>>>>>>>>>> and I
> >>>>>>>>>>>>>>>> was
> >>>>>>>>>>>>>>>>> able to create a VM in Dev Environment.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> How can I troubleshoot the Prod Virtual router. I
> launched
> >>>>>>>>> the
> >>>>>>>>>>>>>> console
> >>>>>>>>>>>>>>>> but
> >>>>>>>>>>>>>>>>> couldn't login.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Please note that we need to take into consideration that
> the
> >>>>>>>>>>>>>> affected
> >>>>>>>>>>>>>>>>> network is the Production network and we need to minimise
> >>>>>>>>>>> service
> >>>>>>>>>>>>>>> outage
> >>>>>>>>>>>>>>>> as
> >>>>>>>>>>>>>>>>> much as possible.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Your help is much appreciated,
> >>>>>>>>>>>>>>>>> Nezar
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On 16 March 2016 at 08:37, Simon Weller <sweller@ena.com
> >
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> I've seen this before where a Virtual Router has crashed
> >>>>>>>>> and
> >>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> management server can't communicate with it.
> >>>>>>>>>>>>>>>>>> Are you using an advanced network setup?
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> If so, double check your routers tied to the network
> where
> >>>>>>>>>> you
> >>>>>>>>>>>> are
> >>>>>>>>>>>>>>>>>> attempting to start an instance.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> - Si
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> ________________________________________
> >>>>>>>>>>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
> >>>>>>>>>>>>>>>>>> Sent: Tuesday, March 15, 2016 4:02 PM
> >>>>>>>>>>>>>>>>>> To: users@cloudstack.apache.org
> >>>>>>>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message:
> >>>>>>>>>> Unable
> >>>>>>>>>>>> to
> >>>>>>>>>>>>>>> start
> >>>>>>>>>>>>>>>>>> instance due to Unable to get answer that is of class
> >>>>>>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Hi Glenn,
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> We have 16 hosts, all KVM.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> The error is not specific to Host 5. Every time I try to
> >>>>>>>>>>> create
> >>>>>>>>>>>> a
> >>>>>>>>>>>>>> VM
> >>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>> gives me the same error message but on different host.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> How do I restart the agent on the host. Is it service
> >>>>>>>>>>> affecting?
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Thanks very much for your help.
> >>>>>>>>>>>>>>>>>> Neza
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On 15 March 2016 at 17:37, Glenn Wagner <
> >>>>>>>>>>>>>> glenn.wagner@shapeblue.com>
> >>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Hi,
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Can you explain your environment first , how many hosts
> >>>>>>>>> ,
> >>>>>>>>>>> what
> >>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>> your
> >>>>>>>>>>>>>>>>>>> hypervisors (KVM,Xenserver,VMware)
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> The error seems to be with your Host 5 Hypervisor
> >>>>>>>>>>>>>>>>>>> Resource [Host:5] is unreachable:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Go to the infrastructure section and see what host5 is
> >>>>>>>>>>>>>> reporting,
> >>>>>>>>>>>>>>>>>>> Have you tried to restart the agent on the Host?
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Thanks
> >>>>>>>>>>>>>>>>>>> Glenn
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> [image: ShapeBlue] <http://www.shapeblue.com>
> >>>>>>>>>>>>>>>>>>> Glenn Wagner
> >>>>>>>>>>>>>>>>>>> Senior Consultant , ShapeBlue
> >>>>>>>>>>>>>>>>>>> d: * | s: +27 21 527 0091*
> >>>>>>>>>>>>>> <%7C%20s:%20+27%2021%20527%200091> |
> >>>>>>>>>>>>>>>> m:
> >>>>>>>>>>>>>>>>>>> *+27 73 917 4111* <+27%2073%20917%204111>
> >>>>>>>>>>>>>>>>>>> e: *glenn.wagner@shapeblue.com | t: *
> >>>>>>>>>>>>>>>>>>> <glenn.wagner@shapeblue.com%20%7C%20t:> | w: *
> >>>>>>>>>>>>>> www.shapeblue.com
> >>>>>>>>>>>>>>> *
> >>>>>>>>>>>>>>>>>>> <http://www.shapeblue.com>
> >>>>>>>>>>>>>>>>>>> a:
> >>>>>>>>>>>>>>>>>>> 2nd Floor, Oudehuis Centre, 122 Main Rd, Somerset West
> >>>>>>>>>> Cape
> >>>>>>>>>>>> Town
> >>>>>>>>>>>>>>> 7130
> >>>>>>>>>>>>>>>>>> South Africa
> >>>>>>>>>>>>>>>>>>> Shape Blue Ltd is a company incorporated in England &
> >>>>>>>>>> Wales.
> >>>>>>>>>>>>>>>> ShapeBlue
> >>>>>>>>>>>>>>>>>>> Services India LLP is a company incorporated in India
> >>>>>>>>> and
> >>>>>>>>>> is
> >>>>>>>>>>>>>>> operated
> >>>>>>>>>>>>>>>>>> under
> >>>>>>>>>>>>>>>>>>> license from Shape Blue Ltd. Shape Blue Brasil
> >>>>>>>>> Consultoria
> >>>>>>>>>>>> Ltda
> >>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>> company incorporated in Brasil and is operated under
> >>>>>>>>>> license
> >>>>>>>>>>>>>> from
> >>>>>>>>>>>>>>>> Shape
> >>>>>>>>>>>>>>>>>>> Blue Ltd. ShapeBlue SA Pty Ltd is a company registered
> >>>>>>>>> by
> >>>>>>>>>>> The
> >>>>>>>>>>>>>>>> Republic
> >>>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>> South Africa and is traded under license from Shape
> Blue
> >>>>>>>>>>> Ltd.
> >>>>>>>>>>>>>>>> ShapeBlue
> >>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>> a registered trademark.
> >>>>>>>>>>>>>>>>>>> This email and any attachments to it may be
> confidential
> >>>>>>>>>> and
> >>>>>>>>>>>> are
> >>>>>>>>>>>>>>>>> intended
> >>>>>>>>>>>>>>>>>>> solely for the use of the individual to whom it is
> >>>>>>>>>>> addressed.
> >>>>>>>>>>>>>> Any
> >>>>>>>>>>>>>>>> views
> >>>>>>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>>>> opinions expressed are solely those of the author and
> do
> >>>>>>>>>> not
> >>>>>>>>>>>>>>>>> necessarily
> >>>>>>>>>>>>>>>>>>> represent those of Shape Blue Ltd or related companies.
> >>>>>>>>> If
> >>>>>>>>>>> you
> >>>>>>>>>>>>>> are
> >>>>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> intended recipient of this email, you must neither take
> >>>>>>>>>> any
> >>>>>>>>>>>>>> action
> >>>>>>>>>>>>>>>>> based
> >>>>>>>>>>>>>>>>>>> upon its contents, nor copy or show it to anyone.
> Please
> >>>>>>>>>>>> contact
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> sender
> >>>>>>>>>>>>>>>>>>> if you believe you have received this email in error.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> -----Original Message-----
> >>>>>>>>>>>>>>>>>>> From: Nezar Madbouh [mailto:
> >>>>>>>>> nezar.madbouh@ipscape.com.au]
> >>>>>>>>>>>>>>>>>>> Sent: Tuesday, 15 March 2016 5:45 AM
> >>>>>>>>>>>>>>>>>>> To: users@cloudstack.apache.org
> >>>>>>>>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message:
> >>>>>>>>>>> Unable
> >>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>> start
> >>>>>>>>>>>>>>>>>>> instance due to Unable to get answer that is of class
> >>>>>>>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Hi Jeremy,
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> I have uploaded the contents to
> >>>>>>>>>>> http://pastebin.com/1J3mmgTh
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> your help is much appreciated,
> >>>>>>>>>>>>>>>>>>> Nezar.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> On 15 March 2016 at 13:46, Jeremy Peterson <
> >>>>>>>>>>>>>> jpeterson@acentek.net>
> >>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> I don't see any attachments can you create a pastebin.
> >>>>>>>>>>>> That's
> >>>>>>>>>>>>>>>> easier
> >>>>>>>>>>>>>>>>>>>> to look through as I think attachments are removed
> >>>>>>>>>> before
> >>>>>>>>>>>>>> being
> >>>>>>>>>>>>>>>> sent
> >>>>>>>>>>>>>>>>>>>> to user group.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Jeremy
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> -------- Original message --------
> >>>>>>>>>>>>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
> >>>>>>>>>>>>>>>>>>>> Date: 3/14/2016 6:38 PM (GMT-06:00)
> >>>>>>>>>>>>>>>>>>>> To: users@cloudstack.apache.org
> >>>>>>>>>>>>>>>>>>>> Subject: Unable to create a New VM - Error message:
> >>>>>>>>>> Unable
> >>>>>>>>>>>> to
> >>>>>>>>>>>>>>> start
> >>>>>>>>>>>>>>>>>>>> instance due to Unable to get answer that is of class
> >>>>>>>>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Hi Shweta and Sadhu,
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Please find attached managerment-server.log which
> >>>>>>>>>> contains
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>>> logs
> >>>>>>>>>>>>>>>>>>>> since I started creating a new VM (VM id 600).
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Please note that we don't have agent.log files
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Your help is much appreciated.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Regards,
> >>>>>>>>>>>>>>>>>>>> Nezar.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Find out more about ShapeBlue and our range of
> >>>>>>>>> CloudStack
> >>>>>>>>>>>>>> related
> >>>>>>>>>>>>>>>>>> services:
> >>>>>>>>>>>>>>>>>>> IaaS Cloud Design & Build
> >>>>>>>>>>>>>>>>>>> <http://shapeblue.com/iaas-cloud-design-and-build//> |
> >>>>>>>>>>>> CSForge
> >>>>>>>>>>>>>> -
> >>>>>>>>>>>>>>>> rapid
> >>>>>>>>>>>>>>>>>>> IaaS deployment framework <
> >>>>>>>>> http://shapeblue.com/csforge/>
> >>>>>>>>>>>>>>>>>>> CloudStack Consulting <
> >>>>>>>>>>>>>>> http://shapeblue.com/cloudstack-consultancy/>
> >>>>>>>>>>>>>>>> |
> >>>>>>>>>>>>>>>>>> CloudStack
> >>>>>>>>>>>>>>>>>>> Software Engineering
> >>>>>>>>>>>>>>>>>>> <http://shapeblue.com/cloudstack-software-engineering/
> >
> >>>>>>>>>>>>>>>>>>> CloudStack Infrastructure Support
> >>>>>>>>>>>>>>>>>>> <
> >>>>>>>>> http://shapeblue.com/cloudstack-infrastructure-support/>
> >>>>>>>>>> |
> >>>>>>>>>>>>>>>> CloudStack
> >>>>>>>>>>>>>>>>>>> Bootcamp Training Courses <
> >>>>>>>>>>>>>>> http://shapeblue.com/cloudstack-training/
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> DISCLAIMER
> >>>>>>>>>>>>>>> ==========
> >>>>>>>>>>>>>>> This e-mail may contain privileged and confidential
> >>>> information
> >>>>>>>>>>> which
> >>>>>>>>>>>> is
> >>>>>>>>>>>>>>> the property of Accelerite, a Persistent Systems business.
> It
> >>>> is
> >>>>>>>>>>>>>> intended
> >>>>>>>>>>>>>>> only for the use of the individual or entity to which it is
> >>>>>>>>>>> addressed.
> >>>>>>>>>>>>>> If
> >>>>>>>>>>>>>>> you are not the intended recipient, you are not authorized
> to
> >>>>>>>>>> read,
> >>>>>>>>>>>>>> retain,
> >>>>>>>>>>>>>>> copy, print, distribute or use this message. If you have
> >>>>>>>>> received
> >>>>>>>>>>> this
> >>>>>>>>>>>>>>> communication in error, please notify the sender and delete
> >>>> all
> >>>>>>>>>>> copies
> >>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>> this message. Accelerite, a Persistent Systems business
> does
> >>>> not
> >>>>>>>>>>>> accept
> >>>>>>>>>>>>>> any
> >>>>>>>>>>>>>>> liability for virus infected mails.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>>
> >>>
> >>
>
>
Re: Unable to create a New VM - Error message: Unable to start instance
due to Unable to get answer that is of class com.cloud.agent.api.StartAnswer
Posted by ilya <il...@gmail.com>.
Here is what i need to understand, if you are using SharedMountPoint,
then /mnt/storage must be preset on all KVM hosts in a persistent manner
(i.e. fstab).
However, if you are using NFS, you can just let cloudstack manage the
NFS mount points by adding the storage as NFS type.
On 4/15/16 12:46 AM, ilya wrote:
> Can you goto primary storage in cloudstack UI, is this what you have
> defined for your NFS Server?
>
> "SharedMountPoint","host":"localhost","path":"/mnt/storage"
>
> If so, please use IP or FQDN instead..
>
>
> On 4/14/16 10:39 PM, Nezar Madbouh wrote:
>> Hi Ilya,
>>
>> Here is the output of mount command:
>>
>> [root@eqx-cs-cmp-01 ~]# mount
>> /dev/mapper/vg_root-lv_root on / type ext4 (rw)
>> proc on /proc type proc (rw)
>> sysfs on /sys type sysfs (rw)
>> devpts on /dev/pts type devpts (rw,gid=5,mode=620)
>> tmpfs on /dev/shm type tmpfs (rw)
>> /dev/sda1 on /boot type ext4 (rw)
>> none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
>> sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
>> nfsd on /proc/fs/nfsd type nfsd (rw)
>> 10.61.154.10:/Cloudstack/primary on
>> /mnt/b3647480-5709-36eb-9ddd-e2c666f30b57 type nfs
>> (rw,noatime,nfsvers=3,proto=tcp,hard,nolock,intr,addr=10.61.154.10)
>> [root@eqx-cs-cmp-01 ~]#
>>
>>
>> On 15 April 2016 at 15:22, Nezar Madbouh <ne...@ipscape.com.au>
>> wrote:
>>
>>> Hi Ilya,
>>>
>>> Thanks for the instructions. I followed them and pasted the information on
>>> pastebin:
>>>
>>> management-server logs job-14938
>>> http://pastebin.com/GAiy8u5H
>>>
>>> Agent logs
>>> http://pastebin.com/mfiADsJc
>>>
>>>
>>> Your help is much appreciated.
>>>
>>> Nezar.
>>>
>>> On 15 April 2016 at 13:58, ilya <il...@gmail.com> wrote:
>>>
>>>> Here is how i would troubleshoot this:
>>>>
>>>> 1) ssh to MS host, run
>>>>
>>>> tail -f /var/cloudstack/management/management-server.log | grep -i
>>>> StartVMCmdByAdmin
>>>>
>>>> 2) on cloudstack UI, press start on the VM and watch the tail log
>>>>
>>>> 3) on the tail log, you should see a StartVirtualMachine api call show
>>>> up with job ID. For example:
>>>>
>>>>> 2016-04-15 03:41:57,193 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (API-Job-Executor-24:ctx-7892387a job-291575) Executing AsyncJobVO
>>>> {id:291575, userId: 157, accountId: 2, instanceType: VirtualMachine,
>>>> instanceId: 7648, cmd:
>>>> org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin, cmdInfo:
>>>> {"id":"7648","response":"json","sessionkey":"mMJsduS9KLKApqH4dPpIlGZdt1Y\u003d","ctxDetails":"{\"com.cloud.vm.VirtualMachine\":7648,\"com.cloud.uservm.UserVm\":7648,\"com.cloud.network.router.VirtualRouter\":7648}","cmdEventType":"VM.START","expires":"2016-04-15T03:51:57+0000","ctxUserId":"157","signatureversion":"3","httpmethod":"GET","uuid":"7648","ctxAccountId":"2","ctxStartEventId":"314416"},
>>>> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
>>>> result: null, initMsid: 345051337894, completeMsid: null, lastUpdated:
>>>> null, lastPolled: null, created: null}
>>>>
>>>> Each task in cloudstack spins up its own job, in this case its
>>>> *job-291575*
>>>>
>>>> 4) now execute
>>>>> grep -A3 -B3 job-291575
>>>> /var/cloudstack/management/management-server.log | tee job-291575.log
>>>>> grep job-291575 /var/cloudstack/management/management-server.log | grep
>>>> via | head -1
>>>>
>>>> you should see a response like so,
>>>>
>>>>> 2016-04-15 03:41:59,259 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>>> (Work-Job-Executor-42:ctx-b2c4af6b job-291575/job-291576 ctx-0988152e) Seq
>>>> 441-4181310779036795138: Forwarding Seq 441-4181310779036795138: { Cmd ,
>>>> MgmtId: 345051337894, via: 441(my-kvm-host01.example.com), Ver: v1,
>>>> Flags: 100011,
>>>> [{"com.cloud.agent.api.routing.DhcpEntryCommand":{"vmMac":"06:fe:4e:00:12:ad","vmIpAddress":"10.10.27.178","vmName":"ol6-test-ilya1","defaultRouter":"10.10.27.1","defaultDns":"10.17.27.166","duid":"00:03:00:01:06:fe:4e:00:12:ad","isDefault":true,"executeInSequence":false,"accessDetails":{"zone.network.type":"Advanced","router.guest.ip":"10.10.27.166","router.ip":"169.254.0.238","
>>>> router.name":"r-7301-VM"},"wait":0}}] } to 345051337893
>>>>
>>>> We are looking for this line "via: 441(my-kvm-host01.example.com)"
>>>>
>>>> SSH to my-kvm-host01.example.com, and copy the
>>>> /var/log/cloudstack/agent.log to pastebin
>>>>
>>>> also copy job-291575.log to pastebin
>>>>
>>>> I'll try to have a look, though when i get chance.
>>>>
>>>>
>>>> Also paste the output of mount command on kvm host in question. I'm
>>>> suspecting one of your storage pools maybe bad.
>>>>
>>>> Lastly, what distro are you running for KVM?
>>>>
>>>> Regards
>>>> ilya
>>>>
>>>> On 4/14/16 5:05 PM, Nezar Madbouh wrote:
>>>>> Hi Ilya,
>>>>>
>>>>> I shutdown all the VMs in Pre-prod Environmnet expecting the Pre-prod
>>>>> Virtual router to be deleted automatically because no VM is running on
>>>> this
>>>>> vlan but it wasn't deleted.
>>>>>
>>>>> Am I right to expect the VR to be deleted automatically if no VM is
>>>> running
>>>>> in that vlan?
>>>>>
>>>>> Thanks,
>>>>> Nezar.
>>>>>
>>>>> On 15 April 2016 at 09:53, Nezar Madbouh <ne...@ipscape.com.au>
>>>>> wrote:
>>>>>
>>>>>> Hi Ilya,
>>>>>>
>>>>>> I have run the script and it didn't return any results:
>>>>>>
>>>>>> [root@eqx-cs-mgt-01 ~]# ./async_jobs.sh
>>>>>> [root@eqx-cs-mgt-01 ~]#
>>>>>>
>>>>>>
>>>>>> I also ran the command inside mysql:
>>>>>>
>>>>>> mysql> SELECT id, user_id, account_id, instance_type, instance_id,
>>>>>> job_cmd, job_cmd_info, job_dispatcher, job_pending_signals FROM
>>>>>> cloud.async_job where job_status=0 and job_dispatcher not like
>>>>>> "pseudoJobDispatcher";
>>>>>> Empty set (0.00 sec)
>>>>>>
>>>>>> your help is much appreciated,
>>>>>> Nezar.
>>>>>>
>>>>>>
>>>>>> On 14 April 2016 at 16:49, ilya <il...@gmail.com> wrote:
>>>>>>
>>>>>>> It appears you have pending jobs that are not letting you go through.
>>>>>>>
>>>>>>> Usually stop and starting management service should be enough.
>>>>>>>
>>>>>>> However, try running this script from your management server to show
>>>> us
>>>>>>> what you have running in async_job table.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> #!/bin/bash
>>>>>>>> DATESTAMP=$(date +%m%d%y-%H%M%S)
>>>>>>>> DBPASS=$(java -classpath
>>>>>>> /usr/share/cloudstack-common/lib/jasypt-1.9.0.jar
>>>>>>> org.jasypt.intf.cli.JasyptPBEStringDecryptionCLI input="$(cat
>>>>>>> /etc/cloudstack/management/db.properties | grep db.cloud.password |
>>>> awk
>>>>>>> -F'(' '{print $2}' | sed 's/)//g')" password="$(cat
>>>>>>> /etc/cloudstack/management/key)" | grep -A2 OUTPUT | tail -1)
>>>>>>>> DBHOST=$(cat /etc/cloudstack/management/db.properties | grep
>>>>>>> db.cloud.host | awk -F'=' '{print $2}' | tail -1 )
>>>>>>>> DBUSER=$(cat /etc/cloudstack/management/db.properties | grep
>>>>>>> db.cloud.username | awk -F'=' '{print $2}')
>>>>>>>> DB=$(cat /etc/cloudstack/management/db.properties | grep
>>>> db.cloud.name
>>>>>>> | awk -F'=' '{print $2}')
>>>>>>>> DBPORT=$(cat /etc/cloudstack/management/db.properties | grep
>>>>>>> db.cloud.port | awk -F'=' '{print $2}')
>>>>>>>> MYSQLCMD="mysql -h $DBHOST -u $DBUSER -P $DBPORT -p$DBPASS $DB"
>>>>>>>> echo 'SELECT id, user_id, account_id, instance_type, instance_id,
>>>>>>> job_cmd, job_cmd_info, job_dispatcher, job_pending_signals FROM
>>>>>>> cloud.async_job where job_status=0 and job_dispatcher not like
>>>>>>> "pseudoJobDispatcher"' | $MYSQLCMD
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 4/13/16 5:16 PM, Nezar Madbouh wrote:
>>>>>>>> Hi Simon,
>>>>>>>>
>>>>>>>> The following work was done but unfortunately the problem is not
>>>> fixed:
>>>>>>>>
>>>>>>>>
>>>>>>>> on management server:
>>>>>>>>
>>>>>>>> service cloudstack-management stop
>>>>>>>>
>>>>>>>>
>>>>>>>> on the host:
>>>>>>>>
>>>>>>>> service cloudstack-agent stop
>>>>>>>>
>>>>>>>> service libvirtd restart
>>>>>>>>
>>>>>>>> service cloudstack-agent start
>>>>>>>>
>>>>>>>>
>>>>>>>> on management server:
>>>>>>>>
>>>>>>>> service cloudstack-management start
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Nezar.
>>>>>>>>
>>>>>>>> On 24 March 2016 at 05:06, Simon Weller <sw...@ena.com> wrote:
>>>>>>>>
>>>>>>>>> Yes, I think that order is fine. Make sure the processes are indeed
>>>>>>>>> shutdown before moving on to the next step.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ________________________________________
>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>>>>>>>>> Sent: Tuesday, March 22, 2016 5:33 PM
>>>>>>>>> To: users@cloudstack.apache.org
>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable to
>>>> start
>>>>>>>>> instance due to Unable to get answer that is of class
>>>>>>>>> com.cloud.agent.api.StartAnswer
>>>>>>>>>
>>>>>>>>> Hi Simon,
>>>>>>>>>
>>>>>>>>> Can you please check if the following sequence of action is correct:
>>>>>>>>>
>>>>>>>>> on management server:
>>>>>>>>>
>>>>>>>>> service cloudstack-management stop
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> on the host:
>>>>>>>>>
>>>>>>>>> service cloudstack-agent stop
>>>>>>>>>
>>>>>>>>> service libvirtd restart
>>>>>>>>>
>>>>>>>>> service cloudstack-agent start
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> on management server:
>>>>>>>>>
>>>>>>>>> service cloudstack-management start
>>>>>>>>>
>>>>>>>>> Thanks very much for your help.
>>>>>>>>> Nezar.
>>>>>>>>>
>>>>>>>>> On 23 March 2016 at 08:39, Simon Weller <sw...@ena.com> wrote:
>>>>>>>>>
>>>>>>>>>> Shutting down all the management services should clear the jobs.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ________________________________________
>>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>>>>>>>>>> Sent: Tuesday, March 22, 2016 4:28 PM
>>>>>>>>>> To: users@cloudstack.apache.org
>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable to
>>>>>>> start
>>>>>>>>>> instance due to Unable to get answer that is of class
>>>>>>>>>> com.cloud.agent.api.StartAnswer
>>>>>>>>>>
>>>>>>>>>> Hi Simon,
>>>>>>>>>>
>>>>>>>>>> I will plan to restart agents, libvirtd and management services.
>>>>>>>>>>
>>>>>>>>>> How do I clear hung jobs.
>>>>>>>>>>
>>>>>>>>>> Your help is much appreciated,
>>>>>>>>>> Nezar.
>>>>>>>>>>
>>>>>>>>>> On 22 March 2016 at 23:03, Simon Weller <sw...@ena.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Nezar,
>>>>>>>>>>>
>>>>>>>>>>> I've looked through the code related to your release and it seems
>>>>>>> that
>>>>>>>>> in
>>>>>>>>>>> order to throw this exception, 10 attempts have been made to
>>>> retrieve
>>>>>>>>> the
>>>>>>>>>>> disk information prior to the exception being thrown.
>>>>>>>>>>>
>>>>>>>>>>> It appears there is something really odd going on here. I'm
>>>> starting
>>>>>>>>> to
>>>>>>>>>>> wonder whether it might be worth while for you to stop all of your
>>>>>>>>>> agents,
>>>>>>>>>>> plus management server processes, restart libvirtd on each of the
>>>>>>> hosts
>>>>>>>>>> and
>>>>>>>>>>> then bring the agents back up prior to starting the management
>>>>>>> servers.
>>>>>>>>>>>
>>>>>>>>>>> There are a bunch of logs the pastebin you posted that seems to
>>>>>>>>> indicate
>>>>>>>>>>> you have some jobs hung, for example:
>>>>>>>>>>>
>>>>>>>>>>> 2016-03-14 21:45:31,828 WARN [o.a.c.f.j.i.AsyncJobMonitor]
>>>>>>>>>>> (Timer-1:ctx-0c090bd7) Task (job-9066) has been pending for
>>>> 1660851
>>>>>>>>>> seconds
>>>>>>>>>>> 2016-03-14 21:45:31,828 WARN [o.a.c.f.j.i.AsyncJobMonitor]
>>>>>>>>>>> (Timer-1:ctx-0c090bd7) Task (job-8133) has been pending for
>>>> 4253003
>>>>>>>>>> seconds
>>>>>>>>>>>
>>>>>>>>>>> Restarting the agents, libvirtd and management will not cause an
>>>>>>>>> outage
>>>>>>>>>>> to your cloud, but I would suggest, if you decided to do this, you
>>>>>>>>> should
>>>>>>>>>>> complete it in a maintenance window.
>>>>>>>>>>>
>>>>>>>>>>> - Si
>>>>>>>>>>>
>>>>>>>>>>> ________________________________________
>>>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>>>>>>>>>>> Sent: Sunday, March 20, 2016 10:27 PM
>>>>>>>>>>> To: users@cloudstack.apache.org
>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable to
>>>>>>> start
>>>>>>>>>>> instance due to Unable to get answer that is of class
>>>>>>>>>>> com.cloud.agent.api.StartAnswer
>>>>>>>>>>>
>>>>>>>>>>> Hi Simon,
>>>>>>>>>>>
>>>>>>>>>>> Please see answers to your questions below. I am not sure if I
>>>>>>> provided
>>>>>>>>>> the
>>>>>>>>>>> correct answer for the primary and secondary storage
>>>> configuration.
>>>>>>>>>> Please
>>>>>>>>>>> let me know if this is not what you are after:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Can you screenshot and post somewhere your secondary storage
>>>>>>>>>> configuration
>>>>>>>>>>> and your primary storage configuration from the UI with the
>>>> details?
>>>>>>>>>>>
>>>>>>>>>>> mysql> select * from configuration where name like '%secondary%'\G
>>>>>>>>>>> *************************** 1. row ***************************
>>>>>>>>>>> category: Account Defaults
>>>>>>>>>>> instance: DEFAULT
>>>>>>>>>>> component: management-server
>>>>>>>>>>> name: max.account.secondary.storage
>>>>>>>>>>> value: 400
>>>>>>>>>>> description: The default maximum secondary storage space (in
>>>> GiB)
>>>>>>>>> that
>>>>>>>>>>> can be used for an account
>>>>>>>>>>> default_value: 400
>>>>>>>>>>> updated: NULL
>>>>>>>>>>> scope: NULL
>>>>>>>>>>> is_dynamic: 0
>>>>>>>>>>> *************************** 2. row ***************************
>>>>>>>>>>> category: Project Defaults
>>>>>>>>>>> instance: DEFAULT
>>>>>>>>>>> component: management-server
>>>>>>>>>>> name: max.project.secondary.storage
>>>>>>>>>>> value: 400
>>>>>>>>>>> description: The default maximum secondary storage space (in
>>>> GiB)
>>>>>>>>> that
>>>>>>>>>>> can be used for a project
>>>>>>>>>>> default_value: 400
>>>>>>>>>>> updated: NULL
>>>>>>>>>>> scope: NULL
>>>>>>>>>>> is_dynamic: 0
>>>>>>>>>>> *************************** 3. row ***************************
>>>>>>>>>>> category: Hidden
>>>>>>>>>>> instance: DEFAULT
>>>>>>>>>>> component: management-server
>>>>>>>>>>> name: secondary.storage.vm
>>>>>>>>>>> value: +rwRhJOXPymbQOuAMLnhiw==
>>>>>>>>>>> description: Deploys a VM per zone to manage secondary storage
>>>> if
>>>>>>>>> true,
>>>>>>>>>>> otherwise secondary storage is mounted on management server
>>>>>>>>>>> default_value: NULL
>>>>>>>>>>> updated: NULL
>>>>>>>>>>> scope: NULL
>>>>>>>>>>> is_dynamic: 0
>>>>>>>>>>> *************************** 4. row ***************************
>>>>>>>>>>> category: Advanced
>>>>>>>>>>> instance: DEFAULT
>>>>>>>>>>> component: ExtensionRegistry
>>>>>>>>>>> name: secondary.storage.vm.allocators.exclude
>>>>>>>>>>> value: NULL
>>>>>>>>>>> description: Extensions to exclude from being registered
>>>>>>>>>>> default_value: NULL
>>>>>>>>>>> updated: 2014-12-04 00:18:15
>>>>>>>>>>> scope: Global
>>>>>>>>>>> is_dynamic: 0
>>>>>>>>>>> *************************** 5. row ***************************
>>>>>>>>>>> category: NetworkManager
>>>>>>>>>>> instance: DEFAULT
>>>>>>>>>>> component: management-server
>>>>>>>>>>> name: vm.network.nic.max.secondary.ipaddresses
>>>>>>>>>>> value: NULL
>>>>>>>>>>> description: Specify the number of secondary ip addresses per
>>>> nic
>>>>>>> per
>>>>>>>>>> vm
>>>>>>>>>>> default_value: 256
>>>>>>>>>>> updated: NULL
>>>>>>>>>>> scope: NULL
>>>>>>>>>>> is_dynamic: 0
>>>>>>>>>>> 5 rows in set (0.00 sec)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> mysql> select * from configuration where name like '%primary%'\G
>>>>>>>>>>> *************************** 1. row ***************************
>>>>>>>>>>> category: Account Defaults
>>>>>>>>>>> instance: DEFAULT
>>>>>>>>>>> component: management-server
>>>>>>>>>>> name: max.account.primary.storage
>>>>>>>>>>> value: 200
>>>>>>>>>>> description: The default maximum primary storage space (in GiB)
>>>>>>> that
>>>>>>>>>> can
>>>>>>>>>>> be used for an account
>>>>>>>>>>> default_value: 200
>>>>>>>>>>> updated: NULL
>>>>>>>>>>> scope: NULL
>>>>>>>>>>> is_dynamic: 0
>>>>>>>>>>> *************************** 2. row ***************************
>>>>>>>>>>> category: Project Defaults
>>>>>>>>>>> instance: DEFAULT
>>>>>>>>>>> component: management-server
>>>>>>>>>>> name: max.project.primary.storage
>>>>>>>>>>> value: 200
>>>>>>>>>>> description: The default maximum primary storage space (in GiB)
>>>>>>> that
>>>>>>>>>> can
>>>>>>>>>>> be used for a project
>>>>>>>>>>> default_value: 200
>>>>>>>>>>> updated: NULL
>>>>>>>>>>> scope: NULL
>>>>>>>>>>> is_dynamic: 0
>>>>>>>>>>> *************************** 3. row ***************************
>>>>>>>>>>> category: Storage
>>>>>>>>>>> instance: DEFAULT
>>>>>>>>>>> component: TemplateManager
>>>>>>>>>>> name: primary.storage.download.wait
>>>>>>>>>>> value: 10800
>>>>>>>>>>> description: In second, timeout for download template to primary
>>>>>>>>>> storage
>>>>>>>>>>> default_value: NULL
>>>>>>>>>>> updated: NULL
>>>>>>>>>>> scope: NULL
>>>>>>>>>>> is_dynamic: 0
>>>>>>>>>>> 3 rows in set (0.00 sec)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Also, what is the Cloudstack version you are running and what is
>>>> the
>>>>>>> OS
>>>>>>>>>>> you're running it on?
>>>>>>>>>>>
>>>>>>>>>>> Cloudstack 4.4.1
>>>>>>>>>>>
>>>>>>>>>>> Centos 6.6
>>>>>>>>>>>
>>>>>>>>>>> Are you using actimeo=0 in your NFS mount line?
>>>>>>>>>>>
>>>>>>>>>>> Primary storage on Hypervisors has the following parameters
>>>>>>>>>>> 10.61.154.10:/Cloudstack/primary on
>>>>>>>>>>> /mnt/b3647480-5709-36eb-9ddd-e2c666f30b57 type nfs
>>>>>>>>>>>
>>>> (rw,noatime,nfsvers=3,proto=tcp,hard,nolock,intr,addr=10.61.154.10)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Secondary storage on SSVM has the following parameters
>>>>>>>>>>> 10.61.154.10:/Cloudstack/secondary on
>>>>>>>>>>> /mnt/SecStorage/d11d5c92-6950-30e8-a0b5-3cd46a17ec14 type nfs4
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>> (rw,relatime,vers=4,rsize=262144,wsize=262144,namlen=255,acdirmin=0,acdirmax=0,soft,proto=tcp,port=0,timeo=133,retrans=
>>>>>>> 2147483647
>>>>>>>
>>>> ,sec=sys,clientaddr=10.61.154.70,minorversion=0,local_lock=none,addr=10.61.154.10)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Thanks & Regards,
>>>>>>>>>>> Nezar.
>>>>>>>>>>>
>>>>>>>>>>> On 18 March 2016 at 23:50, Simon Weller <sw...@ena.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Also, what is the Cloudstack version you are running and what is
>>>> the
>>>>>>>>> OS
>>>>>>>>>>>> you're running it on?
>>>>>>>>>>>>
>>>>>>>>>>>> Are you using actimeo=0 in your NFS mount line?
>>>>>>>>>>>>
>>>>>>>>>>>> ________________________________________
>>>>>>>>>>>> From: Simon Weller <sw...@ena.com>
>>>>>>>>>>>> Sent: Friday, March 18, 2016 7:40 AM
>>>>>>>>>>>> To: users@cloudstack.apache.org
>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable to
>>>>>>>>> start
>>>>>>>>>>>> instance due to Unable to get answer that is of class
>>>>>>>>>>>> com.cloud.agent.api.StartAnswer
>>>>>>>>>>>>
>>>>>>>>>>>> Nexar,
>>>>>>>>>>>>
>>>>>>>>>>>> That's normal behaviour. You can only ssh to the link local on
>>>> the
>>>>>>>>> host
>>>>>>>>>>>> where the VM resides.
>>>>>>>>>>>>
>>>>>>>>>>>> Can you screenshot and post somewhere your secondary storage
>>>>>>>>>>> configuration
>>>>>>>>>>>> and your primary storage configuration from the UI with the
>>>> details?
>>>>>>>>>>>>
>>>>>>>>>>>> - Si
>>>>>>>>>>>>
>>>>>>>>>>>> ________________________________________
>>>>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>>>>>>>>>>>> Sent: Thursday, March 17, 2016 8:19 PM
>>>>>>>>>>>> To: users@cloudstack.apache.org
>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable to
>>>>>>>>> start
>>>>>>>>>>>> instance due to Unable to get answer that is of class
>>>>>>>>>>>> com.cloud.agent.api.StartAnswer
>>>>>>>>>>>>
>>>>>>>>>>>> Hi Simon,
>>>>>>>>>>>>
>>>>>>>>>>>> I just noticed that from other Hosts I can't ssh into the SSVM:
>>>>>>>>>>>>
>>>>>>>>>>>> [root@eqx-cs-cmp-01 ~]# ssh -i /root/.ssh/id_rsa.cloud -p 3922
>>>>>>>>>>>> root@169.254.3.161
>>>>>>>>>>>> ssh: connect to host 169.254.3.161 port 3922: No route to host
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I can only ssh from one Host:
>>>>>>>>>>>> [root@eqx-cs-cmp-06 ~]# ssh -i /root/.ssh/id_rsa.cloud -p 3922
>>>>>>>>>>>> root@169.254.3.161
>>>>>>>>>>>> Linux s-11-VM 3.2.0-4-amd64 #1 SMP Debian 3.2.57-3 x86_64
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks & Regards,
>>>>>>>>>>>> Nezar.
>>>>>>>>>>>>
>>>>>>>>>>>> On 18 March 2016 at 10:36, Nezar Madbouh <
>>>>>>>>> nezar.madbouh@ipscape.com.au
>>>>>>>>>>>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Simon,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Yes existing VMs are successfully running. Yesterday I was able
>>>> to
>>>>>>>>>>> create
>>>>>>>>>>>>> a VM in Dev Environment but I can't create VMs in Prod
>>>> environment.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> Nezar.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 18 March 2016 at 09:12, Simon Weller <sw...@ena.com>
>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> I'm assuming you have other VMs successfully running, correct?
>>>> So
>>>>>>>>>> you
>>>>>>>>>>>> can
>>>>>>>>>>>>>> write to your primary storage.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ________________________________________
>>>>>>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>>>>>>>>>>>>>> Sent: Thursday, March 17, 2016 4:43 PM
>>>>>>>>>>>>>> To: users@cloudstack.apache.org
>>>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable
>>>> to
>>>>>>>>>>> start
>>>>>>>>>>>>>> instance due to Unable to get answer that is of class
>>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Simon and Shweta,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Please see the result below:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> root@s-11-VM:~# /usr/local/cloud/systemvm/ssvm-check.sh
>>>>>>>>>>>>>> ================================================
>>>>>>>>>>>>>> First DNS server is 8.8.8.8
>>>>>>>>>>>>>> PING 8.8.8.8 (8.8.8.8): 48 data bytes
>>>>>>>>>>>>>> 56 bytes from 8.8.8.8: icmp_seq=0 ttl=57 time=5.209 ms
>>>>>>>>>>>>>> 56 bytes from 8.8.8.8: icmp_seq=1 ttl=57 time=0.936 ms
>>>>>>>>>>>>>> --- 8.8.8.8 ping statistics ---
>>>>>>>>>>>>>> 2 packets transmitted, 2 packets received, 0% packet loss
>>>>>>>>>>>>>> round-trip min/avg/max/stddev = 0.936/3.072/5.209/2.137 ms
>>>>>>>>>>>>>> Good: Can ping DNS server
>>>>>>>>>>>>>> ================================================
>>>>>>>>>>>>>> Good: DNS resolves download.cloud.com
>>>>>>>>>>>>>> ================================================
>>>>>>>>>>>>>> nfs is currently mounted
>>>>>>>>>>>>>> Mount point is
>>>>>>>>> /mnt/SecStorage/d11d5c92-6950-30e8-a0b5-3cd46a17ec14
>>>>>>>>>>>>>> Good: Can write to mount point
>>>>>>>>>>>>>> ================================================
>>>>>>>>>>>>>> Management server is 10.61.153.20. Checking connectivity.
>>>>>>>>>>>>>> Good: Can connect to management server port 8250
>>>>>>>>>>>>>> ================================================
>>>>>>>>>>>>>> Good: Java process is running
>>>>>>>>>>>>>> ================================================
>>>>>>>>>>>>>> Tests Complete. Look for ERROR or WARNING above.
>>>>>>>>>>>>>> root@s-11-VM:~#
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks & Regards,
>>>>>>>>>>>>>> Nezar.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 17 March 2016 at 09:42, Simon Weller <sw...@ena.com>
>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Nezar,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Log into the Cloudstack UI.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Click on "Infrastructure" on the left menu. Click on View All
>>>>>>>>>> below
>>>>>>>>>>>>>>> "System VMs". find the item marked "Secondary Storage VM" and
>>>>>>>>>> then
>>>>>>>>>>>>>> click
>>>>>>>>>>>>>>> on the Name of that VM on the left (it will be s-XXXX-VM).
>>>>>>>>> You'll
>>>>>>>>>>> see
>>>>>>>>>>>> a
>>>>>>>>>>>>>>> bunch of information about the VM. Find the Host line and also
>>>>>>>>> the
>>>>>>>>>>>> Link
>>>>>>>>>>>>>>> Local IP Address line and then follow the original
>>>> instructions
>>>>>>>>> in
>>>>>>>>>>> my
>>>>>>>>>>>>>>> previous email.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> - Si
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ________________________________________
>>>>>>>>>>>>>>> From: Shweta Agarwal <sh...@accelerite.com>
>>>>>>>>>>>>>>> Sent: Wednesday, March 16, 2016 12:16 AM
>>>>>>>>>>>>>>> To: users@cloudstack.apache.org
>>>>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable
>>>>>>>>> to
>>>>>>>>>>>> start
>>>>>>>>>>>>>>> instance due to Unable to get answer that is of class
>>>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Nezar,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> You can find host name on detail view of SSVM in UI .
>>>>>>>>>>>>>>> You can also use list api command listSystemVms to get the
>>>> host
>>>>>>>>>>>> details
>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>> system vms.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>> Shweta
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ________________________________________
>>>>>>>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>>>>>>>>>>>>>>> Sent: Wednesday, March 16, 2016 9:58 AM
>>>>>>>>>>>>>>> To: users@cloudstack.apache.org
>>>>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable
>>>>>>>>> to
>>>>>>>>>>>> start
>>>>>>>>>>>>>>> instance due to Unable to get answer that is of class
>>>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Simon,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> How can I find out which host the SSVM is on?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks & Regards,
>>>>>>>>>>>>>>> Nezar.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 16 March 2016 at 12:28, Simon Weller <sw...@ena.com>
>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Can you ssh into the Secondary Storage VM?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> This can be done by getting the link-local-ip address from
>>>> the
>>>>>>>>>>>>>>>> infrastructure menu in the UI and then sshing into the host
>>>>>>>>> the
>>>>>>>>>>> SSVM
>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>> and issuing this command as root: ssh -i
>>>>>>>>> /root/.ssh/id_rsa.cloud
>>>>>>>>>>> -p
>>>>>>>>>>>>>> 3922
>>>>>>>>>>>>>>>> root@link-local-ip (replace link-local-ip with the link
>>>>>>>>> local
>>>>>>>>>>> ip).
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Try running the SSVM diangnostic script:
>>>>>>>>>>>>>>>> usr/local/cloud/systemvm/ssvm-check.sh
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> - Si
>>>>>>>>>>>>>>>> ________________________________________
>>>>>>>>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>>>>>>>>>>>>>>>> Sent: Tuesday, March 15, 2016 7:49 PM
>>>>>>>>>>>>>>>> To: users@cloudstack.apache.org
>>>>>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message:
>>>> Unable
>>>>>>>>>> to
>>>>>>>>>>>>>> start
>>>>>>>>>>>>>>>> instance due to Unable to get answer that is of class
>>>>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi Simon,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Please see details below:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> mysql> use cloud; select id, name, status, type from host
>>>>>>>>> where
>>>>>>>>>>> id =
>>>>>>>>>>>>>> 5;
>>>>>>>>>>>>>>>> Reading table information for completion of table and column
>>>>>>>>>> names
>>>>>>>>>>>>>>>> You can turn off this feature to get a quicker startup with
>>>> -A
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Database changed
>>>>>>>>>>>>>>>> +----+------------------------------+--------+---------+
>>>>>>>>>>>>>>>> | id | name | status | type |
>>>>>>>>>>>>>>>> +----+------------------------------+--------+---------+
>>>>>>>>>>>>>>>> | 5 | eqx-cs-cmp-02.ipscape.com.au | up | routing |
>>>>>>>>>>>>>>>> +----+------------------------------+--------+---------+
>>>>>>>>>>>>>>>> 1 row in set (0.00 sec)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> cat /var/log/cloudstack/agent/agent.log
>>>>>>>>>>>>>>>> com.cloud.utils.exception.CloudRuntimeException:
>>>>>>>>>>>>>>>> com.cloud.utils.exception.CloudRuntimeException: Can't find
>>>>>>>>>>>>>>>> volume:089ff3ca-9bff-402c-a14e-ea7a35aaad5c
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>> com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.getPhysicalDisk(KVMStoragePoolManager.java:262)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.createVbd(LibvirtComputingResource.java:3921)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(LibvirtComputingResource.java:3789)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1333)
>>>>>>>>>>>>>>>> at
>>>>>>>>> com.cloud.agent.Agent.processRequest(Agent.java:501)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>
>>>> com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:808)
>>>>>>>>>>>>>>>> at com.cloud.utils.nio.Task.run(Task.java:84)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>>>>>>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>>>>>>>>>>> agent.log.2016-03-03.gz (END)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> [root@eqx-cs-cmp-02 agent]# virsh pool-list
>>>>>>>>>>>>>>>> Name State Autostart
>>>>>>>>>>>>>>>> -----------------------------------------
>>>>>>>>>>>>>>>> 2d0967f8-50d6-4d6d-87b7-4da0596f6924 active no
>>>>>>>>>>>>>>>> b3647480-5709-36eb-9ddd-e2c666f30b57 active no
>>>>>>>>>>>>>>>> e3831246-ff55-4af4-9964-a17565edaccc active no
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> [Root @ eqx-cs-cmp-02 agent] # df -h
>>>>>>>>>>>>>>>> Filesystem Size Used Avail Use% Mounted on
>>>>>>>>>>>>>>>> /dev/mapper/vg_root-lv_root
>>>>>>>>>>>>>>>> 203G 215G 1.3G 1% /
>>>>>>>>>>>>>>>> tmpfs 48g 48g 0 0% / dev / shm
>>>>>>>>>>>>>>>> / Dev / sda1 477m 427m 25M 6% / boot
>>>>>>>>>>>>>>>> xx.xx.xx.xx:/Cloudstack/primary
>>>>>>>>>>>>>>>> 7.6T 4.1T 3.5T 54%
>>>>>>>>>>>>>>>> /mnt/b3647480-5709-36eb-9ddd-e2c666f30b57
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I also noticed the following in cloudstack-agent.err
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> libvirt: QEMU Driver error : Domain not found: no domain with
>>>>>>>>>>>> matching
>>>>>>>>>>>>>>> name
>>>>>>>>>>>>>>>> 'i-11-600-VM'
>>>>>>>>>>>>>>>> libvirt: Storage Driver error : Storage volume not found: no
>>>>>>>>>>> storage
>>>>>>>>>>>>>> vol
>>>>>>>>>>>>>>>> with matching name '089ff3ca-9bff-402c-a14e-ea7a35aaad5c'
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Your help is much appreciated,
>>>>>>>>>>>>>>>> Nezar
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On 16 March 2016 at 11:03, Simon Weller <sw...@ena.com>
>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Nezar,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> ok, I just took a look at a pastebin you posted.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I see it complaining about the inability to find a volume
>>>>>>>>>> prior
>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> throwing the host exception.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Firstly, let's find out who host 5 is. Can you log into your
>>>>>>>>>>> MySQL
>>>>>>>>>>>>>>> server
>>>>>>>>>>>>>>>>> and run this:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> use cloud;select id,name,status,type from host where id=5;
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Can you post the results?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> If it's a host, can you post the
>>>>>>>>>>>> /var/log/cloudstack/agent/agent.log
>>>>>>>>>>>>>>> from
>>>>>>>>>>>>>>>>> that host?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> And also run virsh pool-list on the host in question?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> - Si
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> ________________________________________
>>>>>>>>>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>>>>>>>>>>>>>>>>> Sent: Tuesday, March 15, 2016 6:11 PM
>>>>>>>>>>>>>>>>> To: users@cloudstack.apache.org
>>>>>>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message:
>>>>>>>>> Unable
>>>>>>>>>>> to
>>>>>>>>>>>>>> start
>>>>>>>>>>>>>>>>> instance due to Unable to get answer that is of class
>>>>>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi Simon,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> When you say "double check your routers tied to the
>>>>>>>>> network",
>>>>>>>>>> do
>>>>>>>>>>>> you
>>>>>>>>>>>>>>> mean
>>>>>>>>>>>>>>>>> the physical routers or the virtual routers?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> When I try to create a VM in Prod Environment it fails. I
>>>>>>>>>> can't
>>>>>>>>>>>> ping
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> Prod virtual router from management server or Host.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> But I can ping Dev virtual router from management server and
>>>>>>>>>>> Host
>>>>>>>>>>>>>> and I
>>>>>>>>>>>>>>>> was
>>>>>>>>>>>>>>>>> able to create a VM in Dev Environment.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> How can I troubleshoot the Prod Virtual router. I launched
>>>>>>>>> the
>>>>>>>>>>>>>> console
>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>> couldn't login.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Please note that we need to take into consideration that the
>>>>>>>>>>>>>> affected
>>>>>>>>>>>>>>>>> network is the Production network and we need to minimise
>>>>>>>>>>> service
>>>>>>>>>>>>>>> outage
>>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>>> much as possible.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Your help is much appreciated,
>>>>>>>>>>>>>>>>> Nezar
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On 16 March 2016 at 08:37, Simon Weller <sw...@ena.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I've seen this before where a Virtual Router has crashed
>>>>>>>>> and
>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> management server can't communicate with it.
>>>>>>>>>>>>>>>>>> Are you using an advanced network setup?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> If so, double check your routers tied to the network where
>>>>>>>>>> you
>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>>> attempting to start an instance.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> - Si
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> ________________________________________
>>>>>>>>>>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>>>>>>>>>>>>>>>>>> Sent: Tuesday, March 15, 2016 4:02 PM
>>>>>>>>>>>>>>>>>> To: users@cloudstack.apache.org
>>>>>>>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message:
>>>>>>>>>> Unable
>>>>>>>>>>>> to
>>>>>>>>>>>>>>> start
>>>>>>>>>>>>>>>>>> instance due to Unable to get answer that is of class
>>>>>>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi Glenn,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> We have 16 hosts, all KVM.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> The error is not specific to Host 5. Every time I try to
>>>>>>>>>>> create
>>>>>>>>>>>> a
>>>>>>>>>>>>>> VM
>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>> gives me the same error message but on different host.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> How do I restart the agent on the host. Is it service
>>>>>>>>>>> affecting?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks very much for your help.
>>>>>>>>>>>>>>>>>> Neza
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On 15 March 2016 at 17:37, Glenn Wagner <
>>>>>>>>>>>>>> glenn.wagner@shapeblue.com>
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Can you explain your environment first , how many hosts
>>>>>>>>> ,
>>>>>>>>>>> what
>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>> your
>>>>>>>>>>>>>>>>>>> hypervisors (KVM,Xenserver,VMware)
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> The error seems to be with your Host 5 Hypervisor
>>>>>>>>>>>>>>>>>>> Resource [Host:5] is unreachable:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Go to the infrastructure section and see what host5 is
>>>>>>>>>>>>>> reporting,
>>>>>>>>>>>>>>>>>>> Have you tried to restart the agent on the Host?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>>> Glenn
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> [image: ShapeBlue] <http://www.shapeblue.com>
>>>>>>>>>>>>>>>>>>> Glenn Wagner
>>>>>>>>>>>>>>>>>>> Senior Consultant , ShapeBlue
>>>>>>>>>>>>>>>>>>> d: * | s: +27 21 527 0091*
>>>>>>>>>>>>>> <%7C%20s:%20+27%2021%20527%200091> |
>>>>>>>>>>>>>>>> m:
>>>>>>>>>>>>>>>>>>> *+27 73 917 4111* <+27%2073%20917%204111>
>>>>>>>>>>>>>>>>>>> e: *glenn.wagner@shapeblue.com | t: *
>>>>>>>>>>>>>>>>>>> <glenn.wagner@shapeblue.com%20%7C%20t:> | w: *
>>>>>>>>>>>>>> www.shapeblue.com
>>>>>>>>>>>>>>> *
>>>>>>>>>>>>>>>>>>> <http://www.shapeblue.com>
>>>>>>>>>>>>>>>>>>> a:
>>>>>>>>>>>>>>>>>>> 2nd Floor, Oudehuis Centre, 122 Main Rd, Somerset West
>>>>>>>>>> Cape
>>>>>>>>>>>> Town
>>>>>>>>>>>>>>> 7130
>>>>>>>>>>>>>>>>>> South Africa
>>>>>>>>>>>>>>>>>>> Shape Blue Ltd is a company incorporated in England &
>>>>>>>>>> Wales.
>>>>>>>>>>>>>>>> ShapeBlue
>>>>>>>>>>>>>>>>>>> Services India LLP is a company incorporated in India
>>>>>>>>> and
>>>>>>>>>> is
>>>>>>>>>>>>>>> operated
>>>>>>>>>>>>>>>>>> under
>>>>>>>>>>>>>>>>>>> license from Shape Blue Ltd. Shape Blue Brasil
>>>>>>>>> Consultoria
>>>>>>>>>>>> Ltda
>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>> company incorporated in Brasil and is operated under
>>>>>>>>>> license
>>>>>>>>>>>>>> from
>>>>>>>>>>>>>>>> Shape
>>>>>>>>>>>>>>>>>>> Blue Ltd. ShapeBlue SA Pty Ltd is a company registered
>>>>>>>>> by
>>>>>>>>>>> The
>>>>>>>>>>>>>>>> Republic
>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>> South Africa and is traded under license from Shape Blue
>>>>>>>>>>> Ltd.
>>>>>>>>>>>>>>>> ShapeBlue
>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>> a registered trademark.
>>>>>>>>>>>>>>>>>>> This email and any attachments to it may be confidential
>>>>>>>>>> and
>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>> intended
>>>>>>>>>>>>>>>>>>> solely for the use of the individual to whom it is
>>>>>>>>>>> addressed.
>>>>>>>>>>>>>> Any
>>>>>>>>>>>>>>>> views
>>>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>> opinions expressed are solely those of the author and do
>>>>>>>>>> not
>>>>>>>>>>>>>>>>> necessarily
>>>>>>>>>>>>>>>>>>> represent those of Shape Blue Ltd or related companies.
>>>>>>>>> If
>>>>>>>>>>> you
>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> intended recipient of this email, you must neither take
>>>>>>>>>> any
>>>>>>>>>>>>>> action
>>>>>>>>>>>>>>>>> based
>>>>>>>>>>>>>>>>>>> upon its contents, nor copy or show it to anyone. Please
>>>>>>>>>>>> contact
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> sender
>>>>>>>>>>>>>>>>>>> if you believe you have received this email in error.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>>>>>>>>>> From: Nezar Madbouh [mailto:
>>>>>>>>> nezar.madbouh@ipscape.com.au]
>>>>>>>>>>>>>>>>>>> Sent: Tuesday, 15 March 2016 5:45 AM
>>>>>>>>>>>>>>>>>>> To: users@cloudstack.apache.org
>>>>>>>>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message:
>>>>>>>>>>> Unable
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> start
>>>>>>>>>>>>>>>>>>> instance due to Unable to get answer that is of class
>>>>>>>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi Jeremy,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I have uploaded the contents to
>>>>>>>>>>> http://pastebin.com/1J3mmgTh
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> your help is much appreciated,
>>>>>>>>>>>>>>>>>>> Nezar.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On 15 March 2016 at 13:46, Jeremy Peterson <
>>>>>>>>>>>>>> jpeterson@acentek.net>
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I don't see any attachments can you create a pastebin.
>>>>>>>>>>>> That's
>>>>>>>>>>>>>>>> easier
>>>>>>>>>>>>>>>>>>>> to look through as I think attachments are removed
>>>>>>>>>> before
>>>>>>>>>>>>>> being
>>>>>>>>>>>>>>>> sent
>>>>>>>>>>>>>>>>>>>> to user group.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Jeremy
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> -------- Original message --------
>>>>>>>>>>>>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>>>>>>>>>>>>>>>>>>>> Date: 3/14/2016 6:38 PM (GMT-06:00)
>>>>>>>>>>>>>>>>>>>> To: users@cloudstack.apache.org
>>>>>>>>>>>>>>>>>>>> Subject: Unable to create a New VM - Error message:
>>>>>>>>>> Unable
>>>>>>>>>>>> to
>>>>>>>>>>>>>>> start
>>>>>>>>>>>>>>>>>>>> instance due to Unable to get answer that is of class
>>>>>>>>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hi Shweta and Sadhu,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Please find attached managerment-server.log which
>>>>>>>>>> contains
>>>>>>>>>>>> the
>>>>>>>>>>>>>>> logs
>>>>>>>>>>>>>>>>>>>> since I started creating a new VM (VM id 600).
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Please note that we don't have agent.log files
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Your help is much appreciated.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>>> Nezar.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Find out more about ShapeBlue and our range of
>>>>>>>>> CloudStack
>>>>>>>>>>>>>> related
>>>>>>>>>>>>>>>>>> services:
>>>>>>>>>>>>>>>>>>> IaaS Cloud Design & Build
>>>>>>>>>>>>>>>>>>> <http://shapeblue.com/iaas-cloud-design-and-build//> |
>>>>>>>>>>>> CSForge
>>>>>>>>>>>>>> -
>>>>>>>>>>>>>>>> rapid
>>>>>>>>>>>>>>>>>>> IaaS deployment framework <
>>>>>>>>> http://shapeblue.com/csforge/>
>>>>>>>>>>>>>>>>>>> CloudStack Consulting <
>>>>>>>>>>>>>>> http://shapeblue.com/cloudstack-consultancy/>
>>>>>>>>>>>>>>>> |
>>>>>>>>>>>>>>>>>> CloudStack
>>>>>>>>>>>>>>>>>>> Software Engineering
>>>>>>>>>>>>>>>>>>> <http://shapeblue.com/cloudstack-software-engineering/>
>>>>>>>>>>>>>>>>>>> CloudStack Infrastructure Support
>>>>>>>>>>>>>>>>>>> <
>>>>>>>>> http://shapeblue.com/cloudstack-infrastructure-support/>
>>>>>>>>>> |
>>>>>>>>>>>>>>>> CloudStack
>>>>>>>>>>>>>>>>>>> Bootcamp Training Courses <
>>>>>>>>>>>>>>> http://shapeblue.com/cloudstack-training/
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> DISCLAIMER
>>>>>>>>>>>>>>> ==========
>>>>>>>>>>>>>>> This e-mail may contain privileged and confidential
>>>> information
>>>>>>>>>>> which
>>>>>>>>>>>> is
>>>>>>>>>>>>>>> the property of Accelerite, a Persistent Systems business. It
>>>> is
>>>>>>>>>>>>>> intended
>>>>>>>>>>>>>>> only for the use of the individual or entity to which it is
>>>>>>>>>>> addressed.
>>>>>>>>>>>>>> If
>>>>>>>>>>>>>>> you are not the intended recipient, you are not authorized to
>>>>>>>>>> read,
>>>>>>>>>>>>>> retain,
>>>>>>>>>>>>>>> copy, print, distribute or use this message. If you have
>>>>>>>>> received
>>>>>>>>>>> this
>>>>>>>>>>>>>>> communication in error, please notify the sender and delete
>>>> all
>>>>>>>>>>> copies
>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>> this message. Accelerite, a Persistent Systems business does
>>>> not
>>>>>>>>>>>> accept
>>>>>>>>>>>>>> any
>>>>>>>>>>>>>>> liability for virus infected mails.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>
>>
Re: Unable to create a New VM - Error message: Unable to start instance
due to Unable to get answer that is of class com.cloud.agent.api.StartAnswer
Posted by ilya <il...@gmail.com>.
Please also upload logs from
eqx-cs-cmp-12.ipscape.com.au
On 4/14/16 10:39 PM, Nezar Madbouh wrote:
> Hi Ilya,
>
> Here is the output of mount command:
>
> [root@eqx-cs-cmp-01 ~]# mount
> /dev/mapper/vg_root-lv_root on / type ext4 (rw)
> proc on /proc type proc (rw)
> sysfs on /sys type sysfs (rw)
> devpts on /dev/pts type devpts (rw,gid=5,mode=620)
> tmpfs on /dev/shm type tmpfs (rw)
> /dev/sda1 on /boot type ext4 (rw)
> none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
> sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
> nfsd on /proc/fs/nfsd type nfsd (rw)
> 10.61.154.10:/Cloudstack/primary on
> /mnt/b3647480-5709-36eb-9ddd-e2c666f30b57 type nfs
> (rw,noatime,nfsvers=3,proto=tcp,hard,nolock,intr,addr=10.61.154.10)
> [root@eqx-cs-cmp-01 ~]#
>
>
> On 15 April 2016 at 15:22, Nezar Madbouh <ne...@ipscape.com.au>
> wrote:
>
>> Hi Ilya,
>>
>> Thanks for the instructions. I followed them and pasted the information on
>> pastebin:
>>
>> management-server logs job-14938
>> http://pastebin.com/GAiy8u5H
>>
>> Agent logs
>> http://pastebin.com/mfiADsJc
>>
>>
>> Your help is much appreciated.
>>
>> Nezar.
>>
>> On 15 April 2016 at 13:58, ilya <il...@gmail.com> wrote:
>>
>>> Here is how i would troubleshoot this:
>>>
>>> 1) ssh to MS host, run
>>>
>>> tail -f /var/cloudstack/management/management-server.log | grep -i
>>> StartVMCmdByAdmin
>>>
>>> 2) on cloudstack UI, press start on the VM and watch the tail log
>>>
>>> 3) on the tail log, you should see a StartVirtualMachine api call show
>>> up with job ID. For example:
>>>
>>>> 2016-04-15 03:41:57,193 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> (API-Job-Executor-24:ctx-7892387a job-291575) Executing AsyncJobVO
>>> {id:291575, userId: 157, accountId: 2, instanceType: VirtualMachine,
>>> instanceId: 7648, cmd:
>>> org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin, cmdInfo:
>>> {"id":"7648","response":"json","sessionkey":"mMJsduS9KLKApqH4dPpIlGZdt1Y\u003d","ctxDetails":"{\"com.cloud.vm.VirtualMachine\":7648,\"com.cloud.uservm.UserVm\":7648,\"com.cloud.network.router.VirtualRouter\":7648}","cmdEventType":"VM.START","expires":"2016-04-15T03:51:57+0000","ctxUserId":"157","signatureversion":"3","httpmethod":"GET","uuid":"7648","ctxAccountId":"2","ctxStartEventId":"314416"},
>>> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
>>> result: null, initMsid: 345051337894, completeMsid: null, lastUpdated:
>>> null, lastPolled: null, created: null}
>>>
>>> Each task in cloudstack spins up its own job, in this case its
>>> *job-291575*
>>>
>>> 4) now execute
>>>> grep -A3 -B3 job-291575
>>> /var/cloudstack/management/management-server.log | tee job-291575.log
>>>> grep job-291575 /var/cloudstack/management/management-server.log | grep
>>> via | head -1
>>>
>>> you should see a response like so,
>>>
>>>> 2016-04-15 03:41:59,259 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>> (Work-Job-Executor-42:ctx-b2c4af6b job-291575/job-291576 ctx-0988152e) Seq
>>> 441-4181310779036795138: Forwarding Seq 441-4181310779036795138: { Cmd ,
>>> MgmtId: 345051337894, via: 441(my-kvm-host01.example.com), Ver: v1,
>>> Flags: 100011,
>>> [{"com.cloud.agent.api.routing.DhcpEntryCommand":{"vmMac":"06:fe:4e:00:12:ad","vmIpAddress":"10.10.27.178","vmName":"ol6-test-ilya1","defaultRouter":"10.10.27.1","defaultDns":"10.17.27.166","duid":"00:03:00:01:06:fe:4e:00:12:ad","isDefault":true,"executeInSequence":false,"accessDetails":{"zone.network.type":"Advanced","router.guest.ip":"10.10.27.166","router.ip":"169.254.0.238","
>>> router.name":"r-7301-VM"},"wait":0}}] } to 345051337893
>>>
>>> We are looking for this line "via: 441(my-kvm-host01.example.com)"
>>>
>>> SSH to my-kvm-host01.example.com, and copy the
>>> /var/log/cloudstack/agent.log to pastebin
>>>
>>> also copy job-291575.log to pastebin
>>>
>>> I'll try to have a look, though when i get chance.
>>>
>>>
>>> Also paste the output of mount command on kvm host in question. I'm
>>> suspecting one of your storage pools maybe bad.
>>>
>>> Lastly, what distro are you running for KVM?
>>>
>>> Regards
>>> ilya
>>>
>>> On 4/14/16 5:05 PM, Nezar Madbouh wrote:
>>>> Hi Ilya,
>>>>
>>>> I shutdown all the VMs in Pre-prod Environmnet expecting the Pre-prod
>>>> Virtual router to be deleted automatically because no VM is running on
>>> this
>>>> vlan but it wasn't deleted.
>>>>
>>>> Am I right to expect the VR to be deleted automatically if no VM is
>>> running
>>>> in that vlan?
>>>>
>>>> Thanks,
>>>> Nezar.
>>>>
>>>> On 15 April 2016 at 09:53, Nezar Madbouh <ne...@ipscape.com.au>
>>>> wrote:
>>>>
>>>>> Hi Ilya,
>>>>>
>>>>> I have run the script and it didn't return any results:
>>>>>
>>>>> [root@eqx-cs-mgt-01 ~]# ./async_jobs.sh
>>>>> [root@eqx-cs-mgt-01 ~]#
>>>>>
>>>>>
>>>>> I also ran the command inside mysql:
>>>>>
>>>>> mysql> SELECT id, user_id, account_id, instance_type, instance_id,
>>>>> job_cmd, job_cmd_info, job_dispatcher, job_pending_signals FROM
>>>>> cloud.async_job where job_status=0 and job_dispatcher not like
>>>>> "pseudoJobDispatcher";
>>>>> Empty set (0.00 sec)
>>>>>
>>>>> your help is much appreciated,
>>>>> Nezar.
>>>>>
>>>>>
>>>>> On 14 April 2016 at 16:49, ilya <il...@gmail.com> wrote:
>>>>>
>>>>>> It appears you have pending jobs that are not letting you go through.
>>>>>>
>>>>>> Usually stop and starting management service should be enough.
>>>>>>
>>>>>> However, try running this script from your management server to show
>>> us
>>>>>> what you have running in async_job table.
>>>>>>
>>>>>>
>>>>>>
>>>>>>> #!/bin/bash
>>>>>>> DATESTAMP=$(date +%m%d%y-%H%M%S)
>>>>>>> DBPASS=$(java -classpath
>>>>>> /usr/share/cloudstack-common/lib/jasypt-1.9.0.jar
>>>>>> org.jasypt.intf.cli.JasyptPBEStringDecryptionCLI input="$(cat
>>>>>> /etc/cloudstack/management/db.properties | grep db.cloud.password |
>>> awk
>>>>>> -F'(' '{print $2}' | sed 's/)//g')" password="$(cat
>>>>>> /etc/cloudstack/management/key)" | grep -A2 OUTPUT | tail -1)
>>>>>>> DBHOST=$(cat /etc/cloudstack/management/db.properties | grep
>>>>>> db.cloud.host | awk -F'=' '{print $2}' | tail -1 )
>>>>>>> DBUSER=$(cat /etc/cloudstack/management/db.properties | grep
>>>>>> db.cloud.username | awk -F'=' '{print $2}')
>>>>>>> DB=$(cat /etc/cloudstack/management/db.properties | grep
>>> db.cloud.name
>>>>>> | awk -F'=' '{print $2}')
>>>>>>> DBPORT=$(cat /etc/cloudstack/management/db.properties | grep
>>>>>> db.cloud.port | awk -F'=' '{print $2}')
>>>>>>> MYSQLCMD="mysql -h $DBHOST -u $DBUSER -P $DBPORT -p$DBPASS $DB"
>>>>>>> echo 'SELECT id, user_id, account_id, instance_type, instance_id,
>>>>>> job_cmd, job_cmd_info, job_dispatcher, job_pending_signals FROM
>>>>>> cloud.async_job where job_status=0 and job_dispatcher not like
>>>>>> "pseudoJobDispatcher"' | $MYSQLCMD
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 4/13/16 5:16 PM, Nezar Madbouh wrote:
>>>>>>> Hi Simon,
>>>>>>>
>>>>>>> The following work was done but unfortunately the problem is not
>>> fixed:
>>>>>>>
>>>>>>>
>>>>>>> on management server:
>>>>>>>
>>>>>>> service cloudstack-management stop
>>>>>>>
>>>>>>>
>>>>>>> on the host:
>>>>>>>
>>>>>>> service cloudstack-agent stop
>>>>>>>
>>>>>>> service libvirtd restart
>>>>>>>
>>>>>>> service cloudstack-agent start
>>>>>>>
>>>>>>>
>>>>>>> on management server:
>>>>>>>
>>>>>>> service cloudstack-management start
>>>>>>>
>>>>>>> Regards,
>>>>>>> Nezar.
>>>>>>>
>>>>>>> On 24 March 2016 at 05:06, Simon Weller <sw...@ena.com> wrote:
>>>>>>>
>>>>>>>> Yes, I think that order is fine. Make sure the processes are indeed
>>>>>>>> shutdown before moving on to the next step.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ________________________________________
>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>>>>>>>> Sent: Tuesday, March 22, 2016 5:33 PM
>>>>>>>> To: users@cloudstack.apache.org
>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable to
>>> start
>>>>>>>> instance due to Unable to get answer that is of class
>>>>>>>> com.cloud.agent.api.StartAnswer
>>>>>>>>
>>>>>>>> Hi Simon,
>>>>>>>>
>>>>>>>> Can you please check if the following sequence of action is correct:
>>>>>>>>
>>>>>>>> on management server:
>>>>>>>>
>>>>>>>> service cloudstack-management stop
>>>>>>>>
>>>>>>>>
>>>>>>>> on the host:
>>>>>>>>
>>>>>>>> service cloudstack-agent stop
>>>>>>>>
>>>>>>>> service libvirtd restart
>>>>>>>>
>>>>>>>> service cloudstack-agent start
>>>>>>>>
>>>>>>>>
>>>>>>>> on management server:
>>>>>>>>
>>>>>>>> service cloudstack-management start
>>>>>>>>
>>>>>>>> Thanks very much for your help.
>>>>>>>> Nezar.
>>>>>>>>
>>>>>>>> On 23 March 2016 at 08:39, Simon Weller <sw...@ena.com> wrote:
>>>>>>>>
>>>>>>>>> Shutting down all the management services should clear the jobs.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ________________________________________
>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>>>>>>>>> Sent: Tuesday, March 22, 2016 4:28 PM
>>>>>>>>> To: users@cloudstack.apache.org
>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable to
>>>>>> start
>>>>>>>>> instance due to Unable to get answer that is of class
>>>>>>>>> com.cloud.agent.api.StartAnswer
>>>>>>>>>
>>>>>>>>> Hi Simon,
>>>>>>>>>
>>>>>>>>> I will plan to restart agents, libvirtd and management services.
>>>>>>>>>
>>>>>>>>> How do I clear hung jobs.
>>>>>>>>>
>>>>>>>>> Your help is much appreciated,
>>>>>>>>> Nezar.
>>>>>>>>>
>>>>>>>>> On 22 March 2016 at 23:03, Simon Weller <sw...@ena.com> wrote:
>>>>>>>>>
>>>>>>>>>> Nezar,
>>>>>>>>>>
>>>>>>>>>> I've looked through the code related to your release and it seems
>>>>>> that
>>>>>>>> in
>>>>>>>>>> order to throw this exception, 10 attempts have been made to
>>> retrieve
>>>>>>>> the
>>>>>>>>>> disk information prior to the exception being thrown.
>>>>>>>>>>
>>>>>>>>>> It appears there is something really odd going on here. I'm
>>> starting
>>>>>>>> to
>>>>>>>>>> wonder whether it might be worth while for you to stop all of your
>>>>>>>>> agents,
>>>>>>>>>> plus management server processes, restart libvirtd on each of the
>>>>>> hosts
>>>>>>>>> and
>>>>>>>>>> then bring the agents back up prior to starting the management
>>>>>> servers.
>>>>>>>>>>
>>>>>>>>>> There are a bunch of logs the pastebin you posted that seems to
>>>>>>>> indicate
>>>>>>>>>> you have some jobs hung, for example:
>>>>>>>>>>
>>>>>>>>>> 2016-03-14 21:45:31,828 WARN [o.a.c.f.j.i.AsyncJobMonitor]
>>>>>>>>>> (Timer-1:ctx-0c090bd7) Task (job-9066) has been pending for
>>> 1660851
>>>>>>>>> seconds
>>>>>>>>>> 2016-03-14 21:45:31,828 WARN [o.a.c.f.j.i.AsyncJobMonitor]
>>>>>>>>>> (Timer-1:ctx-0c090bd7) Task (job-8133) has been pending for
>>> 4253003
>>>>>>>>> seconds
>>>>>>>>>>
>>>>>>>>>> Restarting the agents, libvirtd and management will not cause an
>>>>>>>> outage
>>>>>>>>>> to your cloud, but I would suggest, if you decided to do this, you
>>>>>>>> should
>>>>>>>>>> complete it in a maintenance window.
>>>>>>>>>>
>>>>>>>>>> - Si
>>>>>>>>>>
>>>>>>>>>> ________________________________________
>>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>>>>>>>>>> Sent: Sunday, March 20, 2016 10:27 PM
>>>>>>>>>> To: users@cloudstack.apache.org
>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable to
>>>>>> start
>>>>>>>>>> instance due to Unable to get answer that is of class
>>>>>>>>>> com.cloud.agent.api.StartAnswer
>>>>>>>>>>
>>>>>>>>>> Hi Simon,
>>>>>>>>>>
>>>>>>>>>> Please see answers to your questions below. I am not sure if I
>>>>>> provided
>>>>>>>>> the
>>>>>>>>>> correct answer for the primary and secondary storage
>>> configuration.
>>>>>>>>> Please
>>>>>>>>>> let me know if this is not what you are after:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Can you screenshot and post somewhere your secondary storage
>>>>>>>>> configuration
>>>>>>>>>> and your primary storage configuration from the UI with the
>>> details?
>>>>>>>>>>
>>>>>>>>>> mysql> select * from configuration where name like '%secondary%'\G
>>>>>>>>>> *************************** 1. row ***************************
>>>>>>>>>> category: Account Defaults
>>>>>>>>>> instance: DEFAULT
>>>>>>>>>> component: management-server
>>>>>>>>>> name: max.account.secondary.storage
>>>>>>>>>> value: 400
>>>>>>>>>> description: The default maximum secondary storage space (in
>>> GiB)
>>>>>>>> that
>>>>>>>>>> can be used for an account
>>>>>>>>>> default_value: 400
>>>>>>>>>> updated: NULL
>>>>>>>>>> scope: NULL
>>>>>>>>>> is_dynamic: 0
>>>>>>>>>> *************************** 2. row ***************************
>>>>>>>>>> category: Project Defaults
>>>>>>>>>> instance: DEFAULT
>>>>>>>>>> component: management-server
>>>>>>>>>> name: max.project.secondary.storage
>>>>>>>>>> value: 400
>>>>>>>>>> description: The default maximum secondary storage space (in
>>> GiB)
>>>>>>>> that
>>>>>>>>>> can be used for a project
>>>>>>>>>> default_value: 400
>>>>>>>>>> updated: NULL
>>>>>>>>>> scope: NULL
>>>>>>>>>> is_dynamic: 0
>>>>>>>>>> *************************** 3. row ***************************
>>>>>>>>>> category: Hidden
>>>>>>>>>> instance: DEFAULT
>>>>>>>>>> component: management-server
>>>>>>>>>> name: secondary.storage.vm
>>>>>>>>>> value: +rwRhJOXPymbQOuAMLnhiw==
>>>>>>>>>> description: Deploys a VM per zone to manage secondary storage
>>> if
>>>>>>>> true,
>>>>>>>>>> otherwise secondary storage is mounted on management server
>>>>>>>>>> default_value: NULL
>>>>>>>>>> updated: NULL
>>>>>>>>>> scope: NULL
>>>>>>>>>> is_dynamic: 0
>>>>>>>>>> *************************** 4. row ***************************
>>>>>>>>>> category: Advanced
>>>>>>>>>> instance: DEFAULT
>>>>>>>>>> component: ExtensionRegistry
>>>>>>>>>> name: secondary.storage.vm.allocators.exclude
>>>>>>>>>> value: NULL
>>>>>>>>>> description: Extensions to exclude from being registered
>>>>>>>>>> default_value: NULL
>>>>>>>>>> updated: 2014-12-04 00:18:15
>>>>>>>>>> scope: Global
>>>>>>>>>> is_dynamic: 0
>>>>>>>>>> *************************** 5. row ***************************
>>>>>>>>>> category: NetworkManager
>>>>>>>>>> instance: DEFAULT
>>>>>>>>>> component: management-server
>>>>>>>>>> name: vm.network.nic.max.secondary.ipaddresses
>>>>>>>>>> value: NULL
>>>>>>>>>> description: Specify the number of secondary ip addresses per
>>> nic
>>>>>> per
>>>>>>>>> vm
>>>>>>>>>> default_value: 256
>>>>>>>>>> updated: NULL
>>>>>>>>>> scope: NULL
>>>>>>>>>> is_dynamic: 0
>>>>>>>>>> 5 rows in set (0.00 sec)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> mysql> select * from configuration where name like '%primary%'\G
>>>>>>>>>> *************************** 1. row ***************************
>>>>>>>>>> category: Account Defaults
>>>>>>>>>> instance: DEFAULT
>>>>>>>>>> component: management-server
>>>>>>>>>> name: max.account.primary.storage
>>>>>>>>>> value: 200
>>>>>>>>>> description: The default maximum primary storage space (in GiB)
>>>>>> that
>>>>>>>>> can
>>>>>>>>>> be used for an account
>>>>>>>>>> default_value: 200
>>>>>>>>>> updated: NULL
>>>>>>>>>> scope: NULL
>>>>>>>>>> is_dynamic: 0
>>>>>>>>>> *************************** 2. row ***************************
>>>>>>>>>> category: Project Defaults
>>>>>>>>>> instance: DEFAULT
>>>>>>>>>> component: management-server
>>>>>>>>>> name: max.project.primary.storage
>>>>>>>>>> value: 200
>>>>>>>>>> description: The default maximum primary storage space (in GiB)
>>>>>> that
>>>>>>>>> can
>>>>>>>>>> be used for a project
>>>>>>>>>> default_value: 200
>>>>>>>>>> updated: NULL
>>>>>>>>>> scope: NULL
>>>>>>>>>> is_dynamic: 0
>>>>>>>>>> *************************** 3. row ***************************
>>>>>>>>>> category: Storage
>>>>>>>>>> instance: DEFAULT
>>>>>>>>>> component: TemplateManager
>>>>>>>>>> name: primary.storage.download.wait
>>>>>>>>>> value: 10800
>>>>>>>>>> description: In second, timeout for download template to primary
>>>>>>>>> storage
>>>>>>>>>> default_value: NULL
>>>>>>>>>> updated: NULL
>>>>>>>>>> scope: NULL
>>>>>>>>>> is_dynamic: 0
>>>>>>>>>> 3 rows in set (0.00 sec)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Also, what is the Cloudstack version you are running and what is
>>> the
>>>>>> OS
>>>>>>>>>> you're running it on?
>>>>>>>>>>
>>>>>>>>>> Cloudstack 4.4.1
>>>>>>>>>>
>>>>>>>>>> Centos 6.6
>>>>>>>>>>
>>>>>>>>>> Are you using actimeo=0 in your NFS mount line?
>>>>>>>>>>
>>>>>>>>>> Primary storage on Hypervisors has the following parameters
>>>>>>>>>> 10.61.154.10:/Cloudstack/primary on
>>>>>>>>>> /mnt/b3647480-5709-36eb-9ddd-e2c666f30b57 type nfs
>>>>>>>>>>
>>> (rw,noatime,nfsvers=3,proto=tcp,hard,nolock,intr,addr=10.61.154.10)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Secondary storage on SSVM has the following parameters
>>>>>>>>>> 10.61.154.10:/Cloudstack/secondary on
>>>>>>>>>> /mnt/SecStorage/d11d5c92-6950-30e8-a0b5-3cd46a17ec14 type nfs4
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>> (rw,relatime,vers=4,rsize=262144,wsize=262144,namlen=255,acdirmin=0,acdirmax=0,soft,proto=tcp,port=0,timeo=133,retrans=
>>>>>> 2147483647
>>>>>>
>>> ,sec=sys,clientaddr=10.61.154.70,minorversion=0,local_lock=none,addr=10.61.154.10)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thanks & Regards,
>>>>>>>>>> Nezar.
>>>>>>>>>>
>>>>>>>>>> On 18 March 2016 at 23:50, Simon Weller <sw...@ena.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Also, what is the Cloudstack version you are running and what is
>>> the
>>>>>>>> OS
>>>>>>>>>>> you're running it on?
>>>>>>>>>>>
>>>>>>>>>>> Are you using actimeo=0 in your NFS mount line?
>>>>>>>>>>>
>>>>>>>>>>> ________________________________________
>>>>>>>>>>> From: Simon Weller <sw...@ena.com>
>>>>>>>>>>> Sent: Friday, March 18, 2016 7:40 AM
>>>>>>>>>>> To: users@cloudstack.apache.org
>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable to
>>>>>>>> start
>>>>>>>>>>> instance due to Unable to get answer that is of class
>>>>>>>>>>> com.cloud.agent.api.StartAnswer
>>>>>>>>>>>
>>>>>>>>>>> Nexar,
>>>>>>>>>>>
>>>>>>>>>>> That's normal behaviour. You can only ssh to the link local on
>>> the
>>>>>>>> host
>>>>>>>>>>> where the VM resides.
>>>>>>>>>>>
>>>>>>>>>>> Can you screenshot and post somewhere your secondary storage
>>>>>>>>>> configuration
>>>>>>>>>>> and your primary storage configuration from the UI with the
>>> details?
>>>>>>>>>>>
>>>>>>>>>>> - Si
>>>>>>>>>>>
>>>>>>>>>>> ________________________________________
>>>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>>>>>>>>>>> Sent: Thursday, March 17, 2016 8:19 PM
>>>>>>>>>>> To: users@cloudstack.apache.org
>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable to
>>>>>>>> start
>>>>>>>>>>> instance due to Unable to get answer that is of class
>>>>>>>>>>> com.cloud.agent.api.StartAnswer
>>>>>>>>>>>
>>>>>>>>>>> Hi Simon,
>>>>>>>>>>>
>>>>>>>>>>> I just noticed that from other Hosts I can't ssh into the SSVM:
>>>>>>>>>>>
>>>>>>>>>>> [root@eqx-cs-cmp-01 ~]# ssh -i /root/.ssh/id_rsa.cloud -p 3922
>>>>>>>>>>> root@169.254.3.161
>>>>>>>>>>> ssh: connect to host 169.254.3.161 port 3922: No route to host
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I can only ssh from one Host:
>>>>>>>>>>> [root@eqx-cs-cmp-06 ~]# ssh -i /root/.ssh/id_rsa.cloud -p 3922
>>>>>>>>>>> root@169.254.3.161
>>>>>>>>>>> Linux s-11-VM 3.2.0-4-amd64 #1 SMP Debian 3.2.57-3 x86_64
>>>>>>>>>>>
>>>>>>>>>>> Thanks & Regards,
>>>>>>>>>>> Nezar.
>>>>>>>>>>>
>>>>>>>>>>> On 18 March 2016 at 10:36, Nezar Madbouh <
>>>>>>>> nezar.madbouh@ipscape.com.au
>>>>>>>>>>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Simon,
>>>>>>>>>>>>
>>>>>>>>>>>> Yes existing VMs are successfully running. Yesterday I was able
>>> to
>>>>>>>>>> create
>>>>>>>>>>>> a VM in Dev Environment but I can't create VMs in Prod
>>> environment.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Nezar.
>>>>>>>>>>>>
>>>>>>>>>>>> On 18 March 2016 at 09:12, Simon Weller <sw...@ena.com>
>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> I'm assuming you have other VMs successfully running, correct?
>>> So
>>>>>>>>> you
>>>>>>>>>>> can
>>>>>>>>>>>>> write to your primary storage.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> ________________________________________
>>>>>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>>>>>>>>>>>>> Sent: Thursday, March 17, 2016 4:43 PM
>>>>>>>>>>>>> To: users@cloudstack.apache.org
>>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable
>>> to
>>>>>>>>>> start
>>>>>>>>>>>>> instance due to Unable to get answer that is of class
>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Simon and Shweta,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Please see the result below:
>>>>>>>>>>>>>
>>>>>>>>>>>>> root@s-11-VM:~# /usr/local/cloud/systemvm/ssvm-check.sh
>>>>>>>>>>>>> ================================================
>>>>>>>>>>>>> First DNS server is 8.8.8.8
>>>>>>>>>>>>> PING 8.8.8.8 (8.8.8.8): 48 data bytes
>>>>>>>>>>>>> 56 bytes from 8.8.8.8: icmp_seq=0 ttl=57 time=5.209 ms
>>>>>>>>>>>>> 56 bytes from 8.8.8.8: icmp_seq=1 ttl=57 time=0.936 ms
>>>>>>>>>>>>> --- 8.8.8.8 ping statistics ---
>>>>>>>>>>>>> 2 packets transmitted, 2 packets received, 0% packet loss
>>>>>>>>>>>>> round-trip min/avg/max/stddev = 0.936/3.072/5.209/2.137 ms
>>>>>>>>>>>>> Good: Can ping DNS server
>>>>>>>>>>>>> ================================================
>>>>>>>>>>>>> Good: DNS resolves download.cloud.com
>>>>>>>>>>>>> ================================================
>>>>>>>>>>>>> nfs is currently mounted
>>>>>>>>>>>>> Mount point is
>>>>>>>> /mnt/SecStorage/d11d5c92-6950-30e8-a0b5-3cd46a17ec14
>>>>>>>>>>>>> Good: Can write to mount point
>>>>>>>>>>>>> ================================================
>>>>>>>>>>>>> Management server is 10.61.153.20. Checking connectivity.
>>>>>>>>>>>>> Good: Can connect to management server port 8250
>>>>>>>>>>>>> ================================================
>>>>>>>>>>>>> Good: Java process is running
>>>>>>>>>>>>> ================================================
>>>>>>>>>>>>> Tests Complete. Look for ERROR or WARNING above.
>>>>>>>>>>>>> root@s-11-VM:~#
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks & Regards,
>>>>>>>>>>>>> Nezar.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 17 March 2016 at 09:42, Simon Weller <sw...@ena.com>
>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Nezar,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Log into the Cloudstack UI.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Click on "Infrastructure" on the left menu. Click on View All
>>>>>>>>> below
>>>>>>>>>>>>>> "System VMs". find the item marked "Secondary Storage VM" and
>>>>>>>>> then
>>>>>>>>>>>>> click
>>>>>>>>>>>>>> on the Name of that VM on the left (it will be s-XXXX-VM).
>>>>>>>> You'll
>>>>>>>>>> see
>>>>>>>>>>> a
>>>>>>>>>>>>>> bunch of information about the VM. Find the Host line and also
>>>>>>>> the
>>>>>>>>>>> Link
>>>>>>>>>>>>>> Local IP Address line and then follow the original
>>> instructions
>>>>>>>> in
>>>>>>>>>> my
>>>>>>>>>>>>>> previous email.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> - Si
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ________________________________________
>>>>>>>>>>>>>> From: Shweta Agarwal <sh...@accelerite.com>
>>>>>>>>>>>>>> Sent: Wednesday, March 16, 2016 12:16 AM
>>>>>>>>>>>>>> To: users@cloudstack.apache.org
>>>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable
>>>>>>>> to
>>>>>>>>>>> start
>>>>>>>>>>>>>> instance due to Unable to get answer that is of class
>>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Nezar,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> You can find host name on detail view of SSVM in UI .
>>>>>>>>>>>>>> You can also use list api command listSystemVms to get the
>>> host
>>>>>>>>>>> details
>>>>>>>>>>>>> of
>>>>>>>>>>>>>> system vms.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>> Shweta
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ________________________________________
>>>>>>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>>>>>>>>>>>>>> Sent: Wednesday, March 16, 2016 9:58 AM
>>>>>>>>>>>>>> To: users@cloudstack.apache.org
>>>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable
>>>>>>>> to
>>>>>>>>>>> start
>>>>>>>>>>>>>> instance due to Unable to get answer that is of class
>>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Simon,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> How can I find out which host the SSVM is on?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks & Regards,
>>>>>>>>>>>>>> Nezar.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 16 March 2016 at 12:28, Simon Weller <sw...@ena.com>
>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Can you ssh into the Secondary Storage VM?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> This can be done by getting the link-local-ip address from
>>> the
>>>>>>>>>>>>>>> infrastructure menu in the UI and then sshing into the host
>>>>>>>> the
>>>>>>>>>> SSVM
>>>>>>>>>>>>> is
>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>> and issuing this command as root: ssh -i
>>>>>>>> /root/.ssh/id_rsa.cloud
>>>>>>>>>> -p
>>>>>>>>>>>>> 3922
>>>>>>>>>>>>>>> root@link-local-ip (replace link-local-ip with the link
>>>>>>>> local
>>>>>>>>>> ip).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Try running the SSVM diangnostic script:
>>>>>>>>>>>>>>> usr/local/cloud/systemvm/ssvm-check.sh
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> - Si
>>>>>>>>>>>>>>> ________________________________________
>>>>>>>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>>>>>>>>>>>>>>> Sent: Tuesday, March 15, 2016 7:49 PM
>>>>>>>>>>>>>>> To: users@cloudstack.apache.org
>>>>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message:
>>> Unable
>>>>>>>>> to
>>>>>>>>>>>>> start
>>>>>>>>>>>>>>> instance due to Unable to get answer that is of class
>>>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Simon,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Please see details below:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> mysql> use cloud; select id, name, status, type from host
>>>>>>>> where
>>>>>>>>>> id =
>>>>>>>>>>>>> 5;
>>>>>>>>>>>>>>> Reading table information for completion of table and column
>>>>>>>>> names
>>>>>>>>>>>>>>> You can turn off this feature to get a quicker startup with
>>> -A
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Database changed
>>>>>>>>>>>>>>> +----+------------------------------+--------+---------+
>>>>>>>>>>>>>>> | id | name | status | type |
>>>>>>>>>>>>>>> +----+------------------------------+--------+---------+
>>>>>>>>>>>>>>> | 5 | eqx-cs-cmp-02.ipscape.com.au | up | routing |
>>>>>>>>>>>>>>> +----+------------------------------+--------+---------+
>>>>>>>>>>>>>>> 1 row in set (0.00 sec)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> cat /var/log/cloudstack/agent/agent.log
>>>>>>>>>>>>>>> com.cloud.utils.exception.CloudRuntimeException:
>>>>>>>>>>>>>>> com.cloud.utils.exception.CloudRuntimeException: Can't find
>>>>>>>>>>>>>>> volume:089ff3ca-9bff-402c-a14e-ea7a35aaad5c
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>> com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.getPhysicalDisk(KVMStoragePoolManager.java:262)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.createVbd(LibvirtComputingResource.java:3921)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(LibvirtComputingResource.java:3789)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1333)
>>>>>>>>>>>>>>> at
>>>>>>>> com.cloud.agent.Agent.processRequest(Agent.java:501)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>
>>> com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:808)
>>>>>>>>>>>>>>> at com.cloud.utils.nio.Task.run(Task.java:84)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>>>>>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>>>>>>>>>> agent.log.2016-03-03.gz (END)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [root@eqx-cs-cmp-02 agent]# virsh pool-list
>>>>>>>>>>>>>>> Name State Autostart
>>>>>>>>>>>>>>> -----------------------------------------
>>>>>>>>>>>>>>> 2d0967f8-50d6-4d6d-87b7-4da0596f6924 active no
>>>>>>>>>>>>>>> b3647480-5709-36eb-9ddd-e2c666f30b57 active no
>>>>>>>>>>>>>>> e3831246-ff55-4af4-9964-a17565edaccc active no
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [Root @ eqx-cs-cmp-02 agent] # df -h
>>>>>>>>>>>>>>> Filesystem Size Used Avail Use% Mounted on
>>>>>>>>>>>>>>> /dev/mapper/vg_root-lv_root
>>>>>>>>>>>>>>> 203G 215G 1.3G 1% /
>>>>>>>>>>>>>>> tmpfs 48g 48g 0 0% / dev / shm
>>>>>>>>>>>>>>> / Dev / sda1 477m 427m 25M 6% / boot
>>>>>>>>>>>>>>> xx.xx.xx.xx:/Cloudstack/primary
>>>>>>>>>>>>>>> 7.6T 4.1T 3.5T 54%
>>>>>>>>>>>>>>> /mnt/b3647480-5709-36eb-9ddd-e2c666f30b57
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I also noticed the following in cloudstack-agent.err
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> libvirt: QEMU Driver error : Domain not found: no domain with
>>>>>>>>>>> matching
>>>>>>>>>>>>>> name
>>>>>>>>>>>>>>> 'i-11-600-VM'
>>>>>>>>>>>>>>> libvirt: Storage Driver error : Storage volume not found: no
>>>>>>>>>> storage
>>>>>>>>>>>>> vol
>>>>>>>>>>>>>>> with matching name '089ff3ca-9bff-402c-a14e-ea7a35aaad5c'
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Your help is much appreciated,
>>>>>>>>>>>>>>> Nezar
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 16 March 2016 at 11:03, Simon Weller <sw...@ena.com>
>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Nezar,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> ok, I just took a look at a pastebin you posted.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I see it complaining about the inability to find a volume
>>>>>>>>> prior
>>>>>>>>>> to
>>>>>>>>>>>>>>>> throwing the host exception.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Firstly, let's find out who host 5 is. Can you log into your
>>>>>>>>>> MySQL
>>>>>>>>>>>>>> server
>>>>>>>>>>>>>>>> and run this:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> use cloud;select id,name,status,type from host where id=5;
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Can you post the results?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> If it's a host, can you post the
>>>>>>>>>>> /var/log/cloudstack/agent/agent.log
>>>>>>>>>>>>>> from
>>>>>>>>>>>>>>>> that host?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> And also run virsh pool-list on the host in question?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> - Si
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> ________________________________________
>>>>>>>>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>>>>>>>>>>>>>>>> Sent: Tuesday, March 15, 2016 6:11 PM
>>>>>>>>>>>>>>>> To: users@cloudstack.apache.org
>>>>>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message:
>>>>>>>> Unable
>>>>>>>>>> to
>>>>>>>>>>>>> start
>>>>>>>>>>>>>>>> instance due to Unable to get answer that is of class
>>>>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi Simon,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> When you say "double check your routers tied to the
>>>>>>>> network",
>>>>>>>>> do
>>>>>>>>>>> you
>>>>>>>>>>>>>> mean
>>>>>>>>>>>>>>>> the physical routers or the virtual routers?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> When I try to create a VM in Prod Environment it fails. I
>>>>>>>>> can't
>>>>>>>>>>> ping
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> Prod virtual router from management server or Host.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> But I can ping Dev virtual router from management server and
>>>>>>>>>> Host
>>>>>>>>>>>>> and I
>>>>>>>>>>>>>>> was
>>>>>>>>>>>>>>>> able to create a VM in Dev Environment.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> How can I troubleshoot the Prod Virtual router. I launched
>>>>>>>> the
>>>>>>>>>>>>> console
>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>> couldn't login.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Please note that we need to take into consideration that the
>>>>>>>>>>>>> affected
>>>>>>>>>>>>>>>> network is the Production network and we need to minimise
>>>>>>>>>> service
>>>>>>>>>>>>>> outage
>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>> much as possible.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Your help is much appreciated,
>>>>>>>>>>>>>>>> Nezar
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On 16 March 2016 at 08:37, Simon Weller <sw...@ena.com>
>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I've seen this before where a Virtual Router has crashed
>>>>>>>> and
>>>>>>>>>> the
>>>>>>>>>>>>>>>>> management server can't communicate with it.
>>>>>>>>>>>>>>>>> Are you using an advanced network setup?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> If so, double check your routers tied to the network where
>>>>>>>>> you
>>>>>>>>>>> are
>>>>>>>>>>>>>>>>> attempting to start an instance.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> - Si
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> ________________________________________
>>>>>>>>>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>>>>>>>>>>>>>>>>> Sent: Tuesday, March 15, 2016 4:02 PM
>>>>>>>>>>>>>>>>> To: users@cloudstack.apache.org
>>>>>>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message:
>>>>>>>>> Unable
>>>>>>>>>>> to
>>>>>>>>>>>>>> start
>>>>>>>>>>>>>>>>> instance due to Unable to get answer that is of class
>>>>>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi Glenn,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> We have 16 hosts, all KVM.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> The error is not specific to Host 5. Every time I try to
>>>>>>>>>> create
>>>>>>>>>>> a
>>>>>>>>>>>>> VM
>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>> gives me the same error message but on different host.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> How do I restart the agent on the host. Is it service
>>>>>>>>>> affecting?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks very much for your help.
>>>>>>>>>>>>>>>>> Neza
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On 15 March 2016 at 17:37, Glenn Wagner <
>>>>>>>>>>>>> glenn.wagner@shapeblue.com>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Can you explain your environment first , how many hosts
>>>>>>>> ,
>>>>>>>>>> what
>>>>>>>>>>>>> is
>>>>>>>>>>>>>>> your
>>>>>>>>>>>>>>>>>> hypervisors (KVM,Xenserver,VMware)
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> The error seems to be with your Host 5 Hypervisor
>>>>>>>>>>>>>>>>>> Resource [Host:5] is unreachable:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Go to the infrastructure section and see what host5 is
>>>>>>>>>>>>> reporting,
>>>>>>>>>>>>>>>>>> Have you tried to restart the agent on the Host?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>> Glenn
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> [image: ShapeBlue] <http://www.shapeblue.com>
>>>>>>>>>>>>>>>>>> Glenn Wagner
>>>>>>>>>>>>>>>>>> Senior Consultant , ShapeBlue
>>>>>>>>>>>>>>>>>> d: * | s: +27 21 527 0091*
>>>>>>>>>>>>> <%7C%20s:%20+27%2021%20527%200091> |
>>>>>>>>>>>>>>> m:
>>>>>>>>>>>>>>>>>> *+27 73 917 4111* <+27%2073%20917%204111>
>>>>>>>>>>>>>>>>>> e: *glenn.wagner@shapeblue.com | t: *
>>>>>>>>>>>>>>>>>> <glenn.wagner@shapeblue.com%20%7C%20t:> | w: *
>>>>>>>>>>>>> www.shapeblue.com
>>>>>>>>>>>>>> *
>>>>>>>>>>>>>>>>>> <http://www.shapeblue.com>
>>>>>>>>>>>>>>>>>> a:
>>>>>>>>>>>>>>>>>> 2nd Floor, Oudehuis Centre, 122 Main Rd, Somerset West
>>>>>>>>> Cape
>>>>>>>>>>> Town
>>>>>>>>>>>>>> 7130
>>>>>>>>>>>>>>>>> South Africa
>>>>>>>>>>>>>>>>>> Shape Blue Ltd is a company incorporated in England &
>>>>>>>>> Wales.
>>>>>>>>>>>>>>> ShapeBlue
>>>>>>>>>>>>>>>>>> Services India LLP is a company incorporated in India
>>>>>>>> and
>>>>>>>>> is
>>>>>>>>>>>>>> operated
>>>>>>>>>>>>>>>>> under
>>>>>>>>>>>>>>>>>> license from Shape Blue Ltd. Shape Blue Brasil
>>>>>>>> Consultoria
>>>>>>>>>>> Ltda
>>>>>>>>>>>>> is
>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>> company incorporated in Brasil and is operated under
>>>>>>>>> license
>>>>>>>>>>>>> from
>>>>>>>>>>>>>>> Shape
>>>>>>>>>>>>>>>>>> Blue Ltd. ShapeBlue SA Pty Ltd is a company registered
>>>>>>>> by
>>>>>>>>>> The
>>>>>>>>>>>>>>> Republic
>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>> South Africa and is traded under license from Shape Blue
>>>>>>>>>> Ltd.
>>>>>>>>>>>>>>> ShapeBlue
>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>> a registered trademark.
>>>>>>>>>>>>>>>>>> This email and any attachments to it may be confidential
>>>>>>>>> and
>>>>>>>>>>> are
>>>>>>>>>>>>>>>> intended
>>>>>>>>>>>>>>>>>> solely for the use of the individual to whom it is
>>>>>>>>>> addressed.
>>>>>>>>>>>>> Any
>>>>>>>>>>>>>>> views
>>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>> opinions expressed are solely those of the author and do
>>>>>>>>> not
>>>>>>>>>>>>>>>> necessarily
>>>>>>>>>>>>>>>>>> represent those of Shape Blue Ltd or related companies.
>>>>>>>> If
>>>>>>>>>> you
>>>>>>>>>>>>> are
>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> intended recipient of this email, you must neither take
>>>>>>>>> any
>>>>>>>>>>>>> action
>>>>>>>>>>>>>>>> based
>>>>>>>>>>>>>>>>>> upon its contents, nor copy or show it to anyone. Please
>>>>>>>>>>> contact
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> sender
>>>>>>>>>>>>>>>>>> if you believe you have received this email in error.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>>>>>>>>> From: Nezar Madbouh [mailto:
>>>>>>>> nezar.madbouh@ipscape.com.au]
>>>>>>>>>>>>>>>>>> Sent: Tuesday, 15 March 2016 5:45 AM
>>>>>>>>>>>>>>>>>> To: users@cloudstack.apache.org
>>>>>>>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message:
>>>>>>>>>> Unable
>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> start
>>>>>>>>>>>>>>>>>> instance due to Unable to get answer that is of class
>>>>>>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi Jeremy,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I have uploaded the contents to
>>>>>>>>>> http://pastebin.com/1J3mmgTh
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> your help is much appreciated,
>>>>>>>>>>>>>>>>>> Nezar.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On 15 March 2016 at 13:46, Jeremy Peterson <
>>>>>>>>>>>>> jpeterson@acentek.net>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I don't see any attachments can you create a pastebin.
>>>>>>>>>>> That's
>>>>>>>>>>>>>>> easier
>>>>>>>>>>>>>>>>>>> to look through as I think attachments are removed
>>>>>>>>> before
>>>>>>>>>>>>> being
>>>>>>>>>>>>>>> sent
>>>>>>>>>>>>>>>>>>> to user group.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Jeremy
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> -------- Original message --------
>>>>>>>>>>>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>>>>>>>>>>>>>>>>>>> Date: 3/14/2016 6:38 PM (GMT-06:00)
>>>>>>>>>>>>>>>>>>> To: users@cloudstack.apache.org
>>>>>>>>>>>>>>>>>>> Subject: Unable to create a New VM - Error message:
>>>>>>>>> Unable
>>>>>>>>>>> to
>>>>>>>>>>>>>> start
>>>>>>>>>>>>>>>>>>> instance due to Unable to get answer that is of class
>>>>>>>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi Shweta and Sadhu,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Please find attached managerment-server.log which
>>>>>>>>> contains
>>>>>>>>>>> the
>>>>>>>>>>>>>> logs
>>>>>>>>>>>>>>>>>>> since I started creating a new VM (VM id 600).
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Please note that we don't have agent.log files
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Your help is much appreciated.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>> Nezar.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Find out more about ShapeBlue and our range of
>>>>>>>> CloudStack
>>>>>>>>>>>>> related
>>>>>>>>>>>>>>>>> services:
>>>>>>>>>>>>>>>>>> IaaS Cloud Design & Build
>>>>>>>>>>>>>>>>>> <http://shapeblue.com/iaas-cloud-design-and-build//> |
>>>>>>>>>>> CSForge
>>>>>>>>>>>>> -
>>>>>>>>>>>>>>> rapid
>>>>>>>>>>>>>>>>>> IaaS deployment framework <
>>>>>>>> http://shapeblue.com/csforge/>
>>>>>>>>>>>>>>>>>> CloudStack Consulting <
>>>>>>>>>>>>>> http://shapeblue.com/cloudstack-consultancy/>
>>>>>>>>>>>>>>> |
>>>>>>>>>>>>>>>>> CloudStack
>>>>>>>>>>>>>>>>>> Software Engineering
>>>>>>>>>>>>>>>>>> <http://shapeblue.com/cloudstack-software-engineering/>
>>>>>>>>>>>>>>>>>> CloudStack Infrastructure Support
>>>>>>>>>>>>>>>>>> <
>>>>>>>> http://shapeblue.com/cloudstack-infrastructure-support/>
>>>>>>>>> |
>>>>>>>>>>>>>>> CloudStack
>>>>>>>>>>>>>>>>>> Bootcamp Training Courses <
>>>>>>>>>>>>>> http://shapeblue.com/cloudstack-training/
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> DISCLAIMER
>>>>>>>>>>>>>> ==========
>>>>>>>>>>>>>> This e-mail may contain privileged and confidential
>>> information
>>>>>>>>>> which
>>>>>>>>>>> is
>>>>>>>>>>>>>> the property of Accelerite, a Persistent Systems business. It
>>> is
>>>>>>>>>>>>> intended
>>>>>>>>>>>>>> only for the use of the individual or entity to which it is
>>>>>>>>>> addressed.
>>>>>>>>>>>>> If
>>>>>>>>>>>>>> you are not the intended recipient, you are not authorized to
>>>>>>>>> read,
>>>>>>>>>>>>> retain,
>>>>>>>>>>>>>> copy, print, distribute or use this message. If you have
>>>>>>>> received
>>>>>>>>>> this
>>>>>>>>>>>>>> communication in error, please notify the sender and delete
>>> all
>>>>>>>>>> copies
>>>>>>>>>>>>> of
>>>>>>>>>>>>>> this message. Accelerite, a Persistent Systems business does
>>> not
>>>>>>>>>>> accept
>>>>>>>>>>>>> any
>>>>>>>>>>>>>> liability for virus infected mails.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>
>
Re: Unable to create a New VM - Error message: Unable to start instance
due to Unable to get answer that is of class com.cloud.agent.api.StartAnswer
Posted by ilya <il...@gmail.com>.
Can you goto primary storage in cloudstack UI, is this what you have
defined for your NFS Server?
"SharedMountPoint","host":"localhost","path":"/mnt/storage"
If so, please use IP or FQDN instead..
On 4/14/16 10:39 PM, Nezar Madbouh wrote:
> Hi Ilya,
>
> Here is the output of mount command:
>
> [root@eqx-cs-cmp-01 ~]# mount
> /dev/mapper/vg_root-lv_root on / type ext4 (rw)
> proc on /proc type proc (rw)
> sysfs on /sys type sysfs (rw)
> devpts on /dev/pts type devpts (rw,gid=5,mode=620)
> tmpfs on /dev/shm type tmpfs (rw)
> /dev/sda1 on /boot type ext4 (rw)
> none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
> sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
> nfsd on /proc/fs/nfsd type nfsd (rw)
> 10.61.154.10:/Cloudstack/primary on
> /mnt/b3647480-5709-36eb-9ddd-e2c666f30b57 type nfs
> (rw,noatime,nfsvers=3,proto=tcp,hard,nolock,intr,addr=10.61.154.10)
> [root@eqx-cs-cmp-01 ~]#
>
>
> On 15 April 2016 at 15:22, Nezar Madbouh <ne...@ipscape.com.au>
> wrote:
>
>> Hi Ilya,
>>
>> Thanks for the instructions. I followed them and pasted the information on
>> pastebin:
>>
>> management-server logs job-14938
>> http://pastebin.com/GAiy8u5H
>>
>> Agent logs
>> http://pastebin.com/mfiADsJc
>>
>>
>> Your help is much appreciated.
>>
>> Nezar.
>>
>> On 15 April 2016 at 13:58, ilya <il...@gmail.com> wrote:
>>
>>> Here is how i would troubleshoot this:
>>>
>>> 1) ssh to MS host, run
>>>
>>> tail -f /var/cloudstack/management/management-server.log | grep -i
>>> StartVMCmdByAdmin
>>>
>>> 2) on cloudstack UI, press start on the VM and watch the tail log
>>>
>>> 3) on the tail log, you should see a StartVirtualMachine api call show
>>> up with job ID. For example:
>>>
>>>> 2016-04-15 03:41:57,193 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> (API-Job-Executor-24:ctx-7892387a job-291575) Executing AsyncJobVO
>>> {id:291575, userId: 157, accountId: 2, instanceType: VirtualMachine,
>>> instanceId: 7648, cmd:
>>> org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin, cmdInfo:
>>> {"id":"7648","response":"json","sessionkey":"mMJsduS9KLKApqH4dPpIlGZdt1Y\u003d","ctxDetails":"{\"com.cloud.vm.VirtualMachine\":7648,\"com.cloud.uservm.UserVm\":7648,\"com.cloud.network.router.VirtualRouter\":7648}","cmdEventType":"VM.START","expires":"2016-04-15T03:51:57+0000","ctxUserId":"157","signatureversion":"3","httpmethod":"GET","uuid":"7648","ctxAccountId":"2","ctxStartEventId":"314416"},
>>> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
>>> result: null, initMsid: 345051337894, completeMsid: null, lastUpdated:
>>> null, lastPolled: null, created: null}
>>>
>>> Each task in cloudstack spins up its own job, in this case its
>>> *job-291575*
>>>
>>> 4) now execute
>>>> grep -A3 -B3 job-291575
>>> /var/cloudstack/management/management-server.log | tee job-291575.log
>>>> grep job-291575 /var/cloudstack/management/management-server.log | grep
>>> via | head -1
>>>
>>> you should see a response like so,
>>>
>>>> 2016-04-15 03:41:59,259 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>> (Work-Job-Executor-42:ctx-b2c4af6b job-291575/job-291576 ctx-0988152e) Seq
>>> 441-4181310779036795138: Forwarding Seq 441-4181310779036795138: { Cmd ,
>>> MgmtId: 345051337894, via: 441(my-kvm-host01.example.com), Ver: v1,
>>> Flags: 100011,
>>> [{"com.cloud.agent.api.routing.DhcpEntryCommand":{"vmMac":"06:fe:4e:00:12:ad","vmIpAddress":"10.10.27.178","vmName":"ol6-test-ilya1","defaultRouter":"10.10.27.1","defaultDns":"10.17.27.166","duid":"00:03:00:01:06:fe:4e:00:12:ad","isDefault":true,"executeInSequence":false,"accessDetails":{"zone.network.type":"Advanced","router.guest.ip":"10.10.27.166","router.ip":"169.254.0.238","
>>> router.name":"r-7301-VM"},"wait":0}}] } to 345051337893
>>>
>>> We are looking for this line "via: 441(my-kvm-host01.example.com)"
>>>
>>> SSH to my-kvm-host01.example.com, and copy the
>>> /var/log/cloudstack/agent.log to pastebin
>>>
>>> also copy job-291575.log to pastebin
>>>
>>> I'll try to have a look, though when i get chance.
>>>
>>>
>>> Also paste the output of mount command on kvm host in question. I'm
>>> suspecting one of your storage pools maybe bad.
>>>
>>> Lastly, what distro are you running for KVM?
>>>
>>> Regards
>>> ilya
>>>
>>> On 4/14/16 5:05 PM, Nezar Madbouh wrote:
>>>> Hi Ilya,
>>>>
>>>> I shutdown all the VMs in Pre-prod Environmnet expecting the Pre-prod
>>>> Virtual router to be deleted automatically because no VM is running on
>>> this
>>>> vlan but it wasn't deleted.
>>>>
>>>> Am I right to expect the VR to be deleted automatically if no VM is
>>> running
>>>> in that vlan?
>>>>
>>>> Thanks,
>>>> Nezar.
>>>>
>>>> On 15 April 2016 at 09:53, Nezar Madbouh <ne...@ipscape.com.au>
>>>> wrote:
>>>>
>>>>> Hi Ilya,
>>>>>
>>>>> I have run the script and it didn't return any results:
>>>>>
>>>>> [root@eqx-cs-mgt-01 ~]# ./async_jobs.sh
>>>>> [root@eqx-cs-mgt-01 ~]#
>>>>>
>>>>>
>>>>> I also ran the command inside mysql:
>>>>>
>>>>> mysql> SELECT id, user_id, account_id, instance_type, instance_id,
>>>>> job_cmd, job_cmd_info, job_dispatcher, job_pending_signals FROM
>>>>> cloud.async_job where job_status=0 and job_dispatcher not like
>>>>> "pseudoJobDispatcher";
>>>>> Empty set (0.00 sec)
>>>>>
>>>>> your help is much appreciated,
>>>>> Nezar.
>>>>>
>>>>>
>>>>> On 14 April 2016 at 16:49, ilya <il...@gmail.com> wrote:
>>>>>
>>>>>> It appears you have pending jobs that are not letting you go through.
>>>>>>
>>>>>> Usually stop and starting management service should be enough.
>>>>>>
>>>>>> However, try running this script from your management server to show
>>> us
>>>>>> what you have running in async_job table.
>>>>>>
>>>>>>
>>>>>>
>>>>>>> #!/bin/bash
>>>>>>> DATESTAMP=$(date +%m%d%y-%H%M%S)
>>>>>>> DBPASS=$(java -classpath
>>>>>> /usr/share/cloudstack-common/lib/jasypt-1.9.0.jar
>>>>>> org.jasypt.intf.cli.JasyptPBEStringDecryptionCLI input="$(cat
>>>>>> /etc/cloudstack/management/db.properties | grep db.cloud.password |
>>> awk
>>>>>> -F'(' '{print $2}' | sed 's/)//g')" password="$(cat
>>>>>> /etc/cloudstack/management/key)" | grep -A2 OUTPUT | tail -1)
>>>>>>> DBHOST=$(cat /etc/cloudstack/management/db.properties | grep
>>>>>> db.cloud.host | awk -F'=' '{print $2}' | tail -1 )
>>>>>>> DBUSER=$(cat /etc/cloudstack/management/db.properties | grep
>>>>>> db.cloud.username | awk -F'=' '{print $2}')
>>>>>>> DB=$(cat /etc/cloudstack/management/db.properties | grep
>>> db.cloud.name
>>>>>> | awk -F'=' '{print $2}')
>>>>>>> DBPORT=$(cat /etc/cloudstack/management/db.properties | grep
>>>>>> db.cloud.port | awk -F'=' '{print $2}')
>>>>>>> MYSQLCMD="mysql -h $DBHOST -u $DBUSER -P $DBPORT -p$DBPASS $DB"
>>>>>>> echo 'SELECT id, user_id, account_id, instance_type, instance_id,
>>>>>> job_cmd, job_cmd_info, job_dispatcher, job_pending_signals FROM
>>>>>> cloud.async_job where job_status=0 and job_dispatcher not like
>>>>>> "pseudoJobDispatcher"' | $MYSQLCMD
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 4/13/16 5:16 PM, Nezar Madbouh wrote:
>>>>>>> Hi Simon,
>>>>>>>
>>>>>>> The following work was done but unfortunately the problem is not
>>> fixed:
>>>>>>>
>>>>>>>
>>>>>>> on management server:
>>>>>>>
>>>>>>> service cloudstack-management stop
>>>>>>>
>>>>>>>
>>>>>>> on the host:
>>>>>>>
>>>>>>> service cloudstack-agent stop
>>>>>>>
>>>>>>> service libvirtd restart
>>>>>>>
>>>>>>> service cloudstack-agent start
>>>>>>>
>>>>>>>
>>>>>>> on management server:
>>>>>>>
>>>>>>> service cloudstack-management start
>>>>>>>
>>>>>>> Regards,
>>>>>>> Nezar.
>>>>>>>
>>>>>>> On 24 March 2016 at 05:06, Simon Weller <sw...@ena.com> wrote:
>>>>>>>
>>>>>>>> Yes, I think that order is fine. Make sure the processes are indeed
>>>>>>>> shutdown before moving on to the next step.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ________________________________________
>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>>>>>>>> Sent: Tuesday, March 22, 2016 5:33 PM
>>>>>>>> To: users@cloudstack.apache.org
>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable to
>>> start
>>>>>>>> instance due to Unable to get answer that is of class
>>>>>>>> com.cloud.agent.api.StartAnswer
>>>>>>>>
>>>>>>>> Hi Simon,
>>>>>>>>
>>>>>>>> Can you please check if the following sequence of action is correct:
>>>>>>>>
>>>>>>>> on management server:
>>>>>>>>
>>>>>>>> service cloudstack-management stop
>>>>>>>>
>>>>>>>>
>>>>>>>> on the host:
>>>>>>>>
>>>>>>>> service cloudstack-agent stop
>>>>>>>>
>>>>>>>> service libvirtd restart
>>>>>>>>
>>>>>>>> service cloudstack-agent start
>>>>>>>>
>>>>>>>>
>>>>>>>> on management server:
>>>>>>>>
>>>>>>>> service cloudstack-management start
>>>>>>>>
>>>>>>>> Thanks very much for your help.
>>>>>>>> Nezar.
>>>>>>>>
>>>>>>>> On 23 March 2016 at 08:39, Simon Weller <sw...@ena.com> wrote:
>>>>>>>>
>>>>>>>>> Shutting down all the management services should clear the jobs.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ________________________________________
>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>>>>>>>>> Sent: Tuesday, March 22, 2016 4:28 PM
>>>>>>>>> To: users@cloudstack.apache.org
>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable to
>>>>>> start
>>>>>>>>> instance due to Unable to get answer that is of class
>>>>>>>>> com.cloud.agent.api.StartAnswer
>>>>>>>>>
>>>>>>>>> Hi Simon,
>>>>>>>>>
>>>>>>>>> I will plan to restart agents, libvirtd and management services.
>>>>>>>>>
>>>>>>>>> How do I clear hung jobs.
>>>>>>>>>
>>>>>>>>> Your help is much appreciated,
>>>>>>>>> Nezar.
>>>>>>>>>
>>>>>>>>> On 22 March 2016 at 23:03, Simon Weller <sw...@ena.com> wrote:
>>>>>>>>>
>>>>>>>>>> Nezar,
>>>>>>>>>>
>>>>>>>>>> I've looked through the code related to your release and it seems
>>>>>> that
>>>>>>>> in
>>>>>>>>>> order to throw this exception, 10 attempts have been made to
>>> retrieve
>>>>>>>> the
>>>>>>>>>> disk information prior to the exception being thrown.
>>>>>>>>>>
>>>>>>>>>> It appears there is something really odd going on here. I'm
>>> starting
>>>>>>>> to
>>>>>>>>>> wonder whether it might be worth while for you to stop all of your
>>>>>>>>> agents,
>>>>>>>>>> plus management server processes, restart libvirtd on each of the
>>>>>> hosts
>>>>>>>>> and
>>>>>>>>>> then bring the agents back up prior to starting the management
>>>>>> servers.
>>>>>>>>>>
>>>>>>>>>> There are a bunch of logs the pastebin you posted that seems to
>>>>>>>> indicate
>>>>>>>>>> you have some jobs hung, for example:
>>>>>>>>>>
>>>>>>>>>> 2016-03-14 21:45:31,828 WARN [o.a.c.f.j.i.AsyncJobMonitor]
>>>>>>>>>> (Timer-1:ctx-0c090bd7) Task (job-9066) has been pending for
>>> 1660851
>>>>>>>>> seconds
>>>>>>>>>> 2016-03-14 21:45:31,828 WARN [o.a.c.f.j.i.AsyncJobMonitor]
>>>>>>>>>> (Timer-1:ctx-0c090bd7) Task (job-8133) has been pending for
>>> 4253003
>>>>>>>>> seconds
>>>>>>>>>>
>>>>>>>>>> Restarting the agents, libvirtd and management will not cause an
>>>>>>>> outage
>>>>>>>>>> to your cloud, but I would suggest, if you decided to do this, you
>>>>>>>> should
>>>>>>>>>> complete it in a maintenance window.
>>>>>>>>>>
>>>>>>>>>> - Si
>>>>>>>>>>
>>>>>>>>>> ________________________________________
>>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>>>>>>>>>> Sent: Sunday, March 20, 2016 10:27 PM
>>>>>>>>>> To: users@cloudstack.apache.org
>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable to
>>>>>> start
>>>>>>>>>> instance due to Unable to get answer that is of class
>>>>>>>>>> com.cloud.agent.api.StartAnswer
>>>>>>>>>>
>>>>>>>>>> Hi Simon,
>>>>>>>>>>
>>>>>>>>>> Please see answers to your questions below. I am not sure if I
>>>>>> provided
>>>>>>>>> the
>>>>>>>>>> correct answer for the primary and secondary storage
>>> configuration.
>>>>>>>>> Please
>>>>>>>>>> let me know if this is not what you are after:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Can you screenshot and post somewhere your secondary storage
>>>>>>>>> configuration
>>>>>>>>>> and your primary storage configuration from the UI with the
>>> details?
>>>>>>>>>>
>>>>>>>>>> mysql> select * from configuration where name like '%secondary%'\G
>>>>>>>>>> *************************** 1. row ***************************
>>>>>>>>>> category: Account Defaults
>>>>>>>>>> instance: DEFAULT
>>>>>>>>>> component: management-server
>>>>>>>>>> name: max.account.secondary.storage
>>>>>>>>>> value: 400
>>>>>>>>>> description: The default maximum secondary storage space (in
>>> GiB)
>>>>>>>> that
>>>>>>>>>> can be used for an account
>>>>>>>>>> default_value: 400
>>>>>>>>>> updated: NULL
>>>>>>>>>> scope: NULL
>>>>>>>>>> is_dynamic: 0
>>>>>>>>>> *************************** 2. row ***************************
>>>>>>>>>> category: Project Defaults
>>>>>>>>>> instance: DEFAULT
>>>>>>>>>> component: management-server
>>>>>>>>>> name: max.project.secondary.storage
>>>>>>>>>> value: 400
>>>>>>>>>> description: The default maximum secondary storage space (in
>>> GiB)
>>>>>>>> that
>>>>>>>>>> can be used for a project
>>>>>>>>>> default_value: 400
>>>>>>>>>> updated: NULL
>>>>>>>>>> scope: NULL
>>>>>>>>>> is_dynamic: 0
>>>>>>>>>> *************************** 3. row ***************************
>>>>>>>>>> category: Hidden
>>>>>>>>>> instance: DEFAULT
>>>>>>>>>> component: management-server
>>>>>>>>>> name: secondary.storage.vm
>>>>>>>>>> value: +rwRhJOXPymbQOuAMLnhiw==
>>>>>>>>>> description: Deploys a VM per zone to manage secondary storage
>>> if
>>>>>>>> true,
>>>>>>>>>> otherwise secondary storage is mounted on management server
>>>>>>>>>> default_value: NULL
>>>>>>>>>> updated: NULL
>>>>>>>>>> scope: NULL
>>>>>>>>>> is_dynamic: 0
>>>>>>>>>> *************************** 4. row ***************************
>>>>>>>>>> category: Advanced
>>>>>>>>>> instance: DEFAULT
>>>>>>>>>> component: ExtensionRegistry
>>>>>>>>>> name: secondary.storage.vm.allocators.exclude
>>>>>>>>>> value: NULL
>>>>>>>>>> description: Extensions to exclude from being registered
>>>>>>>>>> default_value: NULL
>>>>>>>>>> updated: 2014-12-04 00:18:15
>>>>>>>>>> scope: Global
>>>>>>>>>> is_dynamic: 0
>>>>>>>>>> *************************** 5. row ***************************
>>>>>>>>>> category: NetworkManager
>>>>>>>>>> instance: DEFAULT
>>>>>>>>>> component: management-server
>>>>>>>>>> name: vm.network.nic.max.secondary.ipaddresses
>>>>>>>>>> value: NULL
>>>>>>>>>> description: Specify the number of secondary ip addresses per
>>> nic
>>>>>> per
>>>>>>>>> vm
>>>>>>>>>> default_value: 256
>>>>>>>>>> updated: NULL
>>>>>>>>>> scope: NULL
>>>>>>>>>> is_dynamic: 0
>>>>>>>>>> 5 rows in set (0.00 sec)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> mysql> select * from configuration where name like '%primary%'\G
>>>>>>>>>> *************************** 1. row ***************************
>>>>>>>>>> category: Account Defaults
>>>>>>>>>> instance: DEFAULT
>>>>>>>>>> component: management-server
>>>>>>>>>> name: max.account.primary.storage
>>>>>>>>>> value: 200
>>>>>>>>>> description: The default maximum primary storage space (in GiB)
>>>>>> that
>>>>>>>>> can
>>>>>>>>>> be used for an account
>>>>>>>>>> default_value: 200
>>>>>>>>>> updated: NULL
>>>>>>>>>> scope: NULL
>>>>>>>>>> is_dynamic: 0
>>>>>>>>>> *************************** 2. row ***************************
>>>>>>>>>> category: Project Defaults
>>>>>>>>>> instance: DEFAULT
>>>>>>>>>> component: management-server
>>>>>>>>>> name: max.project.primary.storage
>>>>>>>>>> value: 200
>>>>>>>>>> description: The default maximum primary storage space (in GiB)
>>>>>> that
>>>>>>>>> can
>>>>>>>>>> be used for a project
>>>>>>>>>> default_value: 200
>>>>>>>>>> updated: NULL
>>>>>>>>>> scope: NULL
>>>>>>>>>> is_dynamic: 0
>>>>>>>>>> *************************** 3. row ***************************
>>>>>>>>>> category: Storage
>>>>>>>>>> instance: DEFAULT
>>>>>>>>>> component: TemplateManager
>>>>>>>>>> name: primary.storage.download.wait
>>>>>>>>>> value: 10800
>>>>>>>>>> description: In second, timeout for download template to primary
>>>>>>>>> storage
>>>>>>>>>> default_value: NULL
>>>>>>>>>> updated: NULL
>>>>>>>>>> scope: NULL
>>>>>>>>>> is_dynamic: 0
>>>>>>>>>> 3 rows in set (0.00 sec)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Also, what is the Cloudstack version you are running and what is
>>> the
>>>>>> OS
>>>>>>>>>> you're running it on?
>>>>>>>>>>
>>>>>>>>>> Cloudstack 4.4.1
>>>>>>>>>>
>>>>>>>>>> Centos 6.6
>>>>>>>>>>
>>>>>>>>>> Are you using actimeo=0 in your NFS mount line?
>>>>>>>>>>
>>>>>>>>>> Primary storage on Hypervisors has the following parameters
>>>>>>>>>> 10.61.154.10:/Cloudstack/primary on
>>>>>>>>>> /mnt/b3647480-5709-36eb-9ddd-e2c666f30b57 type nfs
>>>>>>>>>>
>>> (rw,noatime,nfsvers=3,proto=tcp,hard,nolock,intr,addr=10.61.154.10)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Secondary storage on SSVM has the following parameters
>>>>>>>>>> 10.61.154.10:/Cloudstack/secondary on
>>>>>>>>>> /mnt/SecStorage/d11d5c92-6950-30e8-a0b5-3cd46a17ec14 type nfs4
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>> (rw,relatime,vers=4,rsize=262144,wsize=262144,namlen=255,acdirmin=0,acdirmax=0,soft,proto=tcp,port=0,timeo=133,retrans=
>>>>>> 2147483647
>>>>>>
>>> ,sec=sys,clientaddr=10.61.154.70,minorversion=0,local_lock=none,addr=10.61.154.10)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thanks & Regards,
>>>>>>>>>> Nezar.
>>>>>>>>>>
>>>>>>>>>> On 18 March 2016 at 23:50, Simon Weller <sw...@ena.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Also, what is the Cloudstack version you are running and what is
>>> the
>>>>>>>> OS
>>>>>>>>>>> you're running it on?
>>>>>>>>>>>
>>>>>>>>>>> Are you using actimeo=0 in your NFS mount line?
>>>>>>>>>>>
>>>>>>>>>>> ________________________________________
>>>>>>>>>>> From: Simon Weller <sw...@ena.com>
>>>>>>>>>>> Sent: Friday, March 18, 2016 7:40 AM
>>>>>>>>>>> To: users@cloudstack.apache.org
>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable to
>>>>>>>> start
>>>>>>>>>>> instance due to Unable to get answer that is of class
>>>>>>>>>>> com.cloud.agent.api.StartAnswer
>>>>>>>>>>>
>>>>>>>>>>> Nexar,
>>>>>>>>>>>
>>>>>>>>>>> That's normal behaviour. You can only ssh to the link local on
>>> the
>>>>>>>> host
>>>>>>>>>>> where the VM resides.
>>>>>>>>>>>
>>>>>>>>>>> Can you screenshot and post somewhere your secondary storage
>>>>>>>>>> configuration
>>>>>>>>>>> and your primary storage configuration from the UI with the
>>> details?
>>>>>>>>>>>
>>>>>>>>>>> - Si
>>>>>>>>>>>
>>>>>>>>>>> ________________________________________
>>>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>>>>>>>>>>> Sent: Thursday, March 17, 2016 8:19 PM
>>>>>>>>>>> To: users@cloudstack.apache.org
>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable to
>>>>>>>> start
>>>>>>>>>>> instance due to Unable to get answer that is of class
>>>>>>>>>>> com.cloud.agent.api.StartAnswer
>>>>>>>>>>>
>>>>>>>>>>> Hi Simon,
>>>>>>>>>>>
>>>>>>>>>>> I just noticed that from other Hosts I can't ssh into the SSVM:
>>>>>>>>>>>
>>>>>>>>>>> [root@eqx-cs-cmp-01 ~]# ssh -i /root/.ssh/id_rsa.cloud -p 3922
>>>>>>>>>>> root@169.254.3.161
>>>>>>>>>>> ssh: connect to host 169.254.3.161 port 3922: No route to host
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I can only ssh from one Host:
>>>>>>>>>>> [root@eqx-cs-cmp-06 ~]# ssh -i /root/.ssh/id_rsa.cloud -p 3922
>>>>>>>>>>> root@169.254.3.161
>>>>>>>>>>> Linux s-11-VM 3.2.0-4-amd64 #1 SMP Debian 3.2.57-3 x86_64
>>>>>>>>>>>
>>>>>>>>>>> Thanks & Regards,
>>>>>>>>>>> Nezar.
>>>>>>>>>>>
>>>>>>>>>>> On 18 March 2016 at 10:36, Nezar Madbouh <
>>>>>>>> nezar.madbouh@ipscape.com.au
>>>>>>>>>>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Simon,
>>>>>>>>>>>>
>>>>>>>>>>>> Yes existing VMs are successfully running. Yesterday I was able
>>> to
>>>>>>>>>> create
>>>>>>>>>>>> a VM in Dev Environment but I can't create VMs in Prod
>>> environment.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Nezar.
>>>>>>>>>>>>
>>>>>>>>>>>> On 18 March 2016 at 09:12, Simon Weller <sw...@ena.com>
>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> I'm assuming you have other VMs successfully running, correct?
>>> So
>>>>>>>>> you
>>>>>>>>>>> can
>>>>>>>>>>>>> write to your primary storage.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> ________________________________________
>>>>>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>>>>>>>>>>>>> Sent: Thursday, March 17, 2016 4:43 PM
>>>>>>>>>>>>> To: users@cloudstack.apache.org
>>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable
>>> to
>>>>>>>>>> start
>>>>>>>>>>>>> instance due to Unable to get answer that is of class
>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Simon and Shweta,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Please see the result below:
>>>>>>>>>>>>>
>>>>>>>>>>>>> root@s-11-VM:~# /usr/local/cloud/systemvm/ssvm-check.sh
>>>>>>>>>>>>> ================================================
>>>>>>>>>>>>> First DNS server is 8.8.8.8
>>>>>>>>>>>>> PING 8.8.8.8 (8.8.8.8): 48 data bytes
>>>>>>>>>>>>> 56 bytes from 8.8.8.8: icmp_seq=0 ttl=57 time=5.209 ms
>>>>>>>>>>>>> 56 bytes from 8.8.8.8: icmp_seq=1 ttl=57 time=0.936 ms
>>>>>>>>>>>>> --- 8.8.8.8 ping statistics ---
>>>>>>>>>>>>> 2 packets transmitted, 2 packets received, 0% packet loss
>>>>>>>>>>>>> round-trip min/avg/max/stddev = 0.936/3.072/5.209/2.137 ms
>>>>>>>>>>>>> Good: Can ping DNS server
>>>>>>>>>>>>> ================================================
>>>>>>>>>>>>> Good: DNS resolves download.cloud.com
>>>>>>>>>>>>> ================================================
>>>>>>>>>>>>> nfs is currently mounted
>>>>>>>>>>>>> Mount point is
>>>>>>>> /mnt/SecStorage/d11d5c92-6950-30e8-a0b5-3cd46a17ec14
>>>>>>>>>>>>> Good: Can write to mount point
>>>>>>>>>>>>> ================================================
>>>>>>>>>>>>> Management server is 10.61.153.20. Checking connectivity.
>>>>>>>>>>>>> Good: Can connect to management server port 8250
>>>>>>>>>>>>> ================================================
>>>>>>>>>>>>> Good: Java process is running
>>>>>>>>>>>>> ================================================
>>>>>>>>>>>>> Tests Complete. Look for ERROR or WARNING above.
>>>>>>>>>>>>> root@s-11-VM:~#
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks & Regards,
>>>>>>>>>>>>> Nezar.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 17 March 2016 at 09:42, Simon Weller <sw...@ena.com>
>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Nezar,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Log into the Cloudstack UI.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Click on "Infrastructure" on the left menu. Click on View All
>>>>>>>>> below
>>>>>>>>>>>>>> "System VMs". find the item marked "Secondary Storage VM" and
>>>>>>>>> then
>>>>>>>>>>>>> click
>>>>>>>>>>>>>> on the Name of that VM on the left (it will be s-XXXX-VM).
>>>>>>>> You'll
>>>>>>>>>> see
>>>>>>>>>>> a
>>>>>>>>>>>>>> bunch of information about the VM. Find the Host line and also
>>>>>>>> the
>>>>>>>>>>> Link
>>>>>>>>>>>>>> Local IP Address line and then follow the original
>>> instructions
>>>>>>>> in
>>>>>>>>>> my
>>>>>>>>>>>>>> previous email.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> - Si
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ________________________________________
>>>>>>>>>>>>>> From: Shweta Agarwal <sh...@accelerite.com>
>>>>>>>>>>>>>> Sent: Wednesday, March 16, 2016 12:16 AM
>>>>>>>>>>>>>> To: users@cloudstack.apache.org
>>>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable
>>>>>>>> to
>>>>>>>>>>> start
>>>>>>>>>>>>>> instance due to Unable to get answer that is of class
>>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Nezar,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> You can find host name on detail view of SSVM in UI .
>>>>>>>>>>>>>> You can also use list api command listSystemVms to get the
>>> host
>>>>>>>>>>> details
>>>>>>>>>>>>> of
>>>>>>>>>>>>>> system vms.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>> Shweta
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ________________________________________
>>>>>>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>>>>>>>>>>>>>> Sent: Wednesday, March 16, 2016 9:58 AM
>>>>>>>>>>>>>> To: users@cloudstack.apache.org
>>>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable
>>>>>>>> to
>>>>>>>>>>> start
>>>>>>>>>>>>>> instance due to Unable to get answer that is of class
>>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Simon,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> How can I find out which host the SSVM is on?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks & Regards,
>>>>>>>>>>>>>> Nezar.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 16 March 2016 at 12:28, Simon Weller <sw...@ena.com>
>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Can you ssh into the Secondary Storage VM?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> This can be done by getting the link-local-ip address from
>>> the
>>>>>>>>>>>>>>> infrastructure menu in the UI and then sshing into the host
>>>>>>>> the
>>>>>>>>>> SSVM
>>>>>>>>>>>>> is
>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>> and issuing this command as root: ssh -i
>>>>>>>> /root/.ssh/id_rsa.cloud
>>>>>>>>>> -p
>>>>>>>>>>>>> 3922
>>>>>>>>>>>>>>> root@link-local-ip (replace link-local-ip with the link
>>>>>>>> local
>>>>>>>>>> ip).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Try running the SSVM diangnostic script:
>>>>>>>>>>>>>>> usr/local/cloud/systemvm/ssvm-check.sh
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> - Si
>>>>>>>>>>>>>>> ________________________________________
>>>>>>>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>>>>>>>>>>>>>>> Sent: Tuesday, March 15, 2016 7:49 PM
>>>>>>>>>>>>>>> To: users@cloudstack.apache.org
>>>>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message:
>>> Unable
>>>>>>>>> to
>>>>>>>>>>>>> start
>>>>>>>>>>>>>>> instance due to Unable to get answer that is of class
>>>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Simon,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Please see details below:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> mysql> use cloud; select id, name, status, type from host
>>>>>>>> where
>>>>>>>>>> id =
>>>>>>>>>>>>> 5;
>>>>>>>>>>>>>>> Reading table information for completion of table and column
>>>>>>>>> names
>>>>>>>>>>>>>>> You can turn off this feature to get a quicker startup with
>>> -A
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Database changed
>>>>>>>>>>>>>>> +----+------------------------------+--------+---------+
>>>>>>>>>>>>>>> | id | name | status | type |
>>>>>>>>>>>>>>> +----+------------------------------+--------+---------+
>>>>>>>>>>>>>>> | 5 | eqx-cs-cmp-02.ipscape.com.au | up | routing |
>>>>>>>>>>>>>>> +----+------------------------------+--------+---------+
>>>>>>>>>>>>>>> 1 row in set (0.00 sec)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> cat /var/log/cloudstack/agent/agent.log
>>>>>>>>>>>>>>> com.cloud.utils.exception.CloudRuntimeException:
>>>>>>>>>>>>>>> com.cloud.utils.exception.CloudRuntimeException: Can't find
>>>>>>>>>>>>>>> volume:089ff3ca-9bff-402c-a14e-ea7a35aaad5c
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>> com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.getPhysicalDisk(KVMStoragePoolManager.java:262)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.createVbd(LibvirtComputingResource.java:3921)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(LibvirtComputingResource.java:3789)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1333)
>>>>>>>>>>>>>>> at
>>>>>>>> com.cloud.agent.Agent.processRequest(Agent.java:501)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>
>>> com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:808)
>>>>>>>>>>>>>>> at com.cloud.utils.nio.Task.run(Task.java:84)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>>>>>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>>>>>>>>>> agent.log.2016-03-03.gz (END)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [root@eqx-cs-cmp-02 agent]# virsh pool-list
>>>>>>>>>>>>>>> Name State Autostart
>>>>>>>>>>>>>>> -----------------------------------------
>>>>>>>>>>>>>>> 2d0967f8-50d6-4d6d-87b7-4da0596f6924 active no
>>>>>>>>>>>>>>> b3647480-5709-36eb-9ddd-e2c666f30b57 active no
>>>>>>>>>>>>>>> e3831246-ff55-4af4-9964-a17565edaccc active no
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [Root @ eqx-cs-cmp-02 agent] # df -h
>>>>>>>>>>>>>>> Filesystem Size Used Avail Use% Mounted on
>>>>>>>>>>>>>>> /dev/mapper/vg_root-lv_root
>>>>>>>>>>>>>>> 203G 215G 1.3G 1% /
>>>>>>>>>>>>>>> tmpfs 48g 48g 0 0% / dev / shm
>>>>>>>>>>>>>>> / Dev / sda1 477m 427m 25M 6% / boot
>>>>>>>>>>>>>>> xx.xx.xx.xx:/Cloudstack/primary
>>>>>>>>>>>>>>> 7.6T 4.1T 3.5T 54%
>>>>>>>>>>>>>>> /mnt/b3647480-5709-36eb-9ddd-e2c666f30b57
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I also noticed the following in cloudstack-agent.err
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> libvirt: QEMU Driver error : Domain not found: no domain with
>>>>>>>>>>> matching
>>>>>>>>>>>>>> name
>>>>>>>>>>>>>>> 'i-11-600-VM'
>>>>>>>>>>>>>>> libvirt: Storage Driver error : Storage volume not found: no
>>>>>>>>>> storage
>>>>>>>>>>>>> vol
>>>>>>>>>>>>>>> with matching name '089ff3ca-9bff-402c-a14e-ea7a35aaad5c'
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Your help is much appreciated,
>>>>>>>>>>>>>>> Nezar
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 16 March 2016 at 11:03, Simon Weller <sw...@ena.com>
>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Nezar,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> ok, I just took a look at a pastebin you posted.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I see it complaining about the inability to find a volume
>>>>>>>>> prior
>>>>>>>>>> to
>>>>>>>>>>>>>>>> throwing the host exception.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Firstly, let's find out who host 5 is. Can you log into your
>>>>>>>>>> MySQL
>>>>>>>>>>>>>> server
>>>>>>>>>>>>>>>> and run this:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> use cloud;select id,name,status,type from host where id=5;
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Can you post the results?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> If it's a host, can you post the
>>>>>>>>>>> /var/log/cloudstack/agent/agent.log
>>>>>>>>>>>>>> from
>>>>>>>>>>>>>>>> that host?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> And also run virsh pool-list on the host in question?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> - Si
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> ________________________________________
>>>>>>>>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>>>>>>>>>>>>>>>> Sent: Tuesday, March 15, 2016 6:11 PM
>>>>>>>>>>>>>>>> To: users@cloudstack.apache.org
>>>>>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message:
>>>>>>>> Unable
>>>>>>>>>> to
>>>>>>>>>>>>> start
>>>>>>>>>>>>>>>> instance due to Unable to get answer that is of class
>>>>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi Simon,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> When you say "double check your routers tied to the
>>>>>>>> network",
>>>>>>>>> do
>>>>>>>>>>> you
>>>>>>>>>>>>>> mean
>>>>>>>>>>>>>>>> the physical routers or the virtual routers?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> When I try to create a VM in Prod Environment it fails. I
>>>>>>>>> can't
>>>>>>>>>>> ping
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> Prod virtual router from management server or Host.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> But I can ping Dev virtual router from management server and
>>>>>>>>>> Host
>>>>>>>>>>>>> and I
>>>>>>>>>>>>>>> was
>>>>>>>>>>>>>>>> able to create a VM in Dev Environment.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> How can I troubleshoot the Prod Virtual router. I launched
>>>>>>>> the
>>>>>>>>>>>>> console
>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>> couldn't login.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Please note that we need to take into consideration that the
>>>>>>>>>>>>> affected
>>>>>>>>>>>>>>>> network is the Production network and we need to minimise
>>>>>>>>>> service
>>>>>>>>>>>>>> outage
>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>> much as possible.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Your help is much appreciated,
>>>>>>>>>>>>>>>> Nezar
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On 16 March 2016 at 08:37, Simon Weller <sw...@ena.com>
>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I've seen this before where a Virtual Router has crashed
>>>>>>>> and
>>>>>>>>>> the
>>>>>>>>>>>>>>>>> management server can't communicate with it.
>>>>>>>>>>>>>>>>> Are you using an advanced network setup?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> If so, double check your routers tied to the network where
>>>>>>>>> you
>>>>>>>>>>> are
>>>>>>>>>>>>>>>>> attempting to start an instance.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> - Si
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> ________________________________________
>>>>>>>>>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>>>>>>>>>>>>>>>>> Sent: Tuesday, March 15, 2016 4:02 PM
>>>>>>>>>>>>>>>>> To: users@cloudstack.apache.org
>>>>>>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message:
>>>>>>>>> Unable
>>>>>>>>>>> to
>>>>>>>>>>>>>> start
>>>>>>>>>>>>>>>>> instance due to Unable to get answer that is of class
>>>>>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi Glenn,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> We have 16 hosts, all KVM.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> The error is not specific to Host 5. Every time I try to
>>>>>>>>>> create
>>>>>>>>>>> a
>>>>>>>>>>>>> VM
>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>> gives me the same error message but on different host.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> How do I restart the agent on the host. Is it service
>>>>>>>>>> affecting?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks very much for your help.
>>>>>>>>>>>>>>>>> Neza
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On 15 March 2016 at 17:37, Glenn Wagner <
>>>>>>>>>>>>> glenn.wagner@shapeblue.com>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Can you explain your environment first , how many hosts
>>>>>>>> ,
>>>>>>>>>> what
>>>>>>>>>>>>> is
>>>>>>>>>>>>>>> your
>>>>>>>>>>>>>>>>>> hypervisors (KVM,Xenserver,VMware)
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> The error seems to be with your Host 5 Hypervisor
>>>>>>>>>>>>>>>>>> Resource [Host:5] is unreachable:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Go to the infrastructure section and see what host5 is
>>>>>>>>>>>>> reporting,
>>>>>>>>>>>>>>>>>> Have you tried to restart the agent on the Host?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>> Glenn
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> [image: ShapeBlue] <http://www.shapeblue.com>
>>>>>>>>>>>>>>>>>> Glenn Wagner
>>>>>>>>>>>>>>>>>> Senior Consultant , ShapeBlue
>>>>>>>>>>>>>>>>>> d: * | s: +27 21 527 0091*
>>>>>>>>>>>>> <%7C%20s:%20+27%2021%20527%200091> |
>>>>>>>>>>>>>>> m:
>>>>>>>>>>>>>>>>>> *+27 73 917 4111* <+27%2073%20917%204111>
>>>>>>>>>>>>>>>>>> e: *glenn.wagner@shapeblue.com | t: *
>>>>>>>>>>>>>>>>>> <glenn.wagner@shapeblue.com%20%7C%20t:> | w: *
>>>>>>>>>>>>> www.shapeblue.com
>>>>>>>>>>>>>> *
>>>>>>>>>>>>>>>>>> <http://www.shapeblue.com>
>>>>>>>>>>>>>>>>>> a:
>>>>>>>>>>>>>>>>>> 2nd Floor, Oudehuis Centre, 122 Main Rd, Somerset West
>>>>>>>>> Cape
>>>>>>>>>>> Town
>>>>>>>>>>>>>> 7130
>>>>>>>>>>>>>>>>> South Africa
>>>>>>>>>>>>>>>>>> Shape Blue Ltd is a company incorporated in England &
>>>>>>>>> Wales.
>>>>>>>>>>>>>>> ShapeBlue
>>>>>>>>>>>>>>>>>> Services India LLP is a company incorporated in India
>>>>>>>> and
>>>>>>>>> is
>>>>>>>>>>>>>> operated
>>>>>>>>>>>>>>>>> under
>>>>>>>>>>>>>>>>>> license from Shape Blue Ltd. Shape Blue Brasil
>>>>>>>> Consultoria
>>>>>>>>>>> Ltda
>>>>>>>>>>>>> is
>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>> company incorporated in Brasil and is operated under
>>>>>>>>> license
>>>>>>>>>>>>> from
>>>>>>>>>>>>>>> Shape
>>>>>>>>>>>>>>>>>> Blue Ltd. ShapeBlue SA Pty Ltd is a company registered
>>>>>>>> by
>>>>>>>>>> The
>>>>>>>>>>>>>>> Republic
>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>> South Africa and is traded under license from Shape Blue
>>>>>>>>>> Ltd.
>>>>>>>>>>>>>>> ShapeBlue
>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>> a registered trademark.
>>>>>>>>>>>>>>>>>> This email and any attachments to it may be confidential
>>>>>>>>> and
>>>>>>>>>>> are
>>>>>>>>>>>>>>>> intended
>>>>>>>>>>>>>>>>>> solely for the use of the individual to whom it is
>>>>>>>>>> addressed.
>>>>>>>>>>>>> Any
>>>>>>>>>>>>>>> views
>>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>> opinions expressed are solely those of the author and do
>>>>>>>>> not
>>>>>>>>>>>>>>>> necessarily
>>>>>>>>>>>>>>>>>> represent those of Shape Blue Ltd or related companies.
>>>>>>>> If
>>>>>>>>>> you
>>>>>>>>>>>>> are
>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> intended recipient of this email, you must neither take
>>>>>>>>> any
>>>>>>>>>>>>> action
>>>>>>>>>>>>>>>> based
>>>>>>>>>>>>>>>>>> upon its contents, nor copy or show it to anyone. Please
>>>>>>>>>>> contact
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> sender
>>>>>>>>>>>>>>>>>> if you believe you have received this email in error.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>>>>>>>>> From: Nezar Madbouh [mailto:
>>>>>>>> nezar.madbouh@ipscape.com.au]
>>>>>>>>>>>>>>>>>> Sent: Tuesday, 15 March 2016 5:45 AM
>>>>>>>>>>>>>>>>>> To: users@cloudstack.apache.org
>>>>>>>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message:
>>>>>>>>>> Unable
>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> start
>>>>>>>>>>>>>>>>>> instance due to Unable to get answer that is of class
>>>>>>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi Jeremy,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I have uploaded the contents to
>>>>>>>>>> http://pastebin.com/1J3mmgTh
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> your help is much appreciated,
>>>>>>>>>>>>>>>>>> Nezar.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On 15 March 2016 at 13:46, Jeremy Peterson <
>>>>>>>>>>>>> jpeterson@acentek.net>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I don't see any attachments can you create a pastebin.
>>>>>>>>>>> That's
>>>>>>>>>>>>>>> easier
>>>>>>>>>>>>>>>>>>> to look through as I think attachments are removed
>>>>>>>>> before
>>>>>>>>>>>>> being
>>>>>>>>>>>>>>> sent
>>>>>>>>>>>>>>>>>>> to user group.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Jeremy
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> -------- Original message --------
>>>>>>>>>>>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>>>>>>>>>>>>>>>>>>> Date: 3/14/2016 6:38 PM (GMT-06:00)
>>>>>>>>>>>>>>>>>>> To: users@cloudstack.apache.org
>>>>>>>>>>>>>>>>>>> Subject: Unable to create a New VM - Error message:
>>>>>>>>> Unable
>>>>>>>>>>> to
>>>>>>>>>>>>>> start
>>>>>>>>>>>>>>>>>>> instance due to Unable to get answer that is of class
>>>>>>>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi Shweta and Sadhu,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Please find attached managerment-server.log which
>>>>>>>>> contains
>>>>>>>>>>> the
>>>>>>>>>>>>>> logs
>>>>>>>>>>>>>>>>>>> since I started creating a new VM (VM id 600).
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Please note that we don't have agent.log files
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Your help is much appreciated.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>> Nezar.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Find out more about ShapeBlue and our range of
>>>>>>>> CloudStack
>>>>>>>>>>>>> related
>>>>>>>>>>>>>>>>> services:
>>>>>>>>>>>>>>>>>> IaaS Cloud Design & Build
>>>>>>>>>>>>>>>>>> <http://shapeblue.com/iaas-cloud-design-and-build//> |
>>>>>>>>>>> CSForge
>>>>>>>>>>>>> -
>>>>>>>>>>>>>>> rapid
>>>>>>>>>>>>>>>>>> IaaS deployment framework <
>>>>>>>> http://shapeblue.com/csforge/>
>>>>>>>>>>>>>>>>>> CloudStack Consulting <
>>>>>>>>>>>>>> http://shapeblue.com/cloudstack-consultancy/>
>>>>>>>>>>>>>>> |
>>>>>>>>>>>>>>>>> CloudStack
>>>>>>>>>>>>>>>>>> Software Engineering
>>>>>>>>>>>>>>>>>> <http://shapeblue.com/cloudstack-software-engineering/>
>>>>>>>>>>>>>>>>>> CloudStack Infrastructure Support
>>>>>>>>>>>>>>>>>> <
>>>>>>>> http://shapeblue.com/cloudstack-infrastructure-support/>
>>>>>>>>> |
>>>>>>>>>>>>>>> CloudStack
>>>>>>>>>>>>>>>>>> Bootcamp Training Courses <
>>>>>>>>>>>>>> http://shapeblue.com/cloudstack-training/
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> DISCLAIMER
>>>>>>>>>>>>>> ==========
>>>>>>>>>>>>>> This e-mail may contain privileged and confidential
>>> information
>>>>>>>>>> which
>>>>>>>>>>> is
>>>>>>>>>>>>>> the property of Accelerite, a Persistent Systems business. It
>>> is
>>>>>>>>>>>>> intended
>>>>>>>>>>>>>> only for the use of the individual or entity to which it is
>>>>>>>>>> addressed.
>>>>>>>>>>>>> If
>>>>>>>>>>>>>> you are not the intended recipient, you are not authorized to
>>>>>>>>> read,
>>>>>>>>>>>>> retain,
>>>>>>>>>>>>>> copy, print, distribute or use this message. If you have
>>>>>>>> received
>>>>>>>>>> this
>>>>>>>>>>>>>> communication in error, please notify the sender and delete
>>> all
>>>>>>>>>> copies
>>>>>>>>>>>>> of
>>>>>>>>>>>>>> this message. Accelerite, a Persistent Systems business does
>>> not
>>>>>>>>>>> accept
>>>>>>>>>>>>> any
>>>>>>>>>>>>>> liability for virus infected mails.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>
>
Re: Unable to create a New VM - Error message: Unable to start instance
due to Unable to get answer that is of class com.cloud.agent.api.StartAnswer
Posted by ilya <il...@gmail.com>.
as well as output of mount command..
On 4/14/16 10:39 PM, Nezar Madbouh wrote:
> Hi Ilya,
>
> Here is the output of mount command:
>
> [root@eqx-cs-cmp-01 ~]# mount
> /dev/mapper/vg_root-lv_root on / type ext4 (rw)
> proc on /proc type proc (rw)
> sysfs on /sys type sysfs (rw)
> devpts on /dev/pts type devpts (rw,gid=5,mode=620)
> tmpfs on /dev/shm type tmpfs (rw)
> /dev/sda1 on /boot type ext4 (rw)
> none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
> sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
> nfsd on /proc/fs/nfsd type nfsd (rw)
> 10.61.154.10:/Cloudstack/primary on
> /mnt/b3647480-5709-36eb-9ddd-e2c666f30b57 type nfs
> (rw,noatime,nfsvers=3,proto=tcp,hard,nolock,intr,addr=10.61.154.10)
> [root@eqx-cs-cmp-01 ~]#
>
>
> On 15 April 2016 at 15:22, Nezar Madbouh <ne...@ipscape.com.au>
> wrote:
>
>> Hi Ilya,
>>
>> Thanks for the instructions. I followed them and pasted the information on
>> pastebin:
>>
>> management-server logs job-14938
>> http://pastebin.com/GAiy8u5H
>>
>> Agent logs
>> http://pastebin.com/mfiADsJc
>>
>>
>> Your help is much appreciated.
>>
>> Nezar.
>>
>> On 15 April 2016 at 13:58, ilya <il...@gmail.com> wrote:
>>
>>> Here is how i would troubleshoot this:
>>>
>>> 1) ssh to MS host, run
>>>
>>> tail -f /var/cloudstack/management/management-server.log | grep -i
>>> StartVMCmdByAdmin
>>>
>>> 2) on cloudstack UI, press start on the VM and watch the tail log
>>>
>>> 3) on the tail log, you should see a StartVirtualMachine api call show
>>> up with job ID. For example:
>>>
>>>> 2016-04-15 03:41:57,193 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> (API-Job-Executor-24:ctx-7892387a job-291575) Executing AsyncJobVO
>>> {id:291575, userId: 157, accountId: 2, instanceType: VirtualMachine,
>>> instanceId: 7648, cmd:
>>> org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin, cmdInfo:
>>> {"id":"7648","response":"json","sessionkey":"mMJsduS9KLKApqH4dPpIlGZdt1Y\u003d","ctxDetails":"{\"com.cloud.vm.VirtualMachine\":7648,\"com.cloud.uservm.UserVm\":7648,\"com.cloud.network.router.VirtualRouter\":7648}","cmdEventType":"VM.START","expires":"2016-04-15T03:51:57+0000","ctxUserId":"157","signatureversion":"3","httpmethod":"GET","uuid":"7648","ctxAccountId":"2","ctxStartEventId":"314416"},
>>> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
>>> result: null, initMsid: 345051337894, completeMsid: null, lastUpdated:
>>> null, lastPolled: null, created: null}
>>>
>>> Each task in cloudstack spins up its own job, in this case its
>>> *job-291575*
>>>
>>> 4) now execute
>>>> grep -A3 -B3 job-291575
>>> /var/cloudstack/management/management-server.log | tee job-291575.log
>>>> grep job-291575 /var/cloudstack/management/management-server.log | grep
>>> via | head -1
>>>
>>> you should see a response like so,
>>>
>>>> 2016-04-15 03:41:59,259 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>> (Work-Job-Executor-42:ctx-b2c4af6b job-291575/job-291576 ctx-0988152e) Seq
>>> 441-4181310779036795138: Forwarding Seq 441-4181310779036795138: { Cmd ,
>>> MgmtId: 345051337894, via: 441(my-kvm-host01.example.com), Ver: v1,
>>> Flags: 100011,
>>> [{"com.cloud.agent.api.routing.DhcpEntryCommand":{"vmMac":"06:fe:4e:00:12:ad","vmIpAddress":"10.10.27.178","vmName":"ol6-test-ilya1","defaultRouter":"10.10.27.1","defaultDns":"10.17.27.166","duid":"00:03:00:01:06:fe:4e:00:12:ad","isDefault":true,"executeInSequence":false,"accessDetails":{"zone.network.type":"Advanced","router.guest.ip":"10.10.27.166","router.ip":"169.254.0.238","
>>> router.name":"r-7301-VM"},"wait":0}}] } to 345051337893
>>>
>>> We are looking for this line "via: 441(my-kvm-host01.example.com)"
>>>
>>> SSH to my-kvm-host01.example.com, and copy the
>>> /var/log/cloudstack/agent.log to pastebin
>>>
>>> also copy job-291575.log to pastebin
>>>
>>> I'll try to have a look, though when i get chance.
>>>
>>>
>>> Also paste the output of mount command on kvm host in question. I'm
>>> suspecting one of your storage pools maybe bad.
>>>
>>> Lastly, what distro are you running for KVM?
>>>
>>> Regards
>>> ilya
>>>
>>> On 4/14/16 5:05 PM, Nezar Madbouh wrote:
>>>> Hi Ilya,
>>>>
>>>> I shutdown all the VMs in Pre-prod Environmnet expecting the Pre-prod
>>>> Virtual router to be deleted automatically because no VM is running on
>>> this
>>>> vlan but it wasn't deleted.
>>>>
>>>> Am I right to expect the VR to be deleted automatically if no VM is
>>> running
>>>> in that vlan?
>>>>
>>>> Thanks,
>>>> Nezar.
>>>>
>>>> On 15 April 2016 at 09:53, Nezar Madbouh <ne...@ipscape.com.au>
>>>> wrote:
>>>>
>>>>> Hi Ilya,
>>>>>
>>>>> I have run the script and it didn't return any results:
>>>>>
>>>>> [root@eqx-cs-mgt-01 ~]# ./async_jobs.sh
>>>>> [root@eqx-cs-mgt-01 ~]#
>>>>>
>>>>>
>>>>> I also ran the command inside mysql:
>>>>>
>>>>> mysql> SELECT id, user_id, account_id, instance_type, instance_id,
>>>>> job_cmd, job_cmd_info, job_dispatcher, job_pending_signals FROM
>>>>> cloud.async_job where job_status=0 and job_dispatcher not like
>>>>> "pseudoJobDispatcher";
>>>>> Empty set (0.00 sec)
>>>>>
>>>>> your help is much appreciated,
>>>>> Nezar.
>>>>>
>>>>>
>>>>> On 14 April 2016 at 16:49, ilya <il...@gmail.com> wrote:
>>>>>
>>>>>> It appears you have pending jobs that are not letting you go through.
>>>>>>
>>>>>> Usually stop and starting management service should be enough.
>>>>>>
>>>>>> However, try running this script from your management server to show
>>> us
>>>>>> what you have running in async_job table.
>>>>>>
>>>>>>
>>>>>>
>>>>>>> #!/bin/bash
>>>>>>> DATESTAMP=$(date +%m%d%y-%H%M%S)
>>>>>>> DBPASS=$(java -classpath
>>>>>> /usr/share/cloudstack-common/lib/jasypt-1.9.0.jar
>>>>>> org.jasypt.intf.cli.JasyptPBEStringDecryptionCLI input="$(cat
>>>>>> /etc/cloudstack/management/db.properties | grep db.cloud.password |
>>> awk
>>>>>> -F'(' '{print $2}' | sed 's/)//g')" password="$(cat
>>>>>> /etc/cloudstack/management/key)" | grep -A2 OUTPUT | tail -1)
>>>>>>> DBHOST=$(cat /etc/cloudstack/management/db.properties | grep
>>>>>> db.cloud.host | awk -F'=' '{print $2}' | tail -1 )
>>>>>>> DBUSER=$(cat /etc/cloudstack/management/db.properties | grep
>>>>>> db.cloud.username | awk -F'=' '{print $2}')
>>>>>>> DB=$(cat /etc/cloudstack/management/db.properties | grep
>>> db.cloud.name
>>>>>> | awk -F'=' '{print $2}')
>>>>>>> DBPORT=$(cat /etc/cloudstack/management/db.properties | grep
>>>>>> db.cloud.port | awk -F'=' '{print $2}')
>>>>>>> MYSQLCMD="mysql -h $DBHOST -u $DBUSER -P $DBPORT -p$DBPASS $DB"
>>>>>>> echo 'SELECT id, user_id, account_id, instance_type, instance_id,
>>>>>> job_cmd, job_cmd_info, job_dispatcher, job_pending_signals FROM
>>>>>> cloud.async_job where job_status=0 and job_dispatcher not like
>>>>>> "pseudoJobDispatcher"' | $MYSQLCMD
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 4/13/16 5:16 PM, Nezar Madbouh wrote:
>>>>>>> Hi Simon,
>>>>>>>
>>>>>>> The following work was done but unfortunately the problem is not
>>> fixed:
>>>>>>>
>>>>>>>
>>>>>>> on management server:
>>>>>>>
>>>>>>> service cloudstack-management stop
>>>>>>>
>>>>>>>
>>>>>>> on the host:
>>>>>>>
>>>>>>> service cloudstack-agent stop
>>>>>>>
>>>>>>> service libvirtd restart
>>>>>>>
>>>>>>> service cloudstack-agent start
>>>>>>>
>>>>>>>
>>>>>>> on management server:
>>>>>>>
>>>>>>> service cloudstack-management start
>>>>>>>
>>>>>>> Regards,
>>>>>>> Nezar.
>>>>>>>
>>>>>>> On 24 March 2016 at 05:06, Simon Weller <sw...@ena.com> wrote:
>>>>>>>
>>>>>>>> Yes, I think that order is fine. Make sure the processes are indeed
>>>>>>>> shutdown before moving on to the next step.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ________________________________________
>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>>>>>>>> Sent: Tuesday, March 22, 2016 5:33 PM
>>>>>>>> To: users@cloudstack.apache.org
>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable to
>>> start
>>>>>>>> instance due to Unable to get answer that is of class
>>>>>>>> com.cloud.agent.api.StartAnswer
>>>>>>>>
>>>>>>>> Hi Simon,
>>>>>>>>
>>>>>>>> Can you please check if the following sequence of action is correct:
>>>>>>>>
>>>>>>>> on management server:
>>>>>>>>
>>>>>>>> service cloudstack-management stop
>>>>>>>>
>>>>>>>>
>>>>>>>> on the host:
>>>>>>>>
>>>>>>>> service cloudstack-agent stop
>>>>>>>>
>>>>>>>> service libvirtd restart
>>>>>>>>
>>>>>>>> service cloudstack-agent start
>>>>>>>>
>>>>>>>>
>>>>>>>> on management server:
>>>>>>>>
>>>>>>>> service cloudstack-management start
>>>>>>>>
>>>>>>>> Thanks very much for your help.
>>>>>>>> Nezar.
>>>>>>>>
>>>>>>>> On 23 March 2016 at 08:39, Simon Weller <sw...@ena.com> wrote:
>>>>>>>>
>>>>>>>>> Shutting down all the management services should clear the jobs.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ________________________________________
>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>>>>>>>>> Sent: Tuesday, March 22, 2016 4:28 PM
>>>>>>>>> To: users@cloudstack.apache.org
>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable to
>>>>>> start
>>>>>>>>> instance due to Unable to get answer that is of class
>>>>>>>>> com.cloud.agent.api.StartAnswer
>>>>>>>>>
>>>>>>>>> Hi Simon,
>>>>>>>>>
>>>>>>>>> I will plan to restart agents, libvirtd and management services.
>>>>>>>>>
>>>>>>>>> How do I clear hung jobs.
>>>>>>>>>
>>>>>>>>> Your help is much appreciated,
>>>>>>>>> Nezar.
>>>>>>>>>
>>>>>>>>> On 22 March 2016 at 23:03, Simon Weller <sw...@ena.com> wrote:
>>>>>>>>>
>>>>>>>>>> Nezar,
>>>>>>>>>>
>>>>>>>>>> I've looked through the code related to your release and it seems
>>>>>> that
>>>>>>>> in
>>>>>>>>>> order to throw this exception, 10 attempts have been made to
>>> retrieve
>>>>>>>> the
>>>>>>>>>> disk information prior to the exception being thrown.
>>>>>>>>>>
>>>>>>>>>> It appears there is something really odd going on here. I'm
>>> starting
>>>>>>>> to
>>>>>>>>>> wonder whether it might be worth while for you to stop all of your
>>>>>>>>> agents,
>>>>>>>>>> plus management server processes, restart libvirtd on each of the
>>>>>> hosts
>>>>>>>>> and
>>>>>>>>>> then bring the agents back up prior to starting the management
>>>>>> servers.
>>>>>>>>>>
>>>>>>>>>> There are a bunch of logs the pastebin you posted that seems to
>>>>>>>> indicate
>>>>>>>>>> you have some jobs hung, for example:
>>>>>>>>>>
>>>>>>>>>> 2016-03-14 21:45:31,828 WARN [o.a.c.f.j.i.AsyncJobMonitor]
>>>>>>>>>> (Timer-1:ctx-0c090bd7) Task (job-9066) has been pending for
>>> 1660851
>>>>>>>>> seconds
>>>>>>>>>> 2016-03-14 21:45:31,828 WARN [o.a.c.f.j.i.AsyncJobMonitor]
>>>>>>>>>> (Timer-1:ctx-0c090bd7) Task (job-8133) has been pending for
>>> 4253003
>>>>>>>>> seconds
>>>>>>>>>>
>>>>>>>>>> Restarting the agents, libvirtd and management will not cause an
>>>>>>>> outage
>>>>>>>>>> to your cloud, but I would suggest, if you decided to do this, you
>>>>>>>> should
>>>>>>>>>> complete it in a maintenance window.
>>>>>>>>>>
>>>>>>>>>> - Si
>>>>>>>>>>
>>>>>>>>>> ________________________________________
>>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>>>>>>>>>> Sent: Sunday, March 20, 2016 10:27 PM
>>>>>>>>>> To: users@cloudstack.apache.org
>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable to
>>>>>> start
>>>>>>>>>> instance due to Unable to get answer that is of class
>>>>>>>>>> com.cloud.agent.api.StartAnswer
>>>>>>>>>>
>>>>>>>>>> Hi Simon,
>>>>>>>>>>
>>>>>>>>>> Please see answers to your questions below. I am not sure if I
>>>>>> provided
>>>>>>>>> the
>>>>>>>>>> correct answer for the primary and secondary storage
>>> configuration.
>>>>>>>>> Please
>>>>>>>>>> let me know if this is not what you are after:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Can you screenshot and post somewhere your secondary storage
>>>>>>>>> configuration
>>>>>>>>>> and your primary storage configuration from the UI with the
>>> details?
>>>>>>>>>>
>>>>>>>>>> mysql> select * from configuration where name like '%secondary%'\G
>>>>>>>>>> *************************** 1. row ***************************
>>>>>>>>>> category: Account Defaults
>>>>>>>>>> instance: DEFAULT
>>>>>>>>>> component: management-server
>>>>>>>>>> name: max.account.secondary.storage
>>>>>>>>>> value: 400
>>>>>>>>>> description: The default maximum secondary storage space (in
>>> GiB)
>>>>>>>> that
>>>>>>>>>> can be used for an account
>>>>>>>>>> default_value: 400
>>>>>>>>>> updated: NULL
>>>>>>>>>> scope: NULL
>>>>>>>>>> is_dynamic: 0
>>>>>>>>>> *************************** 2. row ***************************
>>>>>>>>>> category: Project Defaults
>>>>>>>>>> instance: DEFAULT
>>>>>>>>>> component: management-server
>>>>>>>>>> name: max.project.secondary.storage
>>>>>>>>>> value: 400
>>>>>>>>>> description: The default maximum secondary storage space (in
>>> GiB)
>>>>>>>> that
>>>>>>>>>> can be used for a project
>>>>>>>>>> default_value: 400
>>>>>>>>>> updated: NULL
>>>>>>>>>> scope: NULL
>>>>>>>>>> is_dynamic: 0
>>>>>>>>>> *************************** 3. row ***************************
>>>>>>>>>> category: Hidden
>>>>>>>>>> instance: DEFAULT
>>>>>>>>>> component: management-server
>>>>>>>>>> name: secondary.storage.vm
>>>>>>>>>> value: +rwRhJOXPymbQOuAMLnhiw==
>>>>>>>>>> description: Deploys a VM per zone to manage secondary storage
>>> if
>>>>>>>> true,
>>>>>>>>>> otherwise secondary storage is mounted on management server
>>>>>>>>>> default_value: NULL
>>>>>>>>>> updated: NULL
>>>>>>>>>> scope: NULL
>>>>>>>>>> is_dynamic: 0
>>>>>>>>>> *************************** 4. row ***************************
>>>>>>>>>> category: Advanced
>>>>>>>>>> instance: DEFAULT
>>>>>>>>>> component: ExtensionRegistry
>>>>>>>>>> name: secondary.storage.vm.allocators.exclude
>>>>>>>>>> value: NULL
>>>>>>>>>> description: Extensions to exclude from being registered
>>>>>>>>>> default_value: NULL
>>>>>>>>>> updated: 2014-12-04 00:18:15
>>>>>>>>>> scope: Global
>>>>>>>>>> is_dynamic: 0
>>>>>>>>>> *************************** 5. row ***************************
>>>>>>>>>> category: NetworkManager
>>>>>>>>>> instance: DEFAULT
>>>>>>>>>> component: management-server
>>>>>>>>>> name: vm.network.nic.max.secondary.ipaddresses
>>>>>>>>>> value: NULL
>>>>>>>>>> description: Specify the number of secondary ip addresses per
>>> nic
>>>>>> per
>>>>>>>>> vm
>>>>>>>>>> default_value: 256
>>>>>>>>>> updated: NULL
>>>>>>>>>> scope: NULL
>>>>>>>>>> is_dynamic: 0
>>>>>>>>>> 5 rows in set (0.00 sec)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> mysql> select * from configuration where name like '%primary%'\G
>>>>>>>>>> *************************** 1. row ***************************
>>>>>>>>>> category: Account Defaults
>>>>>>>>>> instance: DEFAULT
>>>>>>>>>> component: management-server
>>>>>>>>>> name: max.account.primary.storage
>>>>>>>>>> value: 200
>>>>>>>>>> description: The default maximum primary storage space (in GiB)
>>>>>> that
>>>>>>>>> can
>>>>>>>>>> be used for an account
>>>>>>>>>> default_value: 200
>>>>>>>>>> updated: NULL
>>>>>>>>>> scope: NULL
>>>>>>>>>> is_dynamic: 0
>>>>>>>>>> *************************** 2. row ***************************
>>>>>>>>>> category: Project Defaults
>>>>>>>>>> instance: DEFAULT
>>>>>>>>>> component: management-server
>>>>>>>>>> name: max.project.primary.storage
>>>>>>>>>> value: 200
>>>>>>>>>> description: The default maximum primary storage space (in GiB)
>>>>>> that
>>>>>>>>> can
>>>>>>>>>> be used for a project
>>>>>>>>>> default_value: 200
>>>>>>>>>> updated: NULL
>>>>>>>>>> scope: NULL
>>>>>>>>>> is_dynamic: 0
>>>>>>>>>> *************************** 3. row ***************************
>>>>>>>>>> category: Storage
>>>>>>>>>> instance: DEFAULT
>>>>>>>>>> component: TemplateManager
>>>>>>>>>> name: primary.storage.download.wait
>>>>>>>>>> value: 10800
>>>>>>>>>> description: In second, timeout for download template to primary
>>>>>>>>> storage
>>>>>>>>>> default_value: NULL
>>>>>>>>>> updated: NULL
>>>>>>>>>> scope: NULL
>>>>>>>>>> is_dynamic: 0
>>>>>>>>>> 3 rows in set (0.00 sec)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Also, what is the Cloudstack version you are running and what is
>>> the
>>>>>> OS
>>>>>>>>>> you're running it on?
>>>>>>>>>>
>>>>>>>>>> Cloudstack 4.4.1
>>>>>>>>>>
>>>>>>>>>> Centos 6.6
>>>>>>>>>>
>>>>>>>>>> Are you using actimeo=0 in your NFS mount line?
>>>>>>>>>>
>>>>>>>>>> Primary storage on Hypervisors has the following parameters
>>>>>>>>>> 10.61.154.10:/Cloudstack/primary on
>>>>>>>>>> /mnt/b3647480-5709-36eb-9ddd-e2c666f30b57 type nfs
>>>>>>>>>>
>>> (rw,noatime,nfsvers=3,proto=tcp,hard,nolock,intr,addr=10.61.154.10)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Secondary storage on SSVM has the following parameters
>>>>>>>>>> 10.61.154.10:/Cloudstack/secondary on
>>>>>>>>>> /mnt/SecStorage/d11d5c92-6950-30e8-a0b5-3cd46a17ec14 type nfs4
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>> (rw,relatime,vers=4,rsize=262144,wsize=262144,namlen=255,acdirmin=0,acdirmax=0,soft,proto=tcp,port=0,timeo=133,retrans=
>>>>>> 2147483647
>>>>>>
>>> ,sec=sys,clientaddr=10.61.154.70,minorversion=0,local_lock=none,addr=10.61.154.10)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thanks & Regards,
>>>>>>>>>> Nezar.
>>>>>>>>>>
>>>>>>>>>> On 18 March 2016 at 23:50, Simon Weller <sw...@ena.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Also, what is the Cloudstack version you are running and what is
>>> the
>>>>>>>> OS
>>>>>>>>>>> you're running it on?
>>>>>>>>>>>
>>>>>>>>>>> Are you using actimeo=0 in your NFS mount line?
>>>>>>>>>>>
>>>>>>>>>>> ________________________________________
>>>>>>>>>>> From: Simon Weller <sw...@ena.com>
>>>>>>>>>>> Sent: Friday, March 18, 2016 7:40 AM
>>>>>>>>>>> To: users@cloudstack.apache.org
>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable to
>>>>>>>> start
>>>>>>>>>>> instance due to Unable to get answer that is of class
>>>>>>>>>>> com.cloud.agent.api.StartAnswer
>>>>>>>>>>>
>>>>>>>>>>> Nexar,
>>>>>>>>>>>
>>>>>>>>>>> That's normal behaviour. You can only ssh to the link local on
>>> the
>>>>>>>> host
>>>>>>>>>>> where the VM resides.
>>>>>>>>>>>
>>>>>>>>>>> Can you screenshot and post somewhere your secondary storage
>>>>>>>>>> configuration
>>>>>>>>>>> and your primary storage configuration from the UI with the
>>> details?
>>>>>>>>>>>
>>>>>>>>>>> - Si
>>>>>>>>>>>
>>>>>>>>>>> ________________________________________
>>>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>>>>>>>>>>> Sent: Thursday, March 17, 2016 8:19 PM
>>>>>>>>>>> To: users@cloudstack.apache.org
>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable to
>>>>>>>> start
>>>>>>>>>>> instance due to Unable to get answer that is of class
>>>>>>>>>>> com.cloud.agent.api.StartAnswer
>>>>>>>>>>>
>>>>>>>>>>> Hi Simon,
>>>>>>>>>>>
>>>>>>>>>>> I just noticed that from other Hosts I can't ssh into the SSVM:
>>>>>>>>>>>
>>>>>>>>>>> [root@eqx-cs-cmp-01 ~]# ssh -i /root/.ssh/id_rsa.cloud -p 3922
>>>>>>>>>>> root@169.254.3.161
>>>>>>>>>>> ssh: connect to host 169.254.3.161 port 3922: No route to host
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I can only ssh from one Host:
>>>>>>>>>>> [root@eqx-cs-cmp-06 ~]# ssh -i /root/.ssh/id_rsa.cloud -p 3922
>>>>>>>>>>> root@169.254.3.161
>>>>>>>>>>> Linux s-11-VM 3.2.0-4-amd64 #1 SMP Debian 3.2.57-3 x86_64
>>>>>>>>>>>
>>>>>>>>>>> Thanks & Regards,
>>>>>>>>>>> Nezar.
>>>>>>>>>>>
>>>>>>>>>>> On 18 March 2016 at 10:36, Nezar Madbouh <
>>>>>>>> nezar.madbouh@ipscape.com.au
>>>>>>>>>>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Simon,
>>>>>>>>>>>>
>>>>>>>>>>>> Yes existing VMs are successfully running. Yesterday I was able
>>> to
>>>>>>>>>> create
>>>>>>>>>>>> a VM in Dev Environment but I can't create VMs in Prod
>>> environment.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Nezar.
>>>>>>>>>>>>
>>>>>>>>>>>> On 18 March 2016 at 09:12, Simon Weller <sw...@ena.com>
>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> I'm assuming you have other VMs successfully running, correct?
>>> So
>>>>>>>>> you
>>>>>>>>>>> can
>>>>>>>>>>>>> write to your primary storage.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> ________________________________________
>>>>>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>>>>>>>>>>>>> Sent: Thursday, March 17, 2016 4:43 PM
>>>>>>>>>>>>> To: users@cloudstack.apache.org
>>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable
>>> to
>>>>>>>>>> start
>>>>>>>>>>>>> instance due to Unable to get answer that is of class
>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Simon and Shweta,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Please see the result below:
>>>>>>>>>>>>>
>>>>>>>>>>>>> root@s-11-VM:~# /usr/local/cloud/systemvm/ssvm-check.sh
>>>>>>>>>>>>> ================================================
>>>>>>>>>>>>> First DNS server is 8.8.8.8
>>>>>>>>>>>>> PING 8.8.8.8 (8.8.8.8): 48 data bytes
>>>>>>>>>>>>> 56 bytes from 8.8.8.8: icmp_seq=0 ttl=57 time=5.209 ms
>>>>>>>>>>>>> 56 bytes from 8.8.8.8: icmp_seq=1 ttl=57 time=0.936 ms
>>>>>>>>>>>>> --- 8.8.8.8 ping statistics ---
>>>>>>>>>>>>> 2 packets transmitted, 2 packets received, 0% packet loss
>>>>>>>>>>>>> round-trip min/avg/max/stddev = 0.936/3.072/5.209/2.137 ms
>>>>>>>>>>>>> Good: Can ping DNS server
>>>>>>>>>>>>> ================================================
>>>>>>>>>>>>> Good: DNS resolves download.cloud.com
>>>>>>>>>>>>> ================================================
>>>>>>>>>>>>> nfs is currently mounted
>>>>>>>>>>>>> Mount point is
>>>>>>>> /mnt/SecStorage/d11d5c92-6950-30e8-a0b5-3cd46a17ec14
>>>>>>>>>>>>> Good: Can write to mount point
>>>>>>>>>>>>> ================================================
>>>>>>>>>>>>> Management server is 10.61.153.20. Checking connectivity.
>>>>>>>>>>>>> Good: Can connect to management server port 8250
>>>>>>>>>>>>> ================================================
>>>>>>>>>>>>> Good: Java process is running
>>>>>>>>>>>>> ================================================
>>>>>>>>>>>>> Tests Complete. Look for ERROR or WARNING above.
>>>>>>>>>>>>> root@s-11-VM:~#
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks & Regards,
>>>>>>>>>>>>> Nezar.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 17 March 2016 at 09:42, Simon Weller <sw...@ena.com>
>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Nezar,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Log into the Cloudstack UI.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Click on "Infrastructure" on the left menu. Click on View All
>>>>>>>>> below
>>>>>>>>>>>>>> "System VMs". find the item marked "Secondary Storage VM" and
>>>>>>>>> then
>>>>>>>>>>>>> click
>>>>>>>>>>>>>> on the Name of that VM on the left (it will be s-XXXX-VM).
>>>>>>>> You'll
>>>>>>>>>> see
>>>>>>>>>>> a
>>>>>>>>>>>>>> bunch of information about the VM. Find the Host line and also
>>>>>>>> the
>>>>>>>>>>> Link
>>>>>>>>>>>>>> Local IP Address line and then follow the original
>>> instructions
>>>>>>>> in
>>>>>>>>>> my
>>>>>>>>>>>>>> previous email.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> - Si
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ________________________________________
>>>>>>>>>>>>>> From: Shweta Agarwal <sh...@accelerite.com>
>>>>>>>>>>>>>> Sent: Wednesday, March 16, 2016 12:16 AM
>>>>>>>>>>>>>> To: users@cloudstack.apache.org
>>>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable
>>>>>>>> to
>>>>>>>>>>> start
>>>>>>>>>>>>>> instance due to Unable to get answer that is of class
>>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Nezar,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> You can find host name on detail view of SSVM in UI .
>>>>>>>>>>>>>> You can also use list api command listSystemVms to get the
>>> host
>>>>>>>>>>> details
>>>>>>>>>>>>> of
>>>>>>>>>>>>>> system vms.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>> Shweta
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ________________________________________
>>>>>>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>>>>>>>>>>>>>> Sent: Wednesday, March 16, 2016 9:58 AM
>>>>>>>>>>>>>> To: users@cloudstack.apache.org
>>>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable
>>>>>>>> to
>>>>>>>>>>> start
>>>>>>>>>>>>>> instance due to Unable to get answer that is of class
>>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Simon,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> How can I find out which host the SSVM is on?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks & Regards,
>>>>>>>>>>>>>> Nezar.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 16 March 2016 at 12:28, Simon Weller <sw...@ena.com>
>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Can you ssh into the Secondary Storage VM?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> This can be done by getting the link-local-ip address from
>>> the
>>>>>>>>>>>>>>> infrastructure menu in the UI and then sshing into the host
>>>>>>>> the
>>>>>>>>>> SSVM
>>>>>>>>>>>>> is
>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>> and issuing this command as root: ssh -i
>>>>>>>> /root/.ssh/id_rsa.cloud
>>>>>>>>>> -p
>>>>>>>>>>>>> 3922
>>>>>>>>>>>>>>> root@link-local-ip (replace link-local-ip with the link
>>>>>>>> local
>>>>>>>>>> ip).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Try running the SSVM diangnostic script:
>>>>>>>>>>>>>>> usr/local/cloud/systemvm/ssvm-check.sh
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> - Si
>>>>>>>>>>>>>>> ________________________________________
>>>>>>>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>>>>>>>>>>>>>>> Sent: Tuesday, March 15, 2016 7:49 PM
>>>>>>>>>>>>>>> To: users@cloudstack.apache.org
>>>>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message:
>>> Unable
>>>>>>>>> to
>>>>>>>>>>>>> start
>>>>>>>>>>>>>>> instance due to Unable to get answer that is of class
>>>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Simon,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Please see details below:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> mysql> use cloud; select id, name, status, type from host
>>>>>>>> where
>>>>>>>>>> id =
>>>>>>>>>>>>> 5;
>>>>>>>>>>>>>>> Reading table information for completion of table and column
>>>>>>>>> names
>>>>>>>>>>>>>>> You can turn off this feature to get a quicker startup with
>>> -A
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Database changed
>>>>>>>>>>>>>>> +----+------------------------------+--------+---------+
>>>>>>>>>>>>>>> | id | name | status | type |
>>>>>>>>>>>>>>> +----+------------------------------+--------+---------+
>>>>>>>>>>>>>>> | 5 | eqx-cs-cmp-02.ipscape.com.au | up | routing |
>>>>>>>>>>>>>>> +----+------------------------------+--------+---------+
>>>>>>>>>>>>>>> 1 row in set (0.00 sec)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> cat /var/log/cloudstack/agent/agent.log
>>>>>>>>>>>>>>> com.cloud.utils.exception.CloudRuntimeException:
>>>>>>>>>>>>>>> com.cloud.utils.exception.CloudRuntimeException: Can't find
>>>>>>>>>>>>>>> volume:089ff3ca-9bff-402c-a14e-ea7a35aaad5c
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>> com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.getPhysicalDisk(KVMStoragePoolManager.java:262)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.createVbd(LibvirtComputingResource.java:3921)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(LibvirtComputingResource.java:3789)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1333)
>>>>>>>>>>>>>>> at
>>>>>>>> com.cloud.agent.Agent.processRequest(Agent.java:501)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>
>>> com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:808)
>>>>>>>>>>>>>>> at com.cloud.utils.nio.Task.run(Task.java:84)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>>>>>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>>>>>>>>>> agent.log.2016-03-03.gz (END)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [root@eqx-cs-cmp-02 agent]# virsh pool-list
>>>>>>>>>>>>>>> Name State Autostart
>>>>>>>>>>>>>>> -----------------------------------------
>>>>>>>>>>>>>>> 2d0967f8-50d6-4d6d-87b7-4da0596f6924 active no
>>>>>>>>>>>>>>> b3647480-5709-36eb-9ddd-e2c666f30b57 active no
>>>>>>>>>>>>>>> e3831246-ff55-4af4-9964-a17565edaccc active no
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [Root @ eqx-cs-cmp-02 agent] # df -h
>>>>>>>>>>>>>>> Filesystem Size Used Avail Use% Mounted on
>>>>>>>>>>>>>>> /dev/mapper/vg_root-lv_root
>>>>>>>>>>>>>>> 203G 215G 1.3G 1% /
>>>>>>>>>>>>>>> tmpfs 48g 48g 0 0% / dev / shm
>>>>>>>>>>>>>>> / Dev / sda1 477m 427m 25M 6% / boot
>>>>>>>>>>>>>>> xx.xx.xx.xx:/Cloudstack/primary
>>>>>>>>>>>>>>> 7.6T 4.1T 3.5T 54%
>>>>>>>>>>>>>>> /mnt/b3647480-5709-36eb-9ddd-e2c666f30b57
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I also noticed the following in cloudstack-agent.err
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> libvirt: QEMU Driver error : Domain not found: no domain with
>>>>>>>>>>> matching
>>>>>>>>>>>>>> name
>>>>>>>>>>>>>>> 'i-11-600-VM'
>>>>>>>>>>>>>>> libvirt: Storage Driver error : Storage volume not found: no
>>>>>>>>>> storage
>>>>>>>>>>>>> vol
>>>>>>>>>>>>>>> with matching name '089ff3ca-9bff-402c-a14e-ea7a35aaad5c'
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Your help is much appreciated,
>>>>>>>>>>>>>>> Nezar
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 16 March 2016 at 11:03, Simon Weller <sw...@ena.com>
>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Nezar,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> ok, I just took a look at a pastebin you posted.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I see it complaining about the inability to find a volume
>>>>>>>>> prior
>>>>>>>>>> to
>>>>>>>>>>>>>>>> throwing the host exception.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Firstly, let's find out who host 5 is. Can you log into your
>>>>>>>>>> MySQL
>>>>>>>>>>>>>> server
>>>>>>>>>>>>>>>> and run this:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> use cloud;select id,name,status,type from host where id=5;
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Can you post the results?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> If it's a host, can you post the
>>>>>>>>>>> /var/log/cloudstack/agent/agent.log
>>>>>>>>>>>>>> from
>>>>>>>>>>>>>>>> that host?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> And also run virsh pool-list on the host in question?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> - Si
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> ________________________________________
>>>>>>>>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>>>>>>>>>>>>>>>> Sent: Tuesday, March 15, 2016 6:11 PM
>>>>>>>>>>>>>>>> To: users@cloudstack.apache.org
>>>>>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message:
>>>>>>>> Unable
>>>>>>>>>> to
>>>>>>>>>>>>> start
>>>>>>>>>>>>>>>> instance due to Unable to get answer that is of class
>>>>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi Simon,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> When you say "double check your routers tied to the
>>>>>>>> network",
>>>>>>>>> do
>>>>>>>>>>> you
>>>>>>>>>>>>>> mean
>>>>>>>>>>>>>>>> the physical routers or the virtual routers?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> When I try to create a VM in Prod Environment it fails. I
>>>>>>>>> can't
>>>>>>>>>>> ping
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> Prod virtual router from management server or Host.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> But I can ping Dev virtual router from management server and
>>>>>>>>>> Host
>>>>>>>>>>>>> and I
>>>>>>>>>>>>>>> was
>>>>>>>>>>>>>>>> able to create a VM in Dev Environment.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> How can I troubleshoot the Prod Virtual router. I launched
>>>>>>>> the
>>>>>>>>>>>>> console
>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>> couldn't login.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Please note that we need to take into consideration that the
>>>>>>>>>>>>> affected
>>>>>>>>>>>>>>>> network is the Production network and we need to minimise
>>>>>>>>>> service
>>>>>>>>>>>>>> outage
>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>> much as possible.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Your help is much appreciated,
>>>>>>>>>>>>>>>> Nezar
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On 16 March 2016 at 08:37, Simon Weller <sw...@ena.com>
>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I've seen this before where a Virtual Router has crashed
>>>>>>>> and
>>>>>>>>>> the
>>>>>>>>>>>>>>>>> management server can't communicate with it.
>>>>>>>>>>>>>>>>> Are you using an advanced network setup?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> If so, double check your routers tied to the network where
>>>>>>>>> you
>>>>>>>>>>> are
>>>>>>>>>>>>>>>>> attempting to start an instance.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> - Si
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> ________________________________________
>>>>>>>>>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>>>>>>>>>>>>>>>>> Sent: Tuesday, March 15, 2016 4:02 PM
>>>>>>>>>>>>>>>>> To: users@cloudstack.apache.org
>>>>>>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message:
>>>>>>>>> Unable
>>>>>>>>>>> to
>>>>>>>>>>>>>> start
>>>>>>>>>>>>>>>>> instance due to Unable to get answer that is of class
>>>>>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi Glenn,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> We have 16 hosts, all KVM.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> The error is not specific to Host 5. Every time I try to
>>>>>>>>>> create
>>>>>>>>>>> a
>>>>>>>>>>>>> VM
>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>> gives me the same error message but on different host.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> How do I restart the agent on the host. Is it service
>>>>>>>>>> affecting?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks very much for your help.
>>>>>>>>>>>>>>>>> Neza
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On 15 March 2016 at 17:37, Glenn Wagner <
>>>>>>>>>>>>> glenn.wagner@shapeblue.com>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Can you explain your environment first , how many hosts
>>>>>>>> ,
>>>>>>>>>> what
>>>>>>>>>>>>> is
>>>>>>>>>>>>>>> your
>>>>>>>>>>>>>>>>>> hypervisors (KVM,Xenserver,VMware)
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> The error seems to be with your Host 5 Hypervisor
>>>>>>>>>>>>>>>>>> Resource [Host:5] is unreachable:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Go to the infrastructure section and see what host5 is
>>>>>>>>>>>>> reporting,
>>>>>>>>>>>>>>>>>> Have you tried to restart the agent on the Host?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>> Glenn
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> [image: ShapeBlue] <http://www.shapeblue.com>
>>>>>>>>>>>>>>>>>> Glenn Wagner
>>>>>>>>>>>>>>>>>> Senior Consultant , ShapeBlue
>>>>>>>>>>>>>>>>>> d: * | s: +27 21 527 0091*
>>>>>>>>>>>>> <%7C%20s:%20+27%2021%20527%200091> |
>>>>>>>>>>>>>>> m:
>>>>>>>>>>>>>>>>>> *+27 73 917 4111* <+27%2073%20917%204111>
>>>>>>>>>>>>>>>>>> e: *glenn.wagner@shapeblue.com | t: *
>>>>>>>>>>>>>>>>>> <glenn.wagner@shapeblue.com%20%7C%20t:> | w: *
>>>>>>>>>>>>> www.shapeblue.com
>>>>>>>>>>>>>> *
>>>>>>>>>>>>>>>>>> <http://www.shapeblue.com>
>>>>>>>>>>>>>>>>>> a:
>>>>>>>>>>>>>>>>>> 2nd Floor, Oudehuis Centre, 122 Main Rd, Somerset West
>>>>>>>>> Cape
>>>>>>>>>>> Town
>>>>>>>>>>>>>> 7130
>>>>>>>>>>>>>>>>> South Africa
>>>>>>>>>>>>>>>>>> Shape Blue Ltd is a company incorporated in England &
>>>>>>>>> Wales.
>>>>>>>>>>>>>>> ShapeBlue
>>>>>>>>>>>>>>>>>> Services India LLP is a company incorporated in India
>>>>>>>> and
>>>>>>>>> is
>>>>>>>>>>>>>> operated
>>>>>>>>>>>>>>>>> under
>>>>>>>>>>>>>>>>>> license from Shape Blue Ltd. Shape Blue Brasil
>>>>>>>> Consultoria
>>>>>>>>>>> Ltda
>>>>>>>>>>>>> is
>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>> company incorporated in Brasil and is operated under
>>>>>>>>> license
>>>>>>>>>>>>> from
>>>>>>>>>>>>>>> Shape
>>>>>>>>>>>>>>>>>> Blue Ltd. ShapeBlue SA Pty Ltd is a company registered
>>>>>>>> by
>>>>>>>>>> The
>>>>>>>>>>>>>>> Republic
>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>> South Africa and is traded under license from Shape Blue
>>>>>>>>>> Ltd.
>>>>>>>>>>>>>>> ShapeBlue
>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>> a registered trademark.
>>>>>>>>>>>>>>>>>> This email and any attachments to it may be confidential
>>>>>>>>> and
>>>>>>>>>>> are
>>>>>>>>>>>>>>>> intended
>>>>>>>>>>>>>>>>>> solely for the use of the individual to whom it is
>>>>>>>>>> addressed.
>>>>>>>>>>>>> Any
>>>>>>>>>>>>>>> views
>>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>> opinions expressed are solely those of the author and do
>>>>>>>>> not
>>>>>>>>>>>>>>>> necessarily
>>>>>>>>>>>>>>>>>> represent those of Shape Blue Ltd or related companies.
>>>>>>>> If
>>>>>>>>>> you
>>>>>>>>>>>>> are
>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> intended recipient of this email, you must neither take
>>>>>>>>> any
>>>>>>>>>>>>> action
>>>>>>>>>>>>>>>> based
>>>>>>>>>>>>>>>>>> upon its contents, nor copy or show it to anyone. Please
>>>>>>>>>>> contact
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> sender
>>>>>>>>>>>>>>>>>> if you believe you have received this email in error.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>>>>>>>>> From: Nezar Madbouh [mailto:
>>>>>>>> nezar.madbouh@ipscape.com.au]
>>>>>>>>>>>>>>>>>> Sent: Tuesday, 15 March 2016 5:45 AM
>>>>>>>>>>>>>>>>>> To: users@cloudstack.apache.org
>>>>>>>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message:
>>>>>>>>>> Unable
>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> start
>>>>>>>>>>>>>>>>>> instance due to Unable to get answer that is of class
>>>>>>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi Jeremy,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I have uploaded the contents to
>>>>>>>>>> http://pastebin.com/1J3mmgTh
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> your help is much appreciated,
>>>>>>>>>>>>>>>>>> Nezar.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On 15 March 2016 at 13:46, Jeremy Peterson <
>>>>>>>>>>>>> jpeterson@acentek.net>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I don't see any attachments can you create a pastebin.
>>>>>>>>>>> That's
>>>>>>>>>>>>>>> easier
>>>>>>>>>>>>>>>>>>> to look through as I think attachments are removed
>>>>>>>>> before
>>>>>>>>>>>>> being
>>>>>>>>>>>>>>> sent
>>>>>>>>>>>>>>>>>>> to user group.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Jeremy
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> -------- Original message --------
>>>>>>>>>>>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>>>>>>>>>>>>>>>>>>> Date: 3/14/2016 6:38 PM (GMT-06:00)
>>>>>>>>>>>>>>>>>>> To: users@cloudstack.apache.org
>>>>>>>>>>>>>>>>>>> Subject: Unable to create a New VM - Error message:
>>>>>>>>> Unable
>>>>>>>>>>> to
>>>>>>>>>>>>>> start
>>>>>>>>>>>>>>>>>>> instance due to Unable to get answer that is of class
>>>>>>>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi Shweta and Sadhu,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Please find attached managerment-server.log which
>>>>>>>>> contains
>>>>>>>>>>> the
>>>>>>>>>>>>>> logs
>>>>>>>>>>>>>>>>>>> since I started creating a new VM (VM id 600).
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Please note that we don't have agent.log files
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Your help is much appreciated.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>> Nezar.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Find out more about ShapeBlue and our range of
>>>>>>>> CloudStack
>>>>>>>>>>>>> related
>>>>>>>>>>>>>>>>> services:
>>>>>>>>>>>>>>>>>> IaaS Cloud Design & Build
>>>>>>>>>>>>>>>>>> <http://shapeblue.com/iaas-cloud-design-and-build//> |
>>>>>>>>>>> CSForge
>>>>>>>>>>>>> -
>>>>>>>>>>>>>>> rapid
>>>>>>>>>>>>>>>>>> IaaS deployment framework <
>>>>>>>> http://shapeblue.com/csforge/>
>>>>>>>>>>>>>>>>>> CloudStack Consulting <
>>>>>>>>>>>>>> http://shapeblue.com/cloudstack-consultancy/>
>>>>>>>>>>>>>>> |
>>>>>>>>>>>>>>>>> CloudStack
>>>>>>>>>>>>>>>>>> Software Engineering
>>>>>>>>>>>>>>>>>> <http://shapeblue.com/cloudstack-software-engineering/>
>>>>>>>>>>>>>>>>>> CloudStack Infrastructure Support
>>>>>>>>>>>>>>>>>> <
>>>>>>>> http://shapeblue.com/cloudstack-infrastructure-support/>
>>>>>>>>> |
>>>>>>>>>>>>>>> CloudStack
>>>>>>>>>>>>>>>>>> Bootcamp Training Courses <
>>>>>>>>>>>>>> http://shapeblue.com/cloudstack-training/
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> DISCLAIMER
>>>>>>>>>>>>>> ==========
>>>>>>>>>>>>>> This e-mail may contain privileged and confidential
>>> information
>>>>>>>>>> which
>>>>>>>>>>> is
>>>>>>>>>>>>>> the property of Accelerite, a Persistent Systems business. It
>>> is
>>>>>>>>>>>>> intended
>>>>>>>>>>>>>> only for the use of the individual or entity to which it is
>>>>>>>>>> addressed.
>>>>>>>>>>>>> If
>>>>>>>>>>>>>> you are not the intended recipient, you are not authorized to
>>>>>>>>> read,
>>>>>>>>>>>>> retain,
>>>>>>>>>>>>>> copy, print, distribute or use this message. If you have
>>>>>>>> received
>>>>>>>>>> this
>>>>>>>>>>>>>> communication in error, please notify the sender and delete
>>> all
>>>>>>>>>> copies
>>>>>>>>>>>>> of
>>>>>>>>>>>>>> this message. Accelerite, a Persistent Systems business does
>>> not
>>>>>>>>>>> accept
>>>>>>>>>>>>> any
>>>>>>>>>>>>>> liability for virus infected mails.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>
>
Re: Unable to create a New VM - Error message: Unable to start
instance due to Unable to get answer that is of class com.cloud.agent.api.StartAnswer
Posted by Nezar Madbouh <ne...@ipscape.com.au>.
Hi Ilya,
Here is the output of mount command:
[root@eqx-cs-cmp-01 ~]# mount
/dev/mapper/vg_root-lv_root on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
/dev/sda1 on /boot type ext4 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
nfsd on /proc/fs/nfsd type nfsd (rw)
10.61.154.10:/Cloudstack/primary on
/mnt/b3647480-5709-36eb-9ddd-e2c666f30b57 type nfs
(rw,noatime,nfsvers=3,proto=tcp,hard,nolock,intr,addr=10.61.154.10)
[root@eqx-cs-cmp-01 ~]#
On 15 April 2016 at 15:22, Nezar Madbouh <ne...@ipscape.com.au>
wrote:
> Hi Ilya,
>
> Thanks for the instructions. I followed them and pasted the information on
> pastebin:
>
> management-server logs job-14938
> http://pastebin.com/GAiy8u5H
>
> Agent logs
> http://pastebin.com/mfiADsJc
>
>
> Your help is much appreciated.
>
> Nezar.
>
> On 15 April 2016 at 13:58, ilya <il...@gmail.com> wrote:
>
>> Here is how i would troubleshoot this:
>>
>> 1) ssh to MS host, run
>>
>> tail -f /var/cloudstack/management/management-server.log | grep -i
>> StartVMCmdByAdmin
>>
>> 2) on cloudstack UI, press start on the VM and watch the tail log
>>
>> 3) on the tail log, you should see a StartVirtualMachine api call show
>> up with job ID. For example:
>>
>> > 2016-04-15 03:41:57,193 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-24:ctx-7892387a job-291575) Executing AsyncJobVO
>> {id:291575, userId: 157, accountId: 2, instanceType: VirtualMachine,
>> instanceId: 7648, cmd:
>> org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin, cmdInfo:
>> {"id":"7648","response":"json","sessionkey":"mMJsduS9KLKApqH4dPpIlGZdt1Y\u003d","ctxDetails":"{\"com.cloud.vm.VirtualMachine\":7648,\"com.cloud.uservm.UserVm\":7648,\"com.cloud.network.router.VirtualRouter\":7648}","cmdEventType":"VM.START","expires":"2016-04-15T03:51:57+0000","ctxUserId":"157","signatureversion":"3","httpmethod":"GET","uuid":"7648","ctxAccountId":"2","ctxStartEventId":"314416"},
>> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
>> result: null, initMsid: 345051337894, completeMsid: null, lastUpdated:
>> null, lastPolled: null, created: null}
>>
>> Each task in cloudstack spins up its own job, in this case its
>> *job-291575*
>>
>> 4) now execute
>> > grep -A3 -B3 job-291575
>> /var/cloudstack/management/management-server.log | tee job-291575.log
>> > grep job-291575 /var/cloudstack/management/management-server.log | grep
>> via | head -1
>>
>> you should see a response like so,
>>
>> > 2016-04-15 03:41:59,259 DEBUG [c.c.a.m.ClusteredAgentAttache]
>> (Work-Job-Executor-42:ctx-b2c4af6b job-291575/job-291576 ctx-0988152e) Seq
>> 441-4181310779036795138: Forwarding Seq 441-4181310779036795138: { Cmd ,
>> MgmtId: 345051337894, via: 441(my-kvm-host01.example.com), Ver: v1,
>> Flags: 100011,
>> [{"com.cloud.agent.api.routing.DhcpEntryCommand":{"vmMac":"06:fe:4e:00:12:ad","vmIpAddress":"10.10.27.178","vmName":"ol6-test-ilya1","defaultRouter":"10.10.27.1","defaultDns":"10.17.27.166","duid":"00:03:00:01:06:fe:4e:00:12:ad","isDefault":true,"executeInSequence":false,"accessDetails":{"zone.network.type":"Advanced","router.guest.ip":"10.10.27.166","router.ip":"169.254.0.238","
>> router.name":"r-7301-VM"},"wait":0}}] } to 345051337893
>>
>> We are looking for this line "via: 441(my-kvm-host01.example.com)"
>>
>> SSH to my-kvm-host01.example.com, and copy the
>> /var/log/cloudstack/agent.log to pastebin
>>
>> also copy job-291575.log to pastebin
>>
>> I'll try to have a look, though when i get chance.
>>
>>
>> Also paste the output of mount command on kvm host in question. I'm
>> suspecting one of your storage pools maybe bad.
>>
>> Lastly, what distro are you running for KVM?
>>
>> Regards
>> ilya
>>
>> On 4/14/16 5:05 PM, Nezar Madbouh wrote:
>> > Hi Ilya,
>> >
>> > I shutdown all the VMs in Pre-prod Environmnet expecting the Pre-prod
>> > Virtual router to be deleted automatically because no VM is running on
>> this
>> > vlan but it wasn't deleted.
>> >
>> > Am I right to expect the VR to be deleted automatically if no VM is
>> running
>> > in that vlan?
>> >
>> > Thanks,
>> > Nezar.
>> >
>> > On 15 April 2016 at 09:53, Nezar Madbouh <ne...@ipscape.com.au>
>> > wrote:
>> >
>> >> Hi Ilya,
>> >>
>> >> I have run the script and it didn't return any results:
>> >>
>> >> [root@eqx-cs-mgt-01 ~]# ./async_jobs.sh
>> >> [root@eqx-cs-mgt-01 ~]#
>> >>
>> >>
>> >> I also ran the command inside mysql:
>> >>
>> >> mysql> SELECT id, user_id, account_id, instance_type, instance_id,
>> >> job_cmd, job_cmd_info, job_dispatcher, job_pending_signals FROM
>> >> cloud.async_job where job_status=0 and job_dispatcher not like
>> >> "pseudoJobDispatcher";
>> >> Empty set (0.00 sec)
>> >>
>> >> your help is much appreciated,
>> >> Nezar.
>> >>
>> >>
>> >> On 14 April 2016 at 16:49, ilya <il...@gmail.com> wrote:
>> >>
>> >>> It appears you have pending jobs that are not letting you go through.
>> >>>
>> >>> Usually stop and starting management service should be enough.
>> >>>
>> >>> However, try running this script from your management server to show
>> us
>> >>> what you have running in async_job table.
>> >>>
>> >>>
>> >>>
>> >>>> #!/bin/bash
>> >>>> DATESTAMP=$(date +%m%d%y-%H%M%S)
>> >>>> DBPASS=$(java -classpath
>> >>> /usr/share/cloudstack-common/lib/jasypt-1.9.0.jar
>> >>> org.jasypt.intf.cli.JasyptPBEStringDecryptionCLI input="$(cat
>> >>> /etc/cloudstack/management/db.properties | grep db.cloud.password |
>> awk
>> >>> -F'(' '{print $2}' | sed 's/)//g')" password="$(cat
>> >>> /etc/cloudstack/management/key)" | grep -A2 OUTPUT | tail -1)
>> >>>> DBHOST=$(cat /etc/cloudstack/management/db.properties | grep
>> >>> db.cloud.host | awk -F'=' '{print $2}' | tail -1 )
>> >>>> DBUSER=$(cat /etc/cloudstack/management/db.properties | grep
>> >>> db.cloud.username | awk -F'=' '{print $2}')
>> >>>> DB=$(cat /etc/cloudstack/management/db.properties | grep
>> db.cloud.name
>> >>> | awk -F'=' '{print $2}')
>> >>>> DBPORT=$(cat /etc/cloudstack/management/db.properties | grep
>> >>> db.cloud.port | awk -F'=' '{print $2}')
>> >>>> MYSQLCMD="mysql -h $DBHOST -u $DBUSER -P $DBPORT -p$DBPASS $DB"
>> >>>> echo 'SELECT id, user_id, account_id, instance_type, instance_id,
>> >>> job_cmd, job_cmd_info, job_dispatcher, job_pending_signals FROM
>> >>> cloud.async_job where job_status=0 and job_dispatcher not like
>> >>> "pseudoJobDispatcher"' | $MYSQLCMD
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> On 4/13/16 5:16 PM, Nezar Madbouh wrote:
>> >>>> Hi Simon,
>> >>>>
>> >>>> The following work was done but unfortunately the problem is not
>> fixed:
>> >>>>
>> >>>>
>> >>>> on management server:
>> >>>>
>> >>>> service cloudstack-management stop
>> >>>>
>> >>>>
>> >>>> on the host:
>> >>>>
>> >>>> service cloudstack-agent stop
>> >>>>
>> >>>> service libvirtd restart
>> >>>>
>> >>>> service cloudstack-agent start
>> >>>>
>> >>>>
>> >>>> on management server:
>> >>>>
>> >>>> service cloudstack-management start
>> >>>>
>> >>>> Regards,
>> >>>> Nezar.
>> >>>>
>> >>>> On 24 March 2016 at 05:06, Simon Weller <sw...@ena.com> wrote:
>> >>>>
>> >>>>> Yes, I think that order is fine. Make sure the processes are indeed
>> >>>>> shutdown before moving on to the next step.
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> ________________________________________
>> >>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>> >>>>> Sent: Tuesday, March 22, 2016 5:33 PM
>> >>>>> To: users@cloudstack.apache.org
>> >>>>> Subject: Re: Unable to create a New VM - Error message: Unable to
>> start
>> >>>>> instance due to Unable to get answer that is of class
>> >>>>> com.cloud.agent.api.StartAnswer
>> >>>>>
>> >>>>> Hi Simon,
>> >>>>>
>> >>>>> Can you please check if the following sequence of action is correct:
>> >>>>>
>> >>>>> on management server:
>> >>>>>
>> >>>>> service cloudstack-management stop
>> >>>>>
>> >>>>>
>> >>>>> on the host:
>> >>>>>
>> >>>>> service cloudstack-agent stop
>> >>>>>
>> >>>>> service libvirtd restart
>> >>>>>
>> >>>>> service cloudstack-agent start
>> >>>>>
>> >>>>>
>> >>>>> on management server:
>> >>>>>
>> >>>>> service cloudstack-management start
>> >>>>>
>> >>>>> Thanks very much for your help.
>> >>>>> Nezar.
>> >>>>>
>> >>>>> On 23 March 2016 at 08:39, Simon Weller <sw...@ena.com> wrote:
>> >>>>>
>> >>>>>> Shutting down all the management services should clear the jobs.
>> >>>>>>
>> >>>>>>
>> >>>>>> ________________________________________
>> >>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>> >>>>>> Sent: Tuesday, March 22, 2016 4:28 PM
>> >>>>>> To: users@cloudstack.apache.org
>> >>>>>> Subject: Re: Unable to create a New VM - Error message: Unable to
>> >>> start
>> >>>>>> instance due to Unable to get answer that is of class
>> >>>>>> com.cloud.agent.api.StartAnswer
>> >>>>>>
>> >>>>>> Hi Simon,
>> >>>>>>
>> >>>>>> I will plan to restart agents, libvirtd and management services.
>> >>>>>>
>> >>>>>> How do I clear hung jobs.
>> >>>>>>
>> >>>>>> Your help is much appreciated,
>> >>>>>> Nezar.
>> >>>>>>
>> >>>>>> On 22 March 2016 at 23:03, Simon Weller <sw...@ena.com> wrote:
>> >>>>>>
>> >>>>>>> Nezar,
>> >>>>>>>
>> >>>>>>> I've looked through the code related to your release and it seems
>> >>> that
>> >>>>> in
>> >>>>>>> order to throw this exception, 10 attempts have been made to
>> retrieve
>> >>>>> the
>> >>>>>>> disk information prior to the exception being thrown.
>> >>>>>>>
>> >>>>>>> It appears there is something really odd going on here. I'm
>> starting
>> >>>>> to
>> >>>>>>> wonder whether it might be worth while for you to stop all of your
>> >>>>>> agents,
>> >>>>>>> plus management server processes, restart libvirtd on each of the
>> >>> hosts
>> >>>>>> and
>> >>>>>>> then bring the agents back up prior to starting the management
>> >>> servers.
>> >>>>>>>
>> >>>>>>> There are a bunch of logs the pastebin you posted that seems to
>> >>>>> indicate
>> >>>>>>> you have some jobs hung, for example:
>> >>>>>>>
>> >>>>>>> 2016-03-14 21:45:31,828 WARN [o.a.c.f.j.i.AsyncJobMonitor]
>> >>>>>>> (Timer-1:ctx-0c090bd7) Task (job-9066) has been pending for
>> 1660851
>> >>>>>> seconds
>> >>>>>>> 2016-03-14 21:45:31,828 WARN [o.a.c.f.j.i.AsyncJobMonitor]
>> >>>>>>> (Timer-1:ctx-0c090bd7) Task (job-8133) has been pending for
>> 4253003
>> >>>>>> seconds
>> >>>>>>>
>> >>>>>>> Restarting the agents, libvirtd and management will not cause an
>> >>>>> outage
>> >>>>>>> to your cloud, but I would suggest, if you decided to do this, you
>> >>>>> should
>> >>>>>>> complete it in a maintenance window.
>> >>>>>>>
>> >>>>>>> - Si
>> >>>>>>>
>> >>>>>>> ________________________________________
>> >>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>> >>>>>>> Sent: Sunday, March 20, 2016 10:27 PM
>> >>>>>>> To: users@cloudstack.apache.org
>> >>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable to
>> >>> start
>> >>>>>>> instance due to Unable to get answer that is of class
>> >>>>>>> com.cloud.agent.api.StartAnswer
>> >>>>>>>
>> >>>>>>> Hi Simon,
>> >>>>>>>
>> >>>>>>> Please see answers to your questions below. I am not sure if I
>> >>> provided
>> >>>>>> the
>> >>>>>>> correct answer for the primary and secondary storage
>> configuration.
>> >>>>>> Please
>> >>>>>>> let me know if this is not what you are after:
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> Can you screenshot and post somewhere your secondary storage
>> >>>>>> configuration
>> >>>>>>> and your primary storage configuration from the UI with the
>> details?
>> >>>>>>>
>> >>>>>>> mysql> select * from configuration where name like '%secondary%'\G
>> >>>>>>> *************************** 1. row ***************************
>> >>>>>>> category: Account Defaults
>> >>>>>>> instance: DEFAULT
>> >>>>>>> component: management-server
>> >>>>>>> name: max.account.secondary.storage
>> >>>>>>> value: 400
>> >>>>>>> description: The default maximum secondary storage space (in
>> GiB)
>> >>>>> that
>> >>>>>>> can be used for an account
>> >>>>>>> default_value: 400
>> >>>>>>> updated: NULL
>> >>>>>>> scope: NULL
>> >>>>>>> is_dynamic: 0
>> >>>>>>> *************************** 2. row ***************************
>> >>>>>>> category: Project Defaults
>> >>>>>>> instance: DEFAULT
>> >>>>>>> component: management-server
>> >>>>>>> name: max.project.secondary.storage
>> >>>>>>> value: 400
>> >>>>>>> description: The default maximum secondary storage space (in
>> GiB)
>> >>>>> that
>> >>>>>>> can be used for a project
>> >>>>>>> default_value: 400
>> >>>>>>> updated: NULL
>> >>>>>>> scope: NULL
>> >>>>>>> is_dynamic: 0
>> >>>>>>> *************************** 3. row ***************************
>> >>>>>>> category: Hidden
>> >>>>>>> instance: DEFAULT
>> >>>>>>> component: management-server
>> >>>>>>> name: secondary.storage.vm
>> >>>>>>> value: +rwRhJOXPymbQOuAMLnhiw==
>> >>>>>>> description: Deploys a VM per zone to manage secondary storage
>> if
>> >>>>> true,
>> >>>>>>> otherwise secondary storage is mounted on management server
>> >>>>>>> default_value: NULL
>> >>>>>>> updated: NULL
>> >>>>>>> scope: NULL
>> >>>>>>> is_dynamic: 0
>> >>>>>>> *************************** 4. row ***************************
>> >>>>>>> category: Advanced
>> >>>>>>> instance: DEFAULT
>> >>>>>>> component: ExtensionRegistry
>> >>>>>>> name: secondary.storage.vm.allocators.exclude
>> >>>>>>> value: NULL
>> >>>>>>> description: Extensions to exclude from being registered
>> >>>>>>> default_value: NULL
>> >>>>>>> updated: 2014-12-04 00:18:15
>> >>>>>>> scope: Global
>> >>>>>>> is_dynamic: 0
>> >>>>>>> *************************** 5. row ***************************
>> >>>>>>> category: NetworkManager
>> >>>>>>> instance: DEFAULT
>> >>>>>>> component: management-server
>> >>>>>>> name: vm.network.nic.max.secondary.ipaddresses
>> >>>>>>> value: NULL
>> >>>>>>> description: Specify the number of secondary ip addresses per
>> nic
>> >>> per
>> >>>>>> vm
>> >>>>>>> default_value: 256
>> >>>>>>> updated: NULL
>> >>>>>>> scope: NULL
>> >>>>>>> is_dynamic: 0
>> >>>>>>> 5 rows in set (0.00 sec)
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> mysql> select * from configuration where name like '%primary%'\G
>> >>>>>>> *************************** 1. row ***************************
>> >>>>>>> category: Account Defaults
>> >>>>>>> instance: DEFAULT
>> >>>>>>> component: management-server
>> >>>>>>> name: max.account.primary.storage
>> >>>>>>> value: 200
>> >>>>>>> description: The default maximum primary storage space (in GiB)
>> >>> that
>> >>>>>> can
>> >>>>>>> be used for an account
>> >>>>>>> default_value: 200
>> >>>>>>> updated: NULL
>> >>>>>>> scope: NULL
>> >>>>>>> is_dynamic: 0
>> >>>>>>> *************************** 2. row ***************************
>> >>>>>>> category: Project Defaults
>> >>>>>>> instance: DEFAULT
>> >>>>>>> component: management-server
>> >>>>>>> name: max.project.primary.storage
>> >>>>>>> value: 200
>> >>>>>>> description: The default maximum primary storage space (in GiB)
>> >>> that
>> >>>>>> can
>> >>>>>>> be used for a project
>> >>>>>>> default_value: 200
>> >>>>>>> updated: NULL
>> >>>>>>> scope: NULL
>> >>>>>>> is_dynamic: 0
>> >>>>>>> *************************** 3. row ***************************
>> >>>>>>> category: Storage
>> >>>>>>> instance: DEFAULT
>> >>>>>>> component: TemplateManager
>> >>>>>>> name: primary.storage.download.wait
>> >>>>>>> value: 10800
>> >>>>>>> description: In second, timeout for download template to primary
>> >>>>>> storage
>> >>>>>>> default_value: NULL
>> >>>>>>> updated: NULL
>> >>>>>>> scope: NULL
>> >>>>>>> is_dynamic: 0
>> >>>>>>> 3 rows in set (0.00 sec)
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> Also, what is the Cloudstack version you are running and what is
>> the
>> >>> OS
>> >>>>>>> you're running it on?
>> >>>>>>>
>> >>>>>>> Cloudstack 4.4.1
>> >>>>>>>
>> >>>>>>> Centos 6.6
>> >>>>>>>
>> >>>>>>> Are you using actimeo=0 in your NFS mount line?
>> >>>>>>>
>> >>>>>>> Primary storage on Hypervisors has the following parameters
>> >>>>>>> 10.61.154.10:/Cloudstack/primary on
>> >>>>>>> /mnt/b3647480-5709-36eb-9ddd-e2c666f30b57 type nfs
>> >>>>>>>
>> (rw,noatime,nfsvers=3,proto=tcp,hard,nolock,intr,addr=10.61.154.10)
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> Secondary storage on SSVM has the following parameters
>> >>>>>>> 10.61.154.10:/Cloudstack/secondary on
>> >>>>>>> /mnt/SecStorage/d11d5c92-6950-30e8-a0b5-3cd46a17ec14 type nfs4
>> >>>>>>>
>> >>>>>>>
>> >>>>>>
>> >>>>>
>> >>>
>> (rw,relatime,vers=4,rsize=262144,wsize=262144,namlen=255,acdirmin=0,acdirmax=0,soft,proto=tcp,port=0,timeo=133,retrans=
>> >>> 2147483647
>> >>>
>> ,sec=sys,clientaddr=10.61.154.70,minorversion=0,local_lock=none,addr=10.61.154.10)
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> Thanks & Regards,
>> >>>>>>> Nezar.
>> >>>>>>>
>> >>>>>>> On 18 March 2016 at 23:50, Simon Weller <sw...@ena.com> wrote:
>> >>>>>>>
>> >>>>>>>> Also, what is the Cloudstack version you are running and what is
>> the
>> >>>>> OS
>> >>>>>>>> you're running it on?
>> >>>>>>>>
>> >>>>>>>> Are you using actimeo=0 in your NFS mount line?
>> >>>>>>>>
>> >>>>>>>> ________________________________________
>> >>>>>>>> From: Simon Weller <sw...@ena.com>
>> >>>>>>>> Sent: Friday, March 18, 2016 7:40 AM
>> >>>>>>>> To: users@cloudstack.apache.org
>> >>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable to
>> >>>>> start
>> >>>>>>>> instance due to Unable to get answer that is of class
>> >>>>>>>> com.cloud.agent.api.StartAnswer
>> >>>>>>>>
>> >>>>>>>> Nexar,
>> >>>>>>>>
>> >>>>>>>> That's normal behaviour. You can only ssh to the link local on
>> the
>> >>>>> host
>> >>>>>>>> where the VM resides.
>> >>>>>>>>
>> >>>>>>>> Can you screenshot and post somewhere your secondary storage
>> >>>>>>> configuration
>> >>>>>>>> and your primary storage configuration from the UI with the
>> details?
>> >>>>>>>>
>> >>>>>>>> - Si
>> >>>>>>>>
>> >>>>>>>> ________________________________________
>> >>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>> >>>>>>>> Sent: Thursday, March 17, 2016 8:19 PM
>> >>>>>>>> To: users@cloudstack.apache.org
>> >>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable to
>> >>>>> start
>> >>>>>>>> instance due to Unable to get answer that is of class
>> >>>>>>>> com.cloud.agent.api.StartAnswer
>> >>>>>>>>
>> >>>>>>>> Hi Simon,
>> >>>>>>>>
>> >>>>>>>> I just noticed that from other Hosts I can't ssh into the SSVM:
>> >>>>>>>>
>> >>>>>>>> [root@eqx-cs-cmp-01 ~]# ssh -i /root/.ssh/id_rsa.cloud -p 3922
>> >>>>>>>> root@169.254.3.161
>> >>>>>>>> ssh: connect to host 169.254.3.161 port 3922: No route to host
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> I can only ssh from one Host:
>> >>>>>>>> [root@eqx-cs-cmp-06 ~]# ssh -i /root/.ssh/id_rsa.cloud -p 3922
>> >>>>>>>> root@169.254.3.161
>> >>>>>>>> Linux s-11-VM 3.2.0-4-amd64 #1 SMP Debian 3.2.57-3 x86_64
>> >>>>>>>>
>> >>>>>>>> Thanks & Regards,
>> >>>>>>>> Nezar.
>> >>>>>>>>
>> >>>>>>>> On 18 March 2016 at 10:36, Nezar Madbouh <
>> >>>>> nezar.madbouh@ipscape.com.au
>> >>>>>>>
>> >>>>>>>> wrote:
>> >>>>>>>>
>> >>>>>>>>> Hi Simon,
>> >>>>>>>>>
>> >>>>>>>>> Yes existing VMs are successfully running. Yesterday I was able
>> to
>> >>>>>>> create
>> >>>>>>>>> a VM in Dev Environment but I can't create VMs in Prod
>> environment.
>> >>>>>>>>>
>> >>>>>>>>> Thanks,
>> >>>>>>>>> Nezar.
>> >>>>>>>>>
>> >>>>>>>>> On 18 March 2016 at 09:12, Simon Weller <sw...@ena.com>
>> wrote:
>> >>>>>>>>>
>> >>>>>>>>>> I'm assuming you have other VMs successfully running, correct?
>> So
>> >>>>>> you
>> >>>>>>>> can
>> >>>>>>>>>> write to your primary storage.
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> ________________________________________
>> >>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>> >>>>>>>>>> Sent: Thursday, March 17, 2016 4:43 PM
>> >>>>>>>>>> To: users@cloudstack.apache.org
>> >>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable
>> to
>> >>>>>>> start
>> >>>>>>>>>> instance due to Unable to get answer that is of class
>> >>>>>>>>>> com.cloud.agent.api.StartAnswer
>> >>>>>>>>>>
>> >>>>>>>>>> Hi Simon and Shweta,
>> >>>>>>>>>>
>> >>>>>>>>>> Please see the result below:
>> >>>>>>>>>>
>> >>>>>>>>>> root@s-11-VM:~# /usr/local/cloud/systemvm/ssvm-check.sh
>> >>>>>>>>>> ================================================
>> >>>>>>>>>> First DNS server is 8.8.8.8
>> >>>>>>>>>> PING 8.8.8.8 (8.8.8.8): 48 data bytes
>> >>>>>>>>>> 56 bytes from 8.8.8.8: icmp_seq=0 ttl=57 time=5.209 ms
>> >>>>>>>>>> 56 bytes from 8.8.8.8: icmp_seq=1 ttl=57 time=0.936 ms
>> >>>>>>>>>> --- 8.8.8.8 ping statistics ---
>> >>>>>>>>>> 2 packets transmitted, 2 packets received, 0% packet loss
>> >>>>>>>>>> round-trip min/avg/max/stddev = 0.936/3.072/5.209/2.137 ms
>> >>>>>>>>>> Good: Can ping DNS server
>> >>>>>>>>>> ================================================
>> >>>>>>>>>> Good: DNS resolves download.cloud.com
>> >>>>>>>>>> ================================================
>> >>>>>>>>>> nfs is currently mounted
>> >>>>>>>>>> Mount point is
>> >>>>> /mnt/SecStorage/d11d5c92-6950-30e8-a0b5-3cd46a17ec14
>> >>>>>>>>>> Good: Can write to mount point
>> >>>>>>>>>> ================================================
>> >>>>>>>>>> Management server is 10.61.153.20. Checking connectivity.
>> >>>>>>>>>> Good: Can connect to management server port 8250
>> >>>>>>>>>> ================================================
>> >>>>>>>>>> Good: Java process is running
>> >>>>>>>>>> ================================================
>> >>>>>>>>>> Tests Complete. Look for ERROR or WARNING above.
>> >>>>>>>>>> root@s-11-VM:~#
>> >>>>>>>>>>
>> >>>>>>>>>> Thanks & Regards,
>> >>>>>>>>>> Nezar.
>> >>>>>>>>>>
>> >>>>>>>>>> On 17 March 2016 at 09:42, Simon Weller <sw...@ena.com>
>> wrote:
>> >>>>>>>>>>
>> >>>>>>>>>>> Nezar,
>> >>>>>>>>>>>
>> >>>>>>>>>>> Log into the Cloudstack UI.
>> >>>>>>>>>>>
>> >>>>>>>>>>> Click on "Infrastructure" on the left menu. Click on View All
>> >>>>>> below
>> >>>>>>>>>>> "System VMs". find the item marked "Secondary Storage VM" and
>> >>>>>> then
>> >>>>>>>>>> click
>> >>>>>>>>>>> on the Name of that VM on the left (it will be s-XXXX-VM).
>> >>>>> You'll
>> >>>>>>> see
>> >>>>>>>> a
>> >>>>>>>>>>> bunch of information about the VM. Find the Host line and also
>> >>>>> the
>> >>>>>>>> Link
>> >>>>>>>>>>> Local IP Address line and then follow the original
>> instructions
>> >>>>> in
>> >>>>>>> my
>> >>>>>>>>>>> previous email.
>> >>>>>>>>>>>
>> >>>>>>>>>>> - Si
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> ________________________________________
>> >>>>>>>>>>> From: Shweta Agarwal <sh...@accelerite.com>
>> >>>>>>>>>>> Sent: Wednesday, March 16, 2016 12:16 AM
>> >>>>>>>>>>> To: users@cloudstack.apache.org
>> >>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable
>> >>>>> to
>> >>>>>>>> start
>> >>>>>>>>>>> instance due to Unable to get answer that is of class
>> >>>>>>>>>>> com.cloud.agent.api.StartAnswer
>> >>>>>>>>>>>
>> >>>>>>>>>>> Hi Nezar,
>> >>>>>>>>>>>
>> >>>>>>>>>>> You can find host name on detail view of SSVM in UI .
>> >>>>>>>>>>> You can also use list api command listSystemVms to get the
>> host
>> >>>>>>>> details
>> >>>>>>>>>> of
>> >>>>>>>>>>> system vms.
>> >>>>>>>>>>>
>> >>>>>>>>>>> Thanks
>> >>>>>>>>>>> Shweta
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> ________________________________________
>> >>>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>> >>>>>>>>>>> Sent: Wednesday, March 16, 2016 9:58 AM
>> >>>>>>>>>>> To: users@cloudstack.apache.org
>> >>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable
>> >>>>> to
>> >>>>>>>> start
>> >>>>>>>>>>> instance due to Unable to get answer that is of class
>> >>>>>>>>>>> com.cloud.agent.api.StartAnswer
>> >>>>>>>>>>>
>> >>>>>>>>>>> Hi Simon,
>> >>>>>>>>>>>
>> >>>>>>>>>>> How can I find out which host the SSVM is on?
>> >>>>>>>>>>>
>> >>>>>>>>>>> Thanks & Regards,
>> >>>>>>>>>>> Nezar.
>> >>>>>>>>>>>
>> >>>>>>>>>>> On 16 March 2016 at 12:28, Simon Weller <sw...@ena.com>
>> >>>>> wrote:
>> >>>>>>>>>>>
>> >>>>>>>>>>>> Can you ssh into the Secondary Storage VM?
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> This can be done by getting the link-local-ip address from
>> the
>> >>>>>>>>>>>> infrastructure menu in the UI and then sshing into the host
>> >>>>> the
>> >>>>>>> SSVM
>> >>>>>>>>>> is
>> >>>>>>>>>>> on
>> >>>>>>>>>>>> and issuing this command as root: ssh -i
>> >>>>> /root/.ssh/id_rsa.cloud
>> >>>>>>> -p
>> >>>>>>>>>> 3922
>> >>>>>>>>>>>> root@link-local-ip (replace link-local-ip with the link
>> >>>>> local
>> >>>>>>> ip).
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> Try running the SSVM diangnostic script:
>> >>>>>>>>>>>> usr/local/cloud/systemvm/ssvm-check.sh
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> - Si
>> >>>>>>>>>>>> ________________________________________
>> >>>>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>> >>>>>>>>>>>> Sent: Tuesday, March 15, 2016 7:49 PM
>> >>>>>>>>>>>> To: users@cloudstack.apache.org
>> >>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message:
>> Unable
>> >>>>>> to
>> >>>>>>>>>> start
>> >>>>>>>>>>>> instance due to Unable to get answer that is of class
>> >>>>>>>>>>>> com.cloud.agent.api.StartAnswer
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> Hi Simon,
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> Please see details below:
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> mysql> use cloud; select id, name, status, type from host
>> >>>>> where
>> >>>>>>> id =
>> >>>>>>>>>> 5;
>> >>>>>>>>>>>> Reading table information for completion of table and column
>> >>>>>> names
>> >>>>>>>>>>>> You can turn off this feature to get a quicker startup with
>> -A
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> Database changed
>> >>>>>>>>>>>> +----+------------------------------+--------+---------+
>> >>>>>>>>>>>> | id | name | status | type |
>> >>>>>>>>>>>> +----+------------------------------+--------+---------+
>> >>>>>>>>>>>> | 5 | eqx-cs-cmp-02.ipscape.com.au | up | routing |
>> >>>>>>>>>>>> +----+------------------------------+--------+---------+
>> >>>>>>>>>>>> 1 row in set (0.00 sec)
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> cat /var/log/cloudstack/agent/agent.log
>> >>>>>>>>>>>> com.cloud.utils.exception.CloudRuntimeException:
>> >>>>>>>>>>>> com.cloud.utils.exception.CloudRuntimeException: Can't find
>> >>>>>>>>>>>> volume:089ff3ca-9bff-402c-a14e-ea7a35aaad5c
>> >>>>>>>>>>>> at
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>
>> >>>>>>>
>> >>>>>>
>> >>>>>
>> >>>
>> com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.getPhysicalDisk(KVMStoragePoolManager.java:262)
>> >>>>>>>>>>>> at
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>
>> >>>>>>>
>> >>>>>>
>> >>>>>
>> >>>
>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.createVbd(LibvirtComputingResource.java:3921)
>> >>>>>>>>>>>> at
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>
>> >>>>>>>
>> >>>>>>
>> >>>>>
>> >>>
>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(LibvirtComputingResource.java:3789)
>> >>>>>>>>>>>> at
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>
>> >>>>>>>
>> >>>>>>
>> >>>>>
>> >>>
>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1333)
>> >>>>>>>>>>>> at
>> >>>>> com.cloud.agent.Agent.processRequest(Agent.java:501)
>> >>>>>>>>>>>> at
>> >>>>>>>>>>>
>> com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:808)
>> >>>>>>>>>>>> at com.cloud.utils.nio.Task.run(Task.java:84)
>> >>>>>>>>>>>> at
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>
>> >>>>>>>
>> >>>>>>
>> >>>>>
>> >>>
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> >>>>>>>>>>>> at
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>
>> >>>>>>>
>> >>>>>>
>> >>>>>
>> >>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> >>>>>>>>>>>> at java.lang.Thread.run(Thread.java:745)
>> >>>>>>>>>>>> agent.log.2016-03-03.gz (END)
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> [root@eqx-cs-cmp-02 agent]# virsh pool-list
>> >>>>>>>>>>>> Name State Autostart
>> >>>>>>>>>>>> -----------------------------------------
>> >>>>>>>>>>>> 2d0967f8-50d6-4d6d-87b7-4da0596f6924 active no
>> >>>>>>>>>>>> b3647480-5709-36eb-9ddd-e2c666f30b57 active no
>> >>>>>>>>>>>> e3831246-ff55-4af4-9964-a17565edaccc active no
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> [Root @ eqx-cs-cmp-02 agent] # df -h
>> >>>>>>>>>>>> Filesystem Size Used Avail Use% Mounted on
>> >>>>>>>>>>>> /dev/mapper/vg_root-lv_root
>> >>>>>>>>>>>> 203G 215G 1.3G 1% /
>> >>>>>>>>>>>> tmpfs 48g 48g 0 0% / dev / shm
>> >>>>>>>>>>>> / Dev / sda1 477m 427m 25M 6% / boot
>> >>>>>>>>>>>> xx.xx.xx.xx:/Cloudstack/primary
>> >>>>>>>>>>>> 7.6T 4.1T 3.5T 54%
>> >>>>>>>>>>>> /mnt/b3647480-5709-36eb-9ddd-e2c666f30b57
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> I also noticed the following in cloudstack-agent.err
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> libvirt: QEMU Driver error : Domain not found: no domain with
>> >>>>>>>> matching
>> >>>>>>>>>>> name
>> >>>>>>>>>>>> 'i-11-600-VM'
>> >>>>>>>>>>>> libvirt: Storage Driver error : Storage volume not found: no
>> >>>>>>> storage
>> >>>>>>>>>> vol
>> >>>>>>>>>>>> with matching name '089ff3ca-9bff-402c-a14e-ea7a35aaad5c'
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> Your help is much appreciated,
>> >>>>>>>>>>>> Nezar
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> On 16 March 2016 at 11:03, Simon Weller <sw...@ena.com>
>> >>>>>> wrote:
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>> Nezar,
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> ok, I just took a look at a pastebin you posted.
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> I see it complaining about the inability to find a volume
>> >>>>>> prior
>> >>>>>>> to
>> >>>>>>>>>>>>> throwing the host exception.
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Firstly, let's find out who host 5 is. Can you log into your
>> >>>>>>> MySQL
>> >>>>>>>>>>> server
>> >>>>>>>>>>>>> and run this:
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> use cloud;select id,name,status,type from host where id=5;
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Can you post the results?
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> If it's a host, can you post the
>> >>>>>>>> /var/log/cloudstack/agent/agent.log
>> >>>>>>>>>>> from
>> >>>>>>>>>>>>> that host?
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> And also run virsh pool-list on the host in question?
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> - Si
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> ________________________________________
>> >>>>>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>> >>>>>>>>>>>>> Sent: Tuesday, March 15, 2016 6:11 PM
>> >>>>>>>>>>>>> To: users@cloudstack.apache.org
>> >>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message:
>> >>>>> Unable
>> >>>>>>> to
>> >>>>>>>>>> start
>> >>>>>>>>>>>>> instance due to Unable to get answer that is of class
>> >>>>>>>>>>>>> com.cloud.agent.api.StartAnswer
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Hi Simon,
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> When you say "double check your routers tied to the
>> >>>>> network",
>> >>>>>> do
>> >>>>>>>> you
>> >>>>>>>>>>> mean
>> >>>>>>>>>>>>> the physical routers or the virtual routers?
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> When I try to create a VM in Prod Environment it fails. I
>> >>>>>> can't
>> >>>>>>>> ping
>> >>>>>>>>>>> the
>> >>>>>>>>>>>>> Prod virtual router from management server or Host.
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> But I can ping Dev virtual router from management server and
>> >>>>>>> Host
>> >>>>>>>>>> and I
>> >>>>>>>>>>>> was
>> >>>>>>>>>>>>> able to create a VM in Dev Environment.
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> How can I troubleshoot the Prod Virtual router. I launched
>> >>>>> the
>> >>>>>>>>>> console
>> >>>>>>>>>>>> but
>> >>>>>>>>>>>>> couldn't login.
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Please note that we need to take into consideration that the
>> >>>>>>>>>> affected
>> >>>>>>>>>>>>> network is the Production network and we need to minimise
>> >>>>>>> service
>> >>>>>>>>>>> outage
>> >>>>>>>>>>>> as
>> >>>>>>>>>>>>> much as possible.
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Your help is much appreciated,
>> >>>>>>>>>>>>> Nezar
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> On 16 March 2016 at 08:37, Simon Weller <sw...@ena.com>
>> >>>>>>> wrote:
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>> I've seen this before where a Virtual Router has crashed
>> >>>>> and
>> >>>>>>> the
>> >>>>>>>>>>>>>> management server can't communicate with it.
>> >>>>>>>>>>>>>> Are you using an advanced network setup?
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> If so, double check your routers tied to the network where
>> >>>>>> you
>> >>>>>>>> are
>> >>>>>>>>>>>>>> attempting to start an instance.
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> - Si
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> ________________________________________
>> >>>>>>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>> >>>>>>>>>>>>>> Sent: Tuesday, March 15, 2016 4:02 PM
>> >>>>>>>>>>>>>> To: users@cloudstack.apache.org
>> >>>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message:
>> >>>>>> Unable
>> >>>>>>>> to
>> >>>>>>>>>>> start
>> >>>>>>>>>>>>>> instance due to Unable to get answer that is of class
>> >>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> Hi Glenn,
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> We have 16 hosts, all KVM.
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> The error is not specific to Host 5. Every time I try to
>> >>>>>>> create
>> >>>>>>>> a
>> >>>>>>>>>> VM
>> >>>>>>>>>>> it
>> >>>>>>>>>>>>>> gives me the same error message but on different host.
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> How do I restart the agent on the host. Is it service
>> >>>>>>> affecting?
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> Thanks very much for your help.
>> >>>>>>>>>>>>>> Neza
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> On 15 March 2016 at 17:37, Glenn Wagner <
>> >>>>>>>>>> glenn.wagner@shapeblue.com>
>> >>>>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> Hi,
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> Can you explain your environment first , how many hosts
>> >>>>> ,
>> >>>>>>> what
>> >>>>>>>>>> is
>> >>>>>>>>>>>> your
>> >>>>>>>>>>>>>>> hypervisors (KVM,Xenserver,VMware)
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> The error seems to be with your Host 5 Hypervisor
>> >>>>>>>>>>>>>>> Resource [Host:5] is unreachable:
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> Go to the infrastructure section and see what host5 is
>> >>>>>>>>>> reporting,
>> >>>>>>>>>>>>>>> Have you tried to restart the agent on the Host?
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> Thanks
>> >>>>>>>>>>>>>>> Glenn
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> [image: ShapeBlue] <http://www.shapeblue.com>
>> >>>>>>>>>>>>>>> Glenn Wagner
>> >>>>>>>>>>>>>>> Senior Consultant , ShapeBlue
>> >>>>>>>>>>>>>>> d: * | s: +27 21 527 0091*
>> >>>>>>>>>> <%7C%20s:%20+27%2021%20527%200091> |
>> >>>>>>>>>>>> m:
>> >>>>>>>>>>>>>>> *+27 73 917 4111* <+27%2073%20917%204111>
>> >>>>>>>>>>>>>>> e: *glenn.wagner@shapeblue.com | t: *
>> >>>>>>>>>>>>>>> <glenn.wagner@shapeblue.com%20%7C%20t:> | w: *
>> >>>>>>>>>> www.shapeblue.com
>> >>>>>>>>>>> *
>> >>>>>>>>>>>>>>> <http://www.shapeblue.com>
>> >>>>>>>>>>>>>>> a:
>> >>>>>>>>>>>>>>> 2nd Floor, Oudehuis Centre, 122 Main Rd, Somerset West
>> >>>>>> Cape
>> >>>>>>>> Town
>> >>>>>>>>>>> 7130
>> >>>>>>>>>>>>>> South Africa
>> >>>>>>>>>>>>>>> Shape Blue Ltd is a company incorporated in England &
>> >>>>>> Wales.
>> >>>>>>>>>>>> ShapeBlue
>> >>>>>>>>>>>>>>> Services India LLP is a company incorporated in India
>> >>>>> and
>> >>>>>> is
>> >>>>>>>>>>> operated
>> >>>>>>>>>>>>>> under
>> >>>>>>>>>>>>>>> license from Shape Blue Ltd. Shape Blue Brasil
>> >>>>> Consultoria
>> >>>>>>>> Ltda
>> >>>>>>>>>> is
>> >>>>>>>>>>> a
>> >>>>>>>>>>>>>>> company incorporated in Brasil and is operated under
>> >>>>>> license
>> >>>>>>>>>> from
>> >>>>>>>>>>>> Shape
>> >>>>>>>>>>>>>>> Blue Ltd. ShapeBlue SA Pty Ltd is a company registered
>> >>>>> by
>> >>>>>>> The
>> >>>>>>>>>>>> Republic
>> >>>>>>>>>>>>> of
>> >>>>>>>>>>>>>>> South Africa and is traded under license from Shape Blue
>> >>>>>>> Ltd.
>> >>>>>>>>>>>> ShapeBlue
>> >>>>>>>>>>>>>> is
>> >>>>>>>>>>>>>>> a registered trademark.
>> >>>>>>>>>>>>>>> This email and any attachments to it may be confidential
>> >>>>>> and
>> >>>>>>>> are
>> >>>>>>>>>>>>> intended
>> >>>>>>>>>>>>>>> solely for the use of the individual to whom it is
>> >>>>>>> addressed.
>> >>>>>>>>>> Any
>> >>>>>>>>>>>> views
>> >>>>>>>>>>>>>> or
>> >>>>>>>>>>>>>>> opinions expressed are solely those of the author and do
>> >>>>>> not
>> >>>>>>>>>>>>> necessarily
>> >>>>>>>>>>>>>>> represent those of Shape Blue Ltd or related companies.
>> >>>>> If
>> >>>>>>> you
>> >>>>>>>>>> are
>> >>>>>>>>>>>> not
>> >>>>>>>>>>>>>> the
>> >>>>>>>>>>>>>>> intended recipient of this email, you must neither take
>> >>>>>> any
>> >>>>>>>>>> action
>> >>>>>>>>>>>>> based
>> >>>>>>>>>>>>>>> upon its contents, nor copy or show it to anyone. Please
>> >>>>>>>> contact
>> >>>>>>>>>>> the
>> >>>>>>>>>>>>>> sender
>> >>>>>>>>>>>>>>> if you believe you have received this email in error.
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> -----Original Message-----
>> >>>>>>>>>>>>>>> From: Nezar Madbouh [mailto:
>> >>>>> nezar.madbouh@ipscape.com.au]
>> >>>>>>>>>>>>>>> Sent: Tuesday, 15 March 2016 5:45 AM
>> >>>>>>>>>>>>>>> To: users@cloudstack.apache.org
>> >>>>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message:
>> >>>>>>> Unable
>> >>>>>>>>>> to
>> >>>>>>>>>>>> start
>> >>>>>>>>>>>>>>> instance due to Unable to get answer that is of class
>> >>>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> Hi Jeremy,
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> I have uploaded the contents to
>> >>>>>>> http://pastebin.com/1J3mmgTh
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> your help is much appreciated,
>> >>>>>>>>>>>>>>> Nezar.
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> On 15 March 2016 at 13:46, Jeremy Peterson <
>> >>>>>>>>>> jpeterson@acentek.net>
>> >>>>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> I don't see any attachments can you create a pastebin.
>> >>>>>>>> That's
>> >>>>>>>>>>>> easier
>> >>>>>>>>>>>>>>>> to look through as I think attachments are removed
>> >>>>>> before
>> >>>>>>>>>> being
>> >>>>>>>>>>>> sent
>> >>>>>>>>>>>>>>>> to user group.
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> Jeremy
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> -------- Original message --------
>> >>>>>>>>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>> >>>>>>>>>>>>>>>> Date: 3/14/2016 6:38 PM (GMT-06:00)
>> >>>>>>>>>>>>>>>> To: users@cloudstack.apache.org
>> >>>>>>>>>>>>>>>> Subject: Unable to create a New VM - Error message:
>> >>>>>> Unable
>> >>>>>>>> to
>> >>>>>>>>>>> start
>> >>>>>>>>>>>>>>>> instance due to Unable to get answer that is of class
>> >>>>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> Hi Shweta and Sadhu,
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> Please find attached managerment-server.log which
>> >>>>>> contains
>> >>>>>>>> the
>> >>>>>>>>>>> logs
>> >>>>>>>>>>>>>>>> since I started creating a new VM (VM id 600).
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> Please note that we don't have agent.log files
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> Your help is much appreciated.
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> Regards,
>> >>>>>>>>>>>>>>>> Nezar.
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> Find out more about ShapeBlue and our range of
>> >>>>> CloudStack
>> >>>>>>>>>> related
>> >>>>>>>>>>>>>> services:
>> >>>>>>>>>>>>>>> IaaS Cloud Design & Build
>> >>>>>>>>>>>>>>> <http://shapeblue.com/iaas-cloud-design-and-build//> |
>> >>>>>>>> CSForge
>> >>>>>>>>>> -
>> >>>>>>>>>>>> rapid
>> >>>>>>>>>>>>>>> IaaS deployment framework <
>> >>>>> http://shapeblue.com/csforge/>
>> >>>>>>>>>>>>>>> CloudStack Consulting <
>> >>>>>>>>>>> http://shapeblue.com/cloudstack-consultancy/>
>> >>>>>>>>>>>> |
>> >>>>>>>>>>>>>> CloudStack
>> >>>>>>>>>>>>>>> Software Engineering
>> >>>>>>>>>>>>>>> <http://shapeblue.com/cloudstack-software-engineering/>
>> >>>>>>>>>>>>>>> CloudStack Infrastructure Support
>> >>>>>>>>>>>>>>> <
>> >>>>> http://shapeblue.com/cloudstack-infrastructure-support/>
>> >>>>>> |
>> >>>>>>>>>>>> CloudStack
>> >>>>>>>>>>>>>>> Bootcamp Training Courses <
>> >>>>>>>>>>> http://shapeblue.com/cloudstack-training/
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> DISCLAIMER
>> >>>>>>>>>>> ==========
>> >>>>>>>>>>> This e-mail may contain privileged and confidential
>> information
>> >>>>>>> which
>> >>>>>>>> is
>> >>>>>>>>>>> the property of Accelerite, a Persistent Systems business. It
>> is
>> >>>>>>>>>> intended
>> >>>>>>>>>>> only for the use of the individual or entity to which it is
>> >>>>>>> addressed.
>> >>>>>>>>>> If
>> >>>>>>>>>>> you are not the intended recipient, you are not authorized to
>> >>>>>> read,
>> >>>>>>>>>> retain,
>> >>>>>>>>>>> copy, print, distribute or use this message. If you have
>> >>>>> received
>> >>>>>>> this
>> >>>>>>>>>>> communication in error, please notify the sender and delete
>> all
>> >>>>>>> copies
>> >>>>>>>>>> of
>> >>>>>>>>>>> this message. Accelerite, a Persistent Systems business does
>> not
>> >>>>>>>> accept
>> >>>>>>>>>> any
>> >>>>>>>>>>> liability for virus infected mails.
>> >>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>>
>> >>>>>>
>> >>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>
>> >>
>> >
>>
>>
>
Re: Unable to create a New VM - Error message: Unable to start
instance due to Unable to get answer that is of class com.cloud.agent.api.StartAnswer
Posted by Nezar Madbouh <ne...@ipscape.com.au>.
Hi Ilya,
Thanks for the instructions. I followed them and pasted the information on
pastebin:
management-server logs job-14938
http://pastebin.com/GAiy8u5H
Agent logs
http://pastebin.com/mfiADsJc
Your help is much appreciated.
Nezar.
On 15 April 2016 at 13:58, ilya <il...@gmail.com> wrote:
> Here is how i would troubleshoot this:
>
> 1) ssh to MS host, run
>
> tail -f /var/cloudstack/management/management-server.log | grep -i
> StartVMCmdByAdmin
>
> 2) on cloudstack UI, press start on the VM and watch the tail log
>
> 3) on the tail log, you should see a StartVirtualMachine api call show
> up with job ID. For example:
>
> > 2016-04-15 03:41:57,193 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-24:ctx-7892387a job-291575) Executing AsyncJobVO
> {id:291575, userId: 157, accountId: 2, instanceType: VirtualMachine,
> instanceId: 7648, cmd:
> org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin, cmdInfo:
> {"id":"7648","response":"json","sessionkey":"mMJsduS9KLKApqH4dPpIlGZdt1Y\u003d","ctxDetails":"{\"com.cloud.vm.VirtualMachine\":7648,\"com.cloud.uservm.UserVm\":7648,\"com.cloud.network.router.VirtualRouter\":7648}","cmdEventType":"VM.START","expires":"2016-04-15T03:51:57+0000","ctxUserId":"157","signatureversion":"3","httpmethod":"GET","uuid":"7648","ctxAccountId":"2","ctxStartEventId":"314416"},
> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
> result: null, initMsid: 345051337894, completeMsid: null, lastUpdated:
> null, lastPolled: null, created: null}
>
> Each task in cloudstack spins up its own job, in this case its *job-291575*
>
> 4) now execute
> > grep -A3 -B3 job-291575 /var/cloudstack/management/management-server.log
> | tee job-291575.log
> > grep job-291575 /var/cloudstack/management/management-server.log | grep
> via | head -1
>
> you should see a response like so,
>
> > 2016-04-15 03:41:59,259 DEBUG [c.c.a.m.ClusteredAgentAttache]
> (Work-Job-Executor-42:ctx-b2c4af6b job-291575/job-291576 ctx-0988152e) Seq
> 441-4181310779036795138: Forwarding Seq 441-4181310779036795138: { Cmd ,
> MgmtId: 345051337894, via: 441(my-kvm-host01.example.com), Ver: v1,
> Flags: 100011,
> [{"com.cloud.agent.api.routing.DhcpEntryCommand":{"vmMac":"06:fe:4e:00:12:ad","vmIpAddress":"10.10.27.178","vmName":"ol6-test-ilya1","defaultRouter":"10.10.27.1","defaultDns":"10.17.27.166","duid":"00:03:00:01:06:fe:4e:00:12:ad","isDefault":true,"executeInSequence":false,"accessDetails":{"zone.network.type":"Advanced","router.guest.ip":"10.10.27.166","router.ip":"169.254.0.238","
> router.name":"r-7301-VM"},"wait":0}}] } to 345051337893
>
> We are looking for this line "via: 441(my-kvm-host01.example.com)"
>
> SSH to my-kvm-host01.example.com, and copy the
> /var/log/cloudstack/agent.log to pastebin
>
> also copy job-291575.log to pastebin
>
> I'll try to have a look, though when i get chance.
>
>
> Also paste the output of mount command on kvm host in question. I'm
> suspecting one of your storage pools maybe bad.
>
> Lastly, what distro are you running for KVM?
>
> Regards
> ilya
>
> On 4/14/16 5:05 PM, Nezar Madbouh wrote:
> > Hi Ilya,
> >
> > I shutdown all the VMs in Pre-prod Environmnet expecting the Pre-prod
> > Virtual router to be deleted automatically because no VM is running on
> this
> > vlan but it wasn't deleted.
> >
> > Am I right to expect the VR to be deleted automatically if no VM is
> running
> > in that vlan?
> >
> > Thanks,
> > Nezar.
> >
> > On 15 April 2016 at 09:53, Nezar Madbouh <ne...@ipscape.com.au>
> > wrote:
> >
> >> Hi Ilya,
> >>
> >> I have run the script and it didn't return any results:
> >>
> >> [root@eqx-cs-mgt-01 ~]# ./async_jobs.sh
> >> [root@eqx-cs-mgt-01 ~]#
> >>
> >>
> >> I also ran the command inside mysql:
> >>
> >> mysql> SELECT id, user_id, account_id, instance_type, instance_id,
> >> job_cmd, job_cmd_info, job_dispatcher, job_pending_signals FROM
> >> cloud.async_job where job_status=0 and job_dispatcher not like
> >> "pseudoJobDispatcher";
> >> Empty set (0.00 sec)
> >>
> >> your help is much appreciated,
> >> Nezar.
> >>
> >>
> >> On 14 April 2016 at 16:49, ilya <il...@gmail.com> wrote:
> >>
> >>> It appears you have pending jobs that are not letting you go through.
> >>>
> >>> Usually stop and starting management service should be enough.
> >>>
> >>> However, try running this script from your management server to show us
> >>> what you have running in async_job table.
> >>>
> >>>
> >>>
> >>>> #!/bin/bash
> >>>> DATESTAMP=$(date +%m%d%y-%H%M%S)
> >>>> DBPASS=$(java -classpath
> >>> /usr/share/cloudstack-common/lib/jasypt-1.9.0.jar
> >>> org.jasypt.intf.cli.JasyptPBEStringDecryptionCLI input="$(cat
> >>> /etc/cloudstack/management/db.properties | grep db.cloud.password | awk
> >>> -F'(' '{print $2}' | sed 's/)//g')" password="$(cat
> >>> /etc/cloudstack/management/key)" | grep -A2 OUTPUT | tail -1)
> >>>> DBHOST=$(cat /etc/cloudstack/management/db.properties | grep
> >>> db.cloud.host | awk -F'=' '{print $2}' | tail -1 )
> >>>> DBUSER=$(cat /etc/cloudstack/management/db.properties | grep
> >>> db.cloud.username | awk -F'=' '{print $2}')
> >>>> DB=$(cat /etc/cloudstack/management/db.properties | grep
> db.cloud.name
> >>> | awk -F'=' '{print $2}')
> >>>> DBPORT=$(cat /etc/cloudstack/management/db.properties | grep
> >>> db.cloud.port | awk -F'=' '{print $2}')
> >>>> MYSQLCMD="mysql -h $DBHOST -u $DBUSER -P $DBPORT -p$DBPASS $DB"
> >>>> echo 'SELECT id, user_id, account_id, instance_type, instance_id,
> >>> job_cmd, job_cmd_info, job_dispatcher, job_pending_signals FROM
> >>> cloud.async_job where job_status=0 and job_dispatcher not like
> >>> "pseudoJobDispatcher"' | $MYSQLCMD
> >>>
> >>>
> >>>
> >>>
> >>> On 4/13/16 5:16 PM, Nezar Madbouh wrote:
> >>>> Hi Simon,
> >>>>
> >>>> The following work was done but unfortunately the problem is not
> fixed:
> >>>>
> >>>>
> >>>> on management server:
> >>>>
> >>>> service cloudstack-management stop
> >>>>
> >>>>
> >>>> on the host:
> >>>>
> >>>> service cloudstack-agent stop
> >>>>
> >>>> service libvirtd restart
> >>>>
> >>>> service cloudstack-agent start
> >>>>
> >>>>
> >>>> on management server:
> >>>>
> >>>> service cloudstack-management start
> >>>>
> >>>> Regards,
> >>>> Nezar.
> >>>>
> >>>> On 24 March 2016 at 05:06, Simon Weller <sw...@ena.com> wrote:
> >>>>
> >>>>> Yes, I think that order is fine. Make sure the processes are indeed
> >>>>> shutdown before moving on to the next step.
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> ________________________________________
> >>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
> >>>>> Sent: Tuesday, March 22, 2016 5:33 PM
> >>>>> To: users@cloudstack.apache.org
> >>>>> Subject: Re: Unable to create a New VM - Error message: Unable to
> start
> >>>>> instance due to Unable to get answer that is of class
> >>>>> com.cloud.agent.api.StartAnswer
> >>>>>
> >>>>> Hi Simon,
> >>>>>
> >>>>> Can you please check if the following sequence of action is correct:
> >>>>>
> >>>>> on management server:
> >>>>>
> >>>>> service cloudstack-management stop
> >>>>>
> >>>>>
> >>>>> on the host:
> >>>>>
> >>>>> service cloudstack-agent stop
> >>>>>
> >>>>> service libvirtd restart
> >>>>>
> >>>>> service cloudstack-agent start
> >>>>>
> >>>>>
> >>>>> on management server:
> >>>>>
> >>>>> service cloudstack-management start
> >>>>>
> >>>>> Thanks very much for your help.
> >>>>> Nezar.
> >>>>>
> >>>>> On 23 March 2016 at 08:39, Simon Weller <sw...@ena.com> wrote:
> >>>>>
> >>>>>> Shutting down all the management services should clear the jobs.
> >>>>>>
> >>>>>>
> >>>>>> ________________________________________
> >>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
> >>>>>> Sent: Tuesday, March 22, 2016 4:28 PM
> >>>>>> To: users@cloudstack.apache.org
> >>>>>> Subject: Re: Unable to create a New VM - Error message: Unable to
> >>> start
> >>>>>> instance due to Unable to get answer that is of class
> >>>>>> com.cloud.agent.api.StartAnswer
> >>>>>>
> >>>>>> Hi Simon,
> >>>>>>
> >>>>>> I will plan to restart agents, libvirtd and management services.
> >>>>>>
> >>>>>> How do I clear hung jobs.
> >>>>>>
> >>>>>> Your help is much appreciated,
> >>>>>> Nezar.
> >>>>>>
> >>>>>> On 22 March 2016 at 23:03, Simon Weller <sw...@ena.com> wrote:
> >>>>>>
> >>>>>>> Nezar,
> >>>>>>>
> >>>>>>> I've looked through the code related to your release and it seems
> >>> that
> >>>>> in
> >>>>>>> order to throw this exception, 10 attempts have been made to
> retrieve
> >>>>> the
> >>>>>>> disk information prior to the exception being thrown.
> >>>>>>>
> >>>>>>> It appears there is something really odd going on here. I'm
> starting
> >>>>> to
> >>>>>>> wonder whether it might be worth while for you to stop all of your
> >>>>>> agents,
> >>>>>>> plus management server processes, restart libvirtd on each of the
> >>> hosts
> >>>>>> and
> >>>>>>> then bring the agents back up prior to starting the management
> >>> servers.
> >>>>>>>
> >>>>>>> There are a bunch of logs the pastebin you posted that seems to
> >>>>> indicate
> >>>>>>> you have some jobs hung, for example:
> >>>>>>>
> >>>>>>> 2016-03-14 21:45:31,828 WARN [o.a.c.f.j.i.AsyncJobMonitor]
> >>>>>>> (Timer-1:ctx-0c090bd7) Task (job-9066) has been pending for 1660851
> >>>>>> seconds
> >>>>>>> 2016-03-14 21:45:31,828 WARN [o.a.c.f.j.i.AsyncJobMonitor]
> >>>>>>> (Timer-1:ctx-0c090bd7) Task (job-8133) has been pending for 4253003
> >>>>>> seconds
> >>>>>>>
> >>>>>>> Restarting the agents, libvirtd and management will not cause an
> >>>>> outage
> >>>>>>> to your cloud, but I would suggest, if you decided to do this, you
> >>>>> should
> >>>>>>> complete it in a maintenance window.
> >>>>>>>
> >>>>>>> - Si
> >>>>>>>
> >>>>>>> ________________________________________
> >>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
> >>>>>>> Sent: Sunday, March 20, 2016 10:27 PM
> >>>>>>> To: users@cloudstack.apache.org
> >>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable to
> >>> start
> >>>>>>> instance due to Unable to get answer that is of class
> >>>>>>> com.cloud.agent.api.StartAnswer
> >>>>>>>
> >>>>>>> Hi Simon,
> >>>>>>>
> >>>>>>> Please see answers to your questions below. I am not sure if I
> >>> provided
> >>>>>> the
> >>>>>>> correct answer for the primary and secondary storage configuration.
> >>>>>> Please
> >>>>>>> let me know if this is not what you are after:
> >>>>>>>
> >>>>>>>
> >>>>>>> Can you screenshot and post somewhere your secondary storage
> >>>>>> configuration
> >>>>>>> and your primary storage configuration from the UI with the
> details?
> >>>>>>>
> >>>>>>> mysql> select * from configuration where name like '%secondary%'\G
> >>>>>>> *************************** 1. row ***************************
> >>>>>>> category: Account Defaults
> >>>>>>> instance: DEFAULT
> >>>>>>> component: management-server
> >>>>>>> name: max.account.secondary.storage
> >>>>>>> value: 400
> >>>>>>> description: The default maximum secondary storage space (in GiB)
> >>>>> that
> >>>>>>> can be used for an account
> >>>>>>> default_value: 400
> >>>>>>> updated: NULL
> >>>>>>> scope: NULL
> >>>>>>> is_dynamic: 0
> >>>>>>> *************************** 2. row ***************************
> >>>>>>> category: Project Defaults
> >>>>>>> instance: DEFAULT
> >>>>>>> component: management-server
> >>>>>>> name: max.project.secondary.storage
> >>>>>>> value: 400
> >>>>>>> description: The default maximum secondary storage space (in GiB)
> >>>>> that
> >>>>>>> can be used for a project
> >>>>>>> default_value: 400
> >>>>>>> updated: NULL
> >>>>>>> scope: NULL
> >>>>>>> is_dynamic: 0
> >>>>>>> *************************** 3. row ***************************
> >>>>>>> category: Hidden
> >>>>>>> instance: DEFAULT
> >>>>>>> component: management-server
> >>>>>>> name: secondary.storage.vm
> >>>>>>> value: +rwRhJOXPymbQOuAMLnhiw==
> >>>>>>> description: Deploys a VM per zone to manage secondary storage if
> >>>>> true,
> >>>>>>> otherwise secondary storage is mounted on management server
> >>>>>>> default_value: NULL
> >>>>>>> updated: NULL
> >>>>>>> scope: NULL
> >>>>>>> is_dynamic: 0
> >>>>>>> *************************** 4. row ***************************
> >>>>>>> category: Advanced
> >>>>>>> instance: DEFAULT
> >>>>>>> component: ExtensionRegistry
> >>>>>>> name: secondary.storage.vm.allocators.exclude
> >>>>>>> value: NULL
> >>>>>>> description: Extensions to exclude from being registered
> >>>>>>> default_value: NULL
> >>>>>>> updated: 2014-12-04 00:18:15
> >>>>>>> scope: Global
> >>>>>>> is_dynamic: 0
> >>>>>>> *************************** 5. row ***************************
> >>>>>>> category: NetworkManager
> >>>>>>> instance: DEFAULT
> >>>>>>> component: management-server
> >>>>>>> name: vm.network.nic.max.secondary.ipaddresses
> >>>>>>> value: NULL
> >>>>>>> description: Specify the number of secondary ip addresses per nic
> >>> per
> >>>>>> vm
> >>>>>>> default_value: 256
> >>>>>>> updated: NULL
> >>>>>>> scope: NULL
> >>>>>>> is_dynamic: 0
> >>>>>>> 5 rows in set (0.00 sec)
> >>>>>>>
> >>>>>>>
> >>>>>>> mysql> select * from configuration where name like '%primary%'\G
> >>>>>>> *************************** 1. row ***************************
> >>>>>>> category: Account Defaults
> >>>>>>> instance: DEFAULT
> >>>>>>> component: management-server
> >>>>>>> name: max.account.primary.storage
> >>>>>>> value: 200
> >>>>>>> description: The default maximum primary storage space (in GiB)
> >>> that
> >>>>>> can
> >>>>>>> be used for an account
> >>>>>>> default_value: 200
> >>>>>>> updated: NULL
> >>>>>>> scope: NULL
> >>>>>>> is_dynamic: 0
> >>>>>>> *************************** 2. row ***************************
> >>>>>>> category: Project Defaults
> >>>>>>> instance: DEFAULT
> >>>>>>> component: management-server
> >>>>>>> name: max.project.primary.storage
> >>>>>>> value: 200
> >>>>>>> description: The default maximum primary storage space (in GiB)
> >>> that
> >>>>>> can
> >>>>>>> be used for a project
> >>>>>>> default_value: 200
> >>>>>>> updated: NULL
> >>>>>>> scope: NULL
> >>>>>>> is_dynamic: 0
> >>>>>>> *************************** 3. row ***************************
> >>>>>>> category: Storage
> >>>>>>> instance: DEFAULT
> >>>>>>> component: TemplateManager
> >>>>>>> name: primary.storage.download.wait
> >>>>>>> value: 10800
> >>>>>>> description: In second, timeout for download template to primary
> >>>>>> storage
> >>>>>>> default_value: NULL
> >>>>>>> updated: NULL
> >>>>>>> scope: NULL
> >>>>>>> is_dynamic: 0
> >>>>>>> 3 rows in set (0.00 sec)
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> Also, what is the Cloudstack version you are running and what is
> the
> >>> OS
> >>>>>>> you're running it on?
> >>>>>>>
> >>>>>>> Cloudstack 4.4.1
> >>>>>>>
> >>>>>>> Centos 6.6
> >>>>>>>
> >>>>>>> Are you using actimeo=0 in your NFS mount line?
> >>>>>>>
> >>>>>>> Primary storage on Hypervisors has the following parameters
> >>>>>>> 10.61.154.10:/Cloudstack/primary on
> >>>>>>> /mnt/b3647480-5709-36eb-9ddd-e2c666f30b57 type nfs
> >>>>>>> (rw,noatime,nfsvers=3,proto=tcp,hard,nolock,intr,addr=10.61.154.10)
> >>>>>>>
> >>>>>>>
> >>>>>>> Secondary storage on SSVM has the following parameters
> >>>>>>> 10.61.154.10:/Cloudstack/secondary on
> >>>>>>> /mnt/SecStorage/d11d5c92-6950-30e8-a0b5-3cd46a17ec14 type nfs4
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>
> (rw,relatime,vers=4,rsize=262144,wsize=262144,namlen=255,acdirmin=0,acdirmax=0,soft,proto=tcp,port=0,timeo=133,retrans=
> >>> 2147483647
> >>>
> ,sec=sys,clientaddr=10.61.154.70,minorversion=0,local_lock=none,addr=10.61.154.10)
> >>>>>>>
> >>>>>>>
> >>>>>>> Thanks & Regards,
> >>>>>>> Nezar.
> >>>>>>>
> >>>>>>> On 18 March 2016 at 23:50, Simon Weller <sw...@ena.com> wrote:
> >>>>>>>
> >>>>>>>> Also, what is the Cloudstack version you are running and what is
> the
> >>>>> OS
> >>>>>>>> you're running it on?
> >>>>>>>>
> >>>>>>>> Are you using actimeo=0 in your NFS mount line?
> >>>>>>>>
> >>>>>>>> ________________________________________
> >>>>>>>> From: Simon Weller <sw...@ena.com>
> >>>>>>>> Sent: Friday, March 18, 2016 7:40 AM
> >>>>>>>> To: users@cloudstack.apache.org
> >>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable to
> >>>>> start
> >>>>>>>> instance due to Unable to get answer that is of class
> >>>>>>>> com.cloud.agent.api.StartAnswer
> >>>>>>>>
> >>>>>>>> Nexar,
> >>>>>>>>
> >>>>>>>> That's normal behaviour. You can only ssh to the link local on the
> >>>>> host
> >>>>>>>> where the VM resides.
> >>>>>>>>
> >>>>>>>> Can you screenshot and post somewhere your secondary storage
> >>>>>>> configuration
> >>>>>>>> and your primary storage configuration from the UI with the
> details?
> >>>>>>>>
> >>>>>>>> - Si
> >>>>>>>>
> >>>>>>>> ________________________________________
> >>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
> >>>>>>>> Sent: Thursday, March 17, 2016 8:19 PM
> >>>>>>>> To: users@cloudstack.apache.org
> >>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable to
> >>>>> start
> >>>>>>>> instance due to Unable to get answer that is of class
> >>>>>>>> com.cloud.agent.api.StartAnswer
> >>>>>>>>
> >>>>>>>> Hi Simon,
> >>>>>>>>
> >>>>>>>> I just noticed that from other Hosts I can't ssh into the SSVM:
> >>>>>>>>
> >>>>>>>> [root@eqx-cs-cmp-01 ~]# ssh -i /root/.ssh/id_rsa.cloud -p 3922
> >>>>>>>> root@169.254.3.161
> >>>>>>>> ssh: connect to host 169.254.3.161 port 3922: No route to host
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> I can only ssh from one Host:
> >>>>>>>> [root@eqx-cs-cmp-06 ~]# ssh -i /root/.ssh/id_rsa.cloud -p 3922
> >>>>>>>> root@169.254.3.161
> >>>>>>>> Linux s-11-VM 3.2.0-4-amd64 #1 SMP Debian 3.2.57-3 x86_64
> >>>>>>>>
> >>>>>>>> Thanks & Regards,
> >>>>>>>> Nezar.
> >>>>>>>>
> >>>>>>>> On 18 March 2016 at 10:36, Nezar Madbouh <
> >>>>> nezar.madbouh@ipscape.com.au
> >>>>>>>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Hi Simon,
> >>>>>>>>>
> >>>>>>>>> Yes existing VMs are successfully running. Yesterday I was able
> to
> >>>>>>> create
> >>>>>>>>> a VM in Dev Environment but I can't create VMs in Prod
> environment.
> >>>>>>>>>
> >>>>>>>>> Thanks,
> >>>>>>>>> Nezar.
> >>>>>>>>>
> >>>>>>>>> On 18 March 2016 at 09:12, Simon Weller <sw...@ena.com> wrote:
> >>>>>>>>>
> >>>>>>>>>> I'm assuming you have other VMs successfully running, correct?
> So
> >>>>>> you
> >>>>>>>> can
> >>>>>>>>>> write to your primary storage.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> ________________________________________
> >>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
> >>>>>>>>>> Sent: Thursday, March 17, 2016 4:43 PM
> >>>>>>>>>> To: users@cloudstack.apache.org
> >>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable
> to
> >>>>>>> start
> >>>>>>>>>> instance due to Unable to get answer that is of class
> >>>>>>>>>> com.cloud.agent.api.StartAnswer
> >>>>>>>>>>
> >>>>>>>>>> Hi Simon and Shweta,
> >>>>>>>>>>
> >>>>>>>>>> Please see the result below:
> >>>>>>>>>>
> >>>>>>>>>> root@s-11-VM:~# /usr/local/cloud/systemvm/ssvm-check.sh
> >>>>>>>>>> ================================================
> >>>>>>>>>> First DNS server is 8.8.8.8
> >>>>>>>>>> PING 8.8.8.8 (8.8.8.8): 48 data bytes
> >>>>>>>>>> 56 bytes from 8.8.8.8: icmp_seq=0 ttl=57 time=5.209 ms
> >>>>>>>>>> 56 bytes from 8.8.8.8: icmp_seq=1 ttl=57 time=0.936 ms
> >>>>>>>>>> --- 8.8.8.8 ping statistics ---
> >>>>>>>>>> 2 packets transmitted, 2 packets received, 0% packet loss
> >>>>>>>>>> round-trip min/avg/max/stddev = 0.936/3.072/5.209/2.137 ms
> >>>>>>>>>> Good: Can ping DNS server
> >>>>>>>>>> ================================================
> >>>>>>>>>> Good: DNS resolves download.cloud.com
> >>>>>>>>>> ================================================
> >>>>>>>>>> nfs is currently mounted
> >>>>>>>>>> Mount point is
> >>>>> /mnt/SecStorage/d11d5c92-6950-30e8-a0b5-3cd46a17ec14
> >>>>>>>>>> Good: Can write to mount point
> >>>>>>>>>> ================================================
> >>>>>>>>>> Management server is 10.61.153.20. Checking connectivity.
> >>>>>>>>>> Good: Can connect to management server port 8250
> >>>>>>>>>> ================================================
> >>>>>>>>>> Good: Java process is running
> >>>>>>>>>> ================================================
> >>>>>>>>>> Tests Complete. Look for ERROR or WARNING above.
> >>>>>>>>>> root@s-11-VM:~#
> >>>>>>>>>>
> >>>>>>>>>> Thanks & Regards,
> >>>>>>>>>> Nezar.
> >>>>>>>>>>
> >>>>>>>>>> On 17 March 2016 at 09:42, Simon Weller <sw...@ena.com>
> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Nezar,
> >>>>>>>>>>>
> >>>>>>>>>>> Log into the Cloudstack UI.
> >>>>>>>>>>>
> >>>>>>>>>>> Click on "Infrastructure" on the left menu. Click on View All
> >>>>>> below
> >>>>>>>>>>> "System VMs". find the item marked "Secondary Storage VM" and
> >>>>>> then
> >>>>>>>>>> click
> >>>>>>>>>>> on the Name of that VM on the left (it will be s-XXXX-VM).
> >>>>> You'll
> >>>>>>> see
> >>>>>>>> a
> >>>>>>>>>>> bunch of information about the VM. Find the Host line and also
> >>>>> the
> >>>>>>>> Link
> >>>>>>>>>>> Local IP Address line and then follow the original instructions
> >>>>> in
> >>>>>>> my
> >>>>>>>>>>> previous email.
> >>>>>>>>>>>
> >>>>>>>>>>> - Si
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> ________________________________________
> >>>>>>>>>>> From: Shweta Agarwal <sh...@accelerite.com>
> >>>>>>>>>>> Sent: Wednesday, March 16, 2016 12:16 AM
> >>>>>>>>>>> To: users@cloudstack.apache.org
> >>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable
> >>>>> to
> >>>>>>>> start
> >>>>>>>>>>> instance due to Unable to get answer that is of class
> >>>>>>>>>>> com.cloud.agent.api.StartAnswer
> >>>>>>>>>>>
> >>>>>>>>>>> Hi Nezar,
> >>>>>>>>>>>
> >>>>>>>>>>> You can find host name on detail view of SSVM in UI .
> >>>>>>>>>>> You can also use list api command listSystemVms to get the host
> >>>>>>>> details
> >>>>>>>>>> of
> >>>>>>>>>>> system vms.
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks
> >>>>>>>>>>> Shweta
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> ________________________________________
> >>>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
> >>>>>>>>>>> Sent: Wednesday, March 16, 2016 9:58 AM
> >>>>>>>>>>> To: users@cloudstack.apache.org
> >>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable
> >>>>> to
> >>>>>>>> start
> >>>>>>>>>>> instance due to Unable to get answer that is of class
> >>>>>>>>>>> com.cloud.agent.api.StartAnswer
> >>>>>>>>>>>
> >>>>>>>>>>> Hi Simon,
> >>>>>>>>>>>
> >>>>>>>>>>> How can I find out which host the SSVM is on?
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks & Regards,
> >>>>>>>>>>> Nezar.
> >>>>>>>>>>>
> >>>>>>>>>>> On 16 March 2016 at 12:28, Simon Weller <sw...@ena.com>
> >>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Can you ssh into the Secondary Storage VM?
> >>>>>>>>>>>>
> >>>>>>>>>>>> This can be done by getting the link-local-ip address from the
> >>>>>>>>>>>> infrastructure menu in the UI and then sshing into the host
> >>>>> the
> >>>>>>> SSVM
> >>>>>>>>>> is
> >>>>>>>>>>> on
> >>>>>>>>>>>> and issuing this command as root: ssh -i
> >>>>> /root/.ssh/id_rsa.cloud
> >>>>>>> -p
> >>>>>>>>>> 3922
> >>>>>>>>>>>> root@link-local-ip (replace link-local-ip with the link
> >>>>> local
> >>>>>>> ip).
> >>>>>>>>>>>>
> >>>>>>>>>>>> Try running the SSVM diangnostic script:
> >>>>>>>>>>>> usr/local/cloud/systemvm/ssvm-check.sh
> >>>>>>>>>>>>
> >>>>>>>>>>>> - Si
> >>>>>>>>>>>> ________________________________________
> >>>>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
> >>>>>>>>>>>> Sent: Tuesday, March 15, 2016 7:49 PM
> >>>>>>>>>>>> To: users@cloudstack.apache.org
> >>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable
> >>>>>> to
> >>>>>>>>>> start
> >>>>>>>>>>>> instance due to Unable to get answer that is of class
> >>>>>>>>>>>> com.cloud.agent.api.StartAnswer
> >>>>>>>>>>>>
> >>>>>>>>>>>> Hi Simon,
> >>>>>>>>>>>>
> >>>>>>>>>>>> Please see details below:
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> mysql> use cloud; select id, name, status, type from host
> >>>>> where
> >>>>>>> id =
> >>>>>>>>>> 5;
> >>>>>>>>>>>> Reading table information for completion of table and column
> >>>>>> names
> >>>>>>>>>>>> You can turn off this feature to get a quicker startup with -A
> >>>>>>>>>>>>
> >>>>>>>>>>>> Database changed
> >>>>>>>>>>>> +----+------------------------------+--------+---------+
> >>>>>>>>>>>> | id | name | status | type |
> >>>>>>>>>>>> +----+------------------------------+--------+---------+
> >>>>>>>>>>>> | 5 | eqx-cs-cmp-02.ipscape.com.au | up | routing |
> >>>>>>>>>>>> +----+------------------------------+--------+---------+
> >>>>>>>>>>>> 1 row in set (0.00 sec)
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> cat /var/log/cloudstack/agent/agent.log
> >>>>>>>>>>>> com.cloud.utils.exception.CloudRuntimeException:
> >>>>>>>>>>>> com.cloud.utils.exception.CloudRuntimeException: Can't find
> >>>>>>>>>>>> volume:089ff3ca-9bff-402c-a14e-ea7a35aaad5c
> >>>>>>>>>>>> at
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>
> com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.getPhysicalDisk(KVMStoragePoolManager.java:262)
> >>>>>>>>>>>> at
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>
> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.createVbd(LibvirtComputingResource.java:3921)
> >>>>>>>>>>>> at
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>
> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(LibvirtComputingResource.java:3789)
> >>>>>>>>>>>> at
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>
> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1333)
> >>>>>>>>>>>> at
> >>>>> com.cloud.agent.Agent.processRequest(Agent.java:501)
> >>>>>>>>>>>> at
> >>>>>>>>>>>
> com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:808)
> >>>>>>>>>>>> at com.cloud.utils.nio.Task.run(Task.java:84)
> >>>>>>>>>>>> at
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> >>>>>>>>>>>> at
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> >>>>>>>>>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>>>>>>>> agent.log.2016-03-03.gz (END)
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> [root@eqx-cs-cmp-02 agent]# virsh pool-list
> >>>>>>>>>>>> Name State Autostart
> >>>>>>>>>>>> -----------------------------------------
> >>>>>>>>>>>> 2d0967f8-50d6-4d6d-87b7-4da0596f6924 active no
> >>>>>>>>>>>> b3647480-5709-36eb-9ddd-e2c666f30b57 active no
> >>>>>>>>>>>> e3831246-ff55-4af4-9964-a17565edaccc active no
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> [Root @ eqx-cs-cmp-02 agent] # df -h
> >>>>>>>>>>>> Filesystem Size Used Avail Use% Mounted on
> >>>>>>>>>>>> /dev/mapper/vg_root-lv_root
> >>>>>>>>>>>> 203G 215G 1.3G 1% /
> >>>>>>>>>>>> tmpfs 48g 48g 0 0% / dev / shm
> >>>>>>>>>>>> / Dev / sda1 477m 427m 25M 6% / boot
> >>>>>>>>>>>> xx.xx.xx.xx:/Cloudstack/primary
> >>>>>>>>>>>> 7.6T 4.1T 3.5T 54%
> >>>>>>>>>>>> /mnt/b3647480-5709-36eb-9ddd-e2c666f30b57
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> I also noticed the following in cloudstack-agent.err
> >>>>>>>>>>>>
> >>>>>>>>>>>> libvirt: QEMU Driver error : Domain not found: no domain with
> >>>>>>>> matching
> >>>>>>>>>>> name
> >>>>>>>>>>>> 'i-11-600-VM'
> >>>>>>>>>>>> libvirt: Storage Driver error : Storage volume not found: no
> >>>>>>> storage
> >>>>>>>>>> vol
> >>>>>>>>>>>> with matching name '089ff3ca-9bff-402c-a14e-ea7a35aaad5c'
> >>>>>>>>>>>>
> >>>>>>>>>>>> Your help is much appreciated,
> >>>>>>>>>>>> Nezar
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> On 16 March 2016 at 11:03, Simon Weller <sw...@ena.com>
> >>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Nezar,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> ok, I just took a look at a pastebin you posted.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I see it complaining about the inability to find a volume
> >>>>>> prior
> >>>>>>> to
> >>>>>>>>>>>>> throwing the host exception.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Firstly, let's find out who host 5 is. Can you log into your
> >>>>>>> MySQL
> >>>>>>>>>>> server
> >>>>>>>>>>>>> and run this:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> use cloud;select id,name,status,type from host where id=5;
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Can you post the results?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> If it's a host, can you post the
> >>>>>>>> /var/log/cloudstack/agent/agent.log
> >>>>>>>>>>> from
> >>>>>>>>>>>>> that host?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> And also run virsh pool-list on the host in question?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> - Si
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> ________________________________________
> >>>>>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
> >>>>>>>>>>>>> Sent: Tuesday, March 15, 2016 6:11 PM
> >>>>>>>>>>>>> To: users@cloudstack.apache.org
> >>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message:
> >>>>> Unable
> >>>>>>> to
> >>>>>>>>>> start
> >>>>>>>>>>>>> instance due to Unable to get answer that is of class
> >>>>>>>>>>>>> com.cloud.agent.api.StartAnswer
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Hi Simon,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> When you say "double check your routers tied to the
> >>>>> network",
> >>>>>> do
> >>>>>>>> you
> >>>>>>>>>>> mean
> >>>>>>>>>>>>> the physical routers or the virtual routers?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> When I try to create a VM in Prod Environment it fails. I
> >>>>>> can't
> >>>>>>>> ping
> >>>>>>>>>>> the
> >>>>>>>>>>>>> Prod virtual router from management server or Host.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> But I can ping Dev virtual router from management server and
> >>>>>>> Host
> >>>>>>>>>> and I
> >>>>>>>>>>>> was
> >>>>>>>>>>>>> able to create a VM in Dev Environment.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> How can I troubleshoot the Prod Virtual router. I launched
> >>>>> the
> >>>>>>>>>> console
> >>>>>>>>>>>> but
> >>>>>>>>>>>>> couldn't login.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Please note that we need to take into consideration that the
> >>>>>>>>>> affected
> >>>>>>>>>>>>> network is the Production network and we need to minimise
> >>>>>>> service
> >>>>>>>>>>> outage
> >>>>>>>>>>>> as
> >>>>>>>>>>>>> much as possible.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Your help is much appreciated,
> >>>>>>>>>>>>> Nezar
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On 16 March 2016 at 08:37, Simon Weller <sw...@ena.com>
> >>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> I've seen this before where a Virtual Router has crashed
> >>>>> and
> >>>>>>> the
> >>>>>>>>>>>>>> management server can't communicate with it.
> >>>>>>>>>>>>>> Are you using an advanced network setup?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> If so, double check your routers tied to the network where
> >>>>>> you
> >>>>>>>> are
> >>>>>>>>>>>>>> attempting to start an instance.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> - Si
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> ________________________________________
> >>>>>>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
> >>>>>>>>>>>>>> Sent: Tuesday, March 15, 2016 4:02 PM
> >>>>>>>>>>>>>> To: users@cloudstack.apache.org
> >>>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message:
> >>>>>> Unable
> >>>>>>>> to
> >>>>>>>>>>> start
> >>>>>>>>>>>>>> instance due to Unable to get answer that is of class
> >>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Hi Glenn,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> We have 16 hosts, all KVM.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> The error is not specific to Host 5. Every time I try to
> >>>>>>> create
> >>>>>>>> a
> >>>>>>>>>> VM
> >>>>>>>>>>> it
> >>>>>>>>>>>>>> gives me the same error message but on different host.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> How do I restart the agent on the host. Is it service
> >>>>>>> affecting?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks very much for your help.
> >>>>>>>>>>>>>> Neza
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On 15 March 2016 at 17:37, Glenn Wagner <
> >>>>>>>>>> glenn.wagner@shapeblue.com>
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hi,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Can you explain your environment first , how many hosts
> >>>>> ,
> >>>>>>> what
> >>>>>>>>>> is
> >>>>>>>>>>>> your
> >>>>>>>>>>>>>>> hypervisors (KVM,Xenserver,VMware)
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> The error seems to be with your Host 5 Hypervisor
> >>>>>>>>>>>>>>> Resource [Host:5] is unreachable:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Go to the infrastructure section and see what host5 is
> >>>>>>>>>> reporting,
> >>>>>>>>>>>>>>> Have you tried to restart the agent on the Host?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks
> >>>>>>>>>>>>>>> Glenn
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> [image: ShapeBlue] <http://www.shapeblue.com>
> >>>>>>>>>>>>>>> Glenn Wagner
> >>>>>>>>>>>>>>> Senior Consultant , ShapeBlue
> >>>>>>>>>>>>>>> d: * | s: +27 21 527 0091*
> >>>>>>>>>> <%7C%20s:%20+27%2021%20527%200091> |
> >>>>>>>>>>>> m:
> >>>>>>>>>>>>>>> *+27 73 917 4111* <+27%2073%20917%204111>
> >>>>>>>>>>>>>>> e: *glenn.wagner@shapeblue.com | t: *
> >>>>>>>>>>>>>>> <glenn.wagner@shapeblue.com%20%7C%20t:> | w: *
> >>>>>>>>>> www.shapeblue.com
> >>>>>>>>>>> *
> >>>>>>>>>>>>>>> <http://www.shapeblue.com>
> >>>>>>>>>>>>>>> a:
> >>>>>>>>>>>>>>> 2nd Floor, Oudehuis Centre, 122 Main Rd, Somerset West
> >>>>>> Cape
> >>>>>>>> Town
> >>>>>>>>>>> 7130
> >>>>>>>>>>>>>> South Africa
> >>>>>>>>>>>>>>> Shape Blue Ltd is a company incorporated in England &
> >>>>>> Wales.
> >>>>>>>>>>>> ShapeBlue
> >>>>>>>>>>>>>>> Services India LLP is a company incorporated in India
> >>>>> and
> >>>>>> is
> >>>>>>>>>>> operated
> >>>>>>>>>>>>>> under
> >>>>>>>>>>>>>>> license from Shape Blue Ltd. Shape Blue Brasil
> >>>>> Consultoria
> >>>>>>>> Ltda
> >>>>>>>>>> is
> >>>>>>>>>>> a
> >>>>>>>>>>>>>>> company incorporated in Brasil and is operated under
> >>>>>> license
> >>>>>>>>>> from
> >>>>>>>>>>>> Shape
> >>>>>>>>>>>>>>> Blue Ltd. ShapeBlue SA Pty Ltd is a company registered
> >>>>> by
> >>>>>>> The
> >>>>>>>>>>>> Republic
> >>>>>>>>>>>>> of
> >>>>>>>>>>>>>>> South Africa and is traded under license from Shape Blue
> >>>>>>> Ltd.
> >>>>>>>>>>>> ShapeBlue
> >>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>> a registered trademark.
> >>>>>>>>>>>>>>> This email and any attachments to it may be confidential
> >>>>>> and
> >>>>>>>> are
> >>>>>>>>>>>>> intended
> >>>>>>>>>>>>>>> solely for the use of the individual to whom it is
> >>>>>>> addressed.
> >>>>>>>>>> Any
> >>>>>>>>>>>> views
> >>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>> opinions expressed are solely those of the author and do
> >>>>>> not
> >>>>>>>>>>>>> necessarily
> >>>>>>>>>>>>>>> represent those of Shape Blue Ltd or related companies.
> >>>>> If
> >>>>>>> you
> >>>>>>>>>> are
> >>>>>>>>>>>> not
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>> intended recipient of this email, you must neither take
> >>>>>> any
> >>>>>>>>>> action
> >>>>>>>>>>>>> based
> >>>>>>>>>>>>>>> upon its contents, nor copy or show it to anyone. Please
> >>>>>>>> contact
> >>>>>>>>>>> the
> >>>>>>>>>>>>>> sender
> >>>>>>>>>>>>>>> if you believe you have received this email in error.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> -----Original Message-----
> >>>>>>>>>>>>>>> From: Nezar Madbouh [mailto:
> >>>>> nezar.madbouh@ipscape.com.au]
> >>>>>>>>>>>>>>> Sent: Tuesday, 15 March 2016 5:45 AM
> >>>>>>>>>>>>>>> To: users@cloudstack.apache.org
> >>>>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message:
> >>>>>>> Unable
> >>>>>>>>>> to
> >>>>>>>>>>>> start
> >>>>>>>>>>>>>>> instance due to Unable to get answer that is of class
> >>>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hi Jeremy,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I have uploaded the contents to
> >>>>>>> http://pastebin.com/1J3mmgTh
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> your help is much appreciated,
> >>>>>>>>>>>>>>> Nezar.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On 15 March 2016 at 13:46, Jeremy Peterson <
> >>>>>>>>>> jpeterson@acentek.net>
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> I don't see any attachments can you create a pastebin.
> >>>>>>>> That's
> >>>>>>>>>>>> easier
> >>>>>>>>>>>>>>>> to look through as I think attachments are removed
> >>>>>> before
> >>>>>>>>>> being
> >>>>>>>>>>>> sent
> >>>>>>>>>>>>>>>> to user group.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Jeremy
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> -------- Original message --------
> >>>>>>>>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
> >>>>>>>>>>>>>>>> Date: 3/14/2016 6:38 PM (GMT-06:00)
> >>>>>>>>>>>>>>>> To: users@cloudstack.apache.org
> >>>>>>>>>>>>>>>> Subject: Unable to create a New VM - Error message:
> >>>>>> Unable
> >>>>>>>> to
> >>>>>>>>>>> start
> >>>>>>>>>>>>>>>> instance due to Unable to get answer that is of class
> >>>>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Hi Shweta and Sadhu,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Please find attached managerment-server.log which
> >>>>>> contains
> >>>>>>>> the
> >>>>>>>>>>> logs
> >>>>>>>>>>>>>>>> since I started creating a new VM (VM id 600).
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Please note that we don't have agent.log files
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Your help is much appreciated.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Regards,
> >>>>>>>>>>>>>>>> Nezar.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Find out more about ShapeBlue and our range of
> >>>>> CloudStack
> >>>>>>>>>> related
> >>>>>>>>>>>>>> services:
> >>>>>>>>>>>>>>> IaaS Cloud Design & Build
> >>>>>>>>>>>>>>> <http://shapeblue.com/iaas-cloud-design-and-build//> |
> >>>>>>>> CSForge
> >>>>>>>>>> -
> >>>>>>>>>>>> rapid
> >>>>>>>>>>>>>>> IaaS deployment framework <
> >>>>> http://shapeblue.com/csforge/>
> >>>>>>>>>>>>>>> CloudStack Consulting <
> >>>>>>>>>>> http://shapeblue.com/cloudstack-consultancy/>
> >>>>>>>>>>>> |
> >>>>>>>>>>>>>> CloudStack
> >>>>>>>>>>>>>>> Software Engineering
> >>>>>>>>>>>>>>> <http://shapeblue.com/cloudstack-software-engineering/>
> >>>>>>>>>>>>>>> CloudStack Infrastructure Support
> >>>>>>>>>>>>>>> <
> >>>>> http://shapeblue.com/cloudstack-infrastructure-support/>
> >>>>>> |
> >>>>>>>>>>>> CloudStack
> >>>>>>>>>>>>>>> Bootcamp Training Courses <
> >>>>>>>>>>> http://shapeblue.com/cloudstack-training/
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> DISCLAIMER
> >>>>>>>>>>> ==========
> >>>>>>>>>>> This e-mail may contain privileged and confidential information
> >>>>>>> which
> >>>>>>>> is
> >>>>>>>>>>> the property of Accelerite, a Persistent Systems business. It
> is
> >>>>>>>>>> intended
> >>>>>>>>>>> only for the use of the individual or entity to which it is
> >>>>>>> addressed.
> >>>>>>>>>> If
> >>>>>>>>>>> you are not the intended recipient, you are not authorized to
> >>>>>> read,
> >>>>>>>>>> retain,
> >>>>>>>>>>> copy, print, distribute or use this message. If you have
> >>>>> received
> >>>>>>> this
> >>>>>>>>>>> communication in error, please notify the sender and delete all
> >>>>>>> copies
> >>>>>>>>>> of
> >>>>>>>>>>> this message. Accelerite, a Persistent Systems business does
> not
> >>>>>>>> accept
> >>>>>>>>>> any
> >>>>>>>>>>> liability for virus infected mails.
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>
> >>>
> >>
> >>
> >
>
>
Re: Unable to create a New VM - Error message: Unable to start instance
due to Unable to get answer that is of class com.cloud.agent.api.StartAnswer
Posted by ilya <il...@gmail.com>.
Here is how i would troubleshoot this:
1) ssh to MS host, run
tail -f /var/cloudstack/management/management-server.log | grep -i
StartVMCmdByAdmin
2) on cloudstack UI, press start on the VM and watch the tail log
3) on the tail log, you should see a StartVirtualMachine api call show
up with job ID. For example:
> 2016-04-15 03:41:57,193 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-24:ctx-7892387a job-291575) Executing AsyncJobVO {id:291575, userId: 157, accountId: 2, instanceType: VirtualMachine, instanceId: 7648, cmd: org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin, cmdInfo: {"id":"7648","response":"json","sessionkey":"mMJsduS9KLKApqH4dPpIlGZdt1Y\u003d","ctxDetails":"{\"com.cloud.vm.VirtualMachine\":7648,\"com.cloud.uservm.UserVm\":7648,\"com.cloud.network.router.VirtualRouter\":7648}","cmdEventType":"VM.START","expires":"2016-04-15T03:51:57+0000","ctxUserId":"157","signatureversion":"3","httpmethod":"GET","uuid":"7648","ctxAccountId":"2","ctxStartEventId":"314416"}, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 345051337894, completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
Each task in cloudstack spins up its own job, in this case its *job-291575*
4) now execute
> grep -A3 -B3 job-291575 /var/cloudstack/management/management-server.log | tee job-291575.log
> grep job-291575 /var/cloudstack/management/management-server.log | grep via | head -1
you should see a response like so,
> 2016-04-15 03:41:59,259 DEBUG [c.c.a.m.ClusteredAgentAttache] (Work-Job-Executor-42:ctx-b2c4af6b job-291575/job-291576 ctx-0988152e) Seq 441-4181310779036795138: Forwarding Seq 441-4181310779036795138: { Cmd , MgmtId: 345051337894, via: 441(my-kvm-host01.example.com), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.routing.DhcpEntryCommand":{"vmMac":"06:fe:4e:00:12:ad","vmIpAddress":"10.10.27.178","vmName":"ol6-test-ilya1","defaultRouter":"10.10.27.1","defaultDns":"10.17.27.166","duid":"00:03:00:01:06:fe:4e:00:12:ad","isDefault":true,"executeInSequence":false,"accessDetails":{"zone.network.type":"Advanced","router.guest.ip":"10.10.27.166","router.ip":"169.254.0.238","router.name":"r-7301-VM"},"wait":0}}] } to 345051337893
We are looking for this line "via: 441(my-kvm-host01.example.com)"
SSH to my-kvm-host01.example.com, and copy the
/var/log/cloudstack/agent.log to pastebin
also copy job-291575.log to pastebin
I'll try to have a look, though when i get chance.
Also paste the output of mount command on kvm host in question. I'm
suspecting one of your storage pools maybe bad.
Lastly, what distro are you running for KVM?
Regards
ilya
On 4/14/16 5:05 PM, Nezar Madbouh wrote:
> Hi Ilya,
>
> I shutdown all the VMs in Pre-prod Environmnet expecting the Pre-prod
> Virtual router to be deleted automatically because no VM is running on this
> vlan but it wasn't deleted.
>
> Am I right to expect the VR to be deleted automatically if no VM is running
> in that vlan?
>
> Thanks,
> Nezar.
>
> On 15 April 2016 at 09:53, Nezar Madbouh <ne...@ipscape.com.au>
> wrote:
>
>> Hi Ilya,
>>
>> I have run the script and it didn't return any results:
>>
>> [root@eqx-cs-mgt-01 ~]# ./async_jobs.sh
>> [root@eqx-cs-mgt-01 ~]#
>>
>>
>> I also ran the command inside mysql:
>>
>> mysql> SELECT id, user_id, account_id, instance_type, instance_id,
>> job_cmd, job_cmd_info, job_dispatcher, job_pending_signals FROM
>> cloud.async_job where job_status=0 and job_dispatcher not like
>> "pseudoJobDispatcher";
>> Empty set (0.00 sec)
>>
>> your help is much appreciated,
>> Nezar.
>>
>>
>> On 14 April 2016 at 16:49, ilya <il...@gmail.com> wrote:
>>
>>> It appears you have pending jobs that are not letting you go through.
>>>
>>> Usually stop and starting management service should be enough.
>>>
>>> However, try running this script from your management server to show us
>>> what you have running in async_job table.
>>>
>>>
>>>
>>>> #!/bin/bash
>>>> DATESTAMP=$(date +%m%d%y-%H%M%S)
>>>> DBPASS=$(java -classpath
>>> /usr/share/cloudstack-common/lib/jasypt-1.9.0.jar
>>> org.jasypt.intf.cli.JasyptPBEStringDecryptionCLI input="$(cat
>>> /etc/cloudstack/management/db.properties | grep db.cloud.password | awk
>>> -F'(' '{print $2}' | sed 's/)//g')" password="$(cat
>>> /etc/cloudstack/management/key)" | grep -A2 OUTPUT | tail -1)
>>>> DBHOST=$(cat /etc/cloudstack/management/db.properties | grep
>>> db.cloud.host | awk -F'=' '{print $2}' | tail -1 )
>>>> DBUSER=$(cat /etc/cloudstack/management/db.properties | grep
>>> db.cloud.username | awk -F'=' '{print $2}')
>>>> DB=$(cat /etc/cloudstack/management/db.properties | grep db.cloud.name
>>> | awk -F'=' '{print $2}')
>>>> DBPORT=$(cat /etc/cloudstack/management/db.properties | grep
>>> db.cloud.port | awk -F'=' '{print $2}')
>>>> MYSQLCMD="mysql -h $DBHOST -u $DBUSER -P $DBPORT -p$DBPASS $DB"
>>>> echo 'SELECT id, user_id, account_id, instance_type, instance_id,
>>> job_cmd, job_cmd_info, job_dispatcher, job_pending_signals FROM
>>> cloud.async_job where job_status=0 and job_dispatcher not like
>>> "pseudoJobDispatcher"' | $MYSQLCMD
>>>
>>>
>>>
>>>
>>> On 4/13/16 5:16 PM, Nezar Madbouh wrote:
>>>> Hi Simon,
>>>>
>>>> The following work was done but unfortunately the problem is not fixed:
>>>>
>>>>
>>>> on management server:
>>>>
>>>> service cloudstack-management stop
>>>>
>>>>
>>>> on the host:
>>>>
>>>> service cloudstack-agent stop
>>>>
>>>> service libvirtd restart
>>>>
>>>> service cloudstack-agent start
>>>>
>>>>
>>>> on management server:
>>>>
>>>> service cloudstack-management start
>>>>
>>>> Regards,
>>>> Nezar.
>>>>
>>>> On 24 March 2016 at 05:06, Simon Weller <sw...@ena.com> wrote:
>>>>
>>>>> Yes, I think that order is fine. Make sure the processes are indeed
>>>>> shutdown before moving on to the next step.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ________________________________________
>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>>>>> Sent: Tuesday, March 22, 2016 5:33 PM
>>>>> To: users@cloudstack.apache.org
>>>>> Subject: Re: Unable to create a New VM - Error message: Unable to start
>>>>> instance due to Unable to get answer that is of class
>>>>> com.cloud.agent.api.StartAnswer
>>>>>
>>>>> Hi Simon,
>>>>>
>>>>> Can you please check if the following sequence of action is correct:
>>>>>
>>>>> on management server:
>>>>>
>>>>> service cloudstack-management stop
>>>>>
>>>>>
>>>>> on the host:
>>>>>
>>>>> service cloudstack-agent stop
>>>>>
>>>>> service libvirtd restart
>>>>>
>>>>> service cloudstack-agent start
>>>>>
>>>>>
>>>>> on management server:
>>>>>
>>>>> service cloudstack-management start
>>>>>
>>>>> Thanks very much for your help.
>>>>> Nezar.
>>>>>
>>>>> On 23 March 2016 at 08:39, Simon Weller <sw...@ena.com> wrote:
>>>>>
>>>>>> Shutting down all the management services should clear the jobs.
>>>>>>
>>>>>>
>>>>>> ________________________________________
>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>>>>>> Sent: Tuesday, March 22, 2016 4:28 PM
>>>>>> To: users@cloudstack.apache.org
>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable to
>>> start
>>>>>> instance due to Unable to get answer that is of class
>>>>>> com.cloud.agent.api.StartAnswer
>>>>>>
>>>>>> Hi Simon,
>>>>>>
>>>>>> I will plan to restart agents, libvirtd and management services.
>>>>>>
>>>>>> How do I clear hung jobs.
>>>>>>
>>>>>> Your help is much appreciated,
>>>>>> Nezar.
>>>>>>
>>>>>> On 22 March 2016 at 23:03, Simon Weller <sw...@ena.com> wrote:
>>>>>>
>>>>>>> Nezar,
>>>>>>>
>>>>>>> I've looked through the code related to your release and it seems
>>> that
>>>>> in
>>>>>>> order to throw this exception, 10 attempts have been made to retrieve
>>>>> the
>>>>>>> disk information prior to the exception being thrown.
>>>>>>>
>>>>>>> It appears there is something really odd going on here. I'm starting
>>>>> to
>>>>>>> wonder whether it might be worth while for you to stop all of your
>>>>>> agents,
>>>>>>> plus management server processes, restart libvirtd on each of the
>>> hosts
>>>>>> and
>>>>>>> then bring the agents back up prior to starting the management
>>> servers.
>>>>>>>
>>>>>>> There are a bunch of logs the pastebin you posted that seems to
>>>>> indicate
>>>>>>> you have some jobs hung, for example:
>>>>>>>
>>>>>>> 2016-03-14 21:45:31,828 WARN [o.a.c.f.j.i.AsyncJobMonitor]
>>>>>>> (Timer-1:ctx-0c090bd7) Task (job-9066) has been pending for 1660851
>>>>>> seconds
>>>>>>> 2016-03-14 21:45:31,828 WARN [o.a.c.f.j.i.AsyncJobMonitor]
>>>>>>> (Timer-1:ctx-0c090bd7) Task (job-8133) has been pending for 4253003
>>>>>> seconds
>>>>>>>
>>>>>>> Restarting the agents, libvirtd and management will not cause an
>>>>> outage
>>>>>>> to your cloud, but I would suggest, if you decided to do this, you
>>>>> should
>>>>>>> complete it in a maintenance window.
>>>>>>>
>>>>>>> - Si
>>>>>>>
>>>>>>> ________________________________________
>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>>>>>>> Sent: Sunday, March 20, 2016 10:27 PM
>>>>>>> To: users@cloudstack.apache.org
>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable to
>>> start
>>>>>>> instance due to Unable to get answer that is of class
>>>>>>> com.cloud.agent.api.StartAnswer
>>>>>>>
>>>>>>> Hi Simon,
>>>>>>>
>>>>>>> Please see answers to your questions below. I am not sure if I
>>> provided
>>>>>> the
>>>>>>> correct answer for the primary and secondary storage configuration.
>>>>>> Please
>>>>>>> let me know if this is not what you are after:
>>>>>>>
>>>>>>>
>>>>>>> Can you screenshot and post somewhere your secondary storage
>>>>>> configuration
>>>>>>> and your primary storage configuration from the UI with the details?
>>>>>>>
>>>>>>> mysql> select * from configuration where name like '%secondary%'\G
>>>>>>> *************************** 1. row ***************************
>>>>>>> category: Account Defaults
>>>>>>> instance: DEFAULT
>>>>>>> component: management-server
>>>>>>> name: max.account.secondary.storage
>>>>>>> value: 400
>>>>>>> description: The default maximum secondary storage space (in GiB)
>>>>> that
>>>>>>> can be used for an account
>>>>>>> default_value: 400
>>>>>>> updated: NULL
>>>>>>> scope: NULL
>>>>>>> is_dynamic: 0
>>>>>>> *************************** 2. row ***************************
>>>>>>> category: Project Defaults
>>>>>>> instance: DEFAULT
>>>>>>> component: management-server
>>>>>>> name: max.project.secondary.storage
>>>>>>> value: 400
>>>>>>> description: The default maximum secondary storage space (in GiB)
>>>>> that
>>>>>>> can be used for a project
>>>>>>> default_value: 400
>>>>>>> updated: NULL
>>>>>>> scope: NULL
>>>>>>> is_dynamic: 0
>>>>>>> *************************** 3. row ***************************
>>>>>>> category: Hidden
>>>>>>> instance: DEFAULT
>>>>>>> component: management-server
>>>>>>> name: secondary.storage.vm
>>>>>>> value: +rwRhJOXPymbQOuAMLnhiw==
>>>>>>> description: Deploys a VM per zone to manage secondary storage if
>>>>> true,
>>>>>>> otherwise secondary storage is mounted on management server
>>>>>>> default_value: NULL
>>>>>>> updated: NULL
>>>>>>> scope: NULL
>>>>>>> is_dynamic: 0
>>>>>>> *************************** 4. row ***************************
>>>>>>> category: Advanced
>>>>>>> instance: DEFAULT
>>>>>>> component: ExtensionRegistry
>>>>>>> name: secondary.storage.vm.allocators.exclude
>>>>>>> value: NULL
>>>>>>> description: Extensions to exclude from being registered
>>>>>>> default_value: NULL
>>>>>>> updated: 2014-12-04 00:18:15
>>>>>>> scope: Global
>>>>>>> is_dynamic: 0
>>>>>>> *************************** 5. row ***************************
>>>>>>> category: NetworkManager
>>>>>>> instance: DEFAULT
>>>>>>> component: management-server
>>>>>>> name: vm.network.nic.max.secondary.ipaddresses
>>>>>>> value: NULL
>>>>>>> description: Specify the number of secondary ip addresses per nic
>>> per
>>>>>> vm
>>>>>>> default_value: 256
>>>>>>> updated: NULL
>>>>>>> scope: NULL
>>>>>>> is_dynamic: 0
>>>>>>> 5 rows in set (0.00 sec)
>>>>>>>
>>>>>>>
>>>>>>> mysql> select * from configuration where name like '%primary%'\G
>>>>>>> *************************** 1. row ***************************
>>>>>>> category: Account Defaults
>>>>>>> instance: DEFAULT
>>>>>>> component: management-server
>>>>>>> name: max.account.primary.storage
>>>>>>> value: 200
>>>>>>> description: The default maximum primary storage space (in GiB)
>>> that
>>>>>> can
>>>>>>> be used for an account
>>>>>>> default_value: 200
>>>>>>> updated: NULL
>>>>>>> scope: NULL
>>>>>>> is_dynamic: 0
>>>>>>> *************************** 2. row ***************************
>>>>>>> category: Project Defaults
>>>>>>> instance: DEFAULT
>>>>>>> component: management-server
>>>>>>> name: max.project.primary.storage
>>>>>>> value: 200
>>>>>>> description: The default maximum primary storage space (in GiB)
>>> that
>>>>>> can
>>>>>>> be used for a project
>>>>>>> default_value: 200
>>>>>>> updated: NULL
>>>>>>> scope: NULL
>>>>>>> is_dynamic: 0
>>>>>>> *************************** 3. row ***************************
>>>>>>> category: Storage
>>>>>>> instance: DEFAULT
>>>>>>> component: TemplateManager
>>>>>>> name: primary.storage.download.wait
>>>>>>> value: 10800
>>>>>>> description: In second, timeout for download template to primary
>>>>>> storage
>>>>>>> default_value: NULL
>>>>>>> updated: NULL
>>>>>>> scope: NULL
>>>>>>> is_dynamic: 0
>>>>>>> 3 rows in set (0.00 sec)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Also, what is the Cloudstack version you are running and what is the
>>> OS
>>>>>>> you're running it on?
>>>>>>>
>>>>>>> Cloudstack 4.4.1
>>>>>>>
>>>>>>> Centos 6.6
>>>>>>>
>>>>>>> Are you using actimeo=0 in your NFS mount line?
>>>>>>>
>>>>>>> Primary storage on Hypervisors has the following parameters
>>>>>>> 10.61.154.10:/Cloudstack/primary on
>>>>>>> /mnt/b3647480-5709-36eb-9ddd-e2c666f30b57 type nfs
>>>>>>> (rw,noatime,nfsvers=3,proto=tcp,hard,nolock,intr,addr=10.61.154.10)
>>>>>>>
>>>>>>>
>>>>>>> Secondary storage on SSVM has the following parameters
>>>>>>> 10.61.154.10:/Cloudstack/secondary on
>>>>>>> /mnt/SecStorage/d11d5c92-6950-30e8-a0b5-3cd46a17ec14 type nfs4
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>> (rw,relatime,vers=4,rsize=262144,wsize=262144,namlen=255,acdirmin=0,acdirmax=0,soft,proto=tcp,port=0,timeo=133,retrans=
>>> 2147483647
>>> ,sec=sys,clientaddr=10.61.154.70,minorversion=0,local_lock=none,addr=10.61.154.10)
>>>>>>>
>>>>>>>
>>>>>>> Thanks & Regards,
>>>>>>> Nezar.
>>>>>>>
>>>>>>> On 18 March 2016 at 23:50, Simon Weller <sw...@ena.com> wrote:
>>>>>>>
>>>>>>>> Also, what is the Cloudstack version you are running and what is the
>>>>> OS
>>>>>>>> you're running it on?
>>>>>>>>
>>>>>>>> Are you using actimeo=0 in your NFS mount line?
>>>>>>>>
>>>>>>>> ________________________________________
>>>>>>>> From: Simon Weller <sw...@ena.com>
>>>>>>>> Sent: Friday, March 18, 2016 7:40 AM
>>>>>>>> To: users@cloudstack.apache.org
>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable to
>>>>> start
>>>>>>>> instance due to Unable to get answer that is of class
>>>>>>>> com.cloud.agent.api.StartAnswer
>>>>>>>>
>>>>>>>> Nexar,
>>>>>>>>
>>>>>>>> That's normal behaviour. You can only ssh to the link local on the
>>>>> host
>>>>>>>> where the VM resides.
>>>>>>>>
>>>>>>>> Can you screenshot and post somewhere your secondary storage
>>>>>>> configuration
>>>>>>>> and your primary storage configuration from the UI with the details?
>>>>>>>>
>>>>>>>> - Si
>>>>>>>>
>>>>>>>> ________________________________________
>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>>>>>>>> Sent: Thursday, March 17, 2016 8:19 PM
>>>>>>>> To: users@cloudstack.apache.org
>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable to
>>>>> start
>>>>>>>> instance due to Unable to get answer that is of class
>>>>>>>> com.cloud.agent.api.StartAnswer
>>>>>>>>
>>>>>>>> Hi Simon,
>>>>>>>>
>>>>>>>> I just noticed that from other Hosts I can't ssh into the SSVM:
>>>>>>>>
>>>>>>>> [root@eqx-cs-cmp-01 ~]# ssh -i /root/.ssh/id_rsa.cloud -p 3922
>>>>>>>> root@169.254.3.161
>>>>>>>> ssh: connect to host 169.254.3.161 port 3922: No route to host
>>>>>>>>
>>>>>>>>
>>>>>>>> I can only ssh from one Host:
>>>>>>>> [root@eqx-cs-cmp-06 ~]# ssh -i /root/.ssh/id_rsa.cloud -p 3922
>>>>>>>> root@169.254.3.161
>>>>>>>> Linux s-11-VM 3.2.0-4-amd64 #1 SMP Debian 3.2.57-3 x86_64
>>>>>>>>
>>>>>>>> Thanks & Regards,
>>>>>>>> Nezar.
>>>>>>>>
>>>>>>>> On 18 March 2016 at 10:36, Nezar Madbouh <
>>>>> nezar.madbouh@ipscape.com.au
>>>>>>>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi Simon,
>>>>>>>>>
>>>>>>>>> Yes existing VMs are successfully running. Yesterday I was able to
>>>>>>> create
>>>>>>>>> a VM in Dev Environment but I can't create VMs in Prod environment.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Nezar.
>>>>>>>>>
>>>>>>>>> On 18 March 2016 at 09:12, Simon Weller <sw...@ena.com> wrote:
>>>>>>>>>
>>>>>>>>>> I'm assuming you have other VMs successfully running, correct? So
>>>>>> you
>>>>>>>> can
>>>>>>>>>> write to your primary storage.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ________________________________________
>>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>>>>>>>>>> Sent: Thursday, March 17, 2016 4:43 PM
>>>>>>>>>> To: users@cloudstack.apache.org
>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable to
>>>>>>> start
>>>>>>>>>> instance due to Unable to get answer that is of class
>>>>>>>>>> com.cloud.agent.api.StartAnswer
>>>>>>>>>>
>>>>>>>>>> Hi Simon and Shweta,
>>>>>>>>>>
>>>>>>>>>> Please see the result below:
>>>>>>>>>>
>>>>>>>>>> root@s-11-VM:~# /usr/local/cloud/systemvm/ssvm-check.sh
>>>>>>>>>> ================================================
>>>>>>>>>> First DNS server is 8.8.8.8
>>>>>>>>>> PING 8.8.8.8 (8.8.8.8): 48 data bytes
>>>>>>>>>> 56 bytes from 8.8.8.8: icmp_seq=0 ttl=57 time=5.209 ms
>>>>>>>>>> 56 bytes from 8.8.8.8: icmp_seq=1 ttl=57 time=0.936 ms
>>>>>>>>>> --- 8.8.8.8 ping statistics ---
>>>>>>>>>> 2 packets transmitted, 2 packets received, 0% packet loss
>>>>>>>>>> round-trip min/avg/max/stddev = 0.936/3.072/5.209/2.137 ms
>>>>>>>>>> Good: Can ping DNS server
>>>>>>>>>> ================================================
>>>>>>>>>> Good: DNS resolves download.cloud.com
>>>>>>>>>> ================================================
>>>>>>>>>> nfs is currently mounted
>>>>>>>>>> Mount point is
>>>>> /mnt/SecStorage/d11d5c92-6950-30e8-a0b5-3cd46a17ec14
>>>>>>>>>> Good: Can write to mount point
>>>>>>>>>> ================================================
>>>>>>>>>> Management server is 10.61.153.20. Checking connectivity.
>>>>>>>>>> Good: Can connect to management server port 8250
>>>>>>>>>> ================================================
>>>>>>>>>> Good: Java process is running
>>>>>>>>>> ================================================
>>>>>>>>>> Tests Complete. Look for ERROR or WARNING above.
>>>>>>>>>> root@s-11-VM:~#
>>>>>>>>>>
>>>>>>>>>> Thanks & Regards,
>>>>>>>>>> Nezar.
>>>>>>>>>>
>>>>>>>>>> On 17 March 2016 at 09:42, Simon Weller <sw...@ena.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Nezar,
>>>>>>>>>>>
>>>>>>>>>>> Log into the Cloudstack UI.
>>>>>>>>>>>
>>>>>>>>>>> Click on "Infrastructure" on the left menu. Click on View All
>>>>>> below
>>>>>>>>>>> "System VMs". find the item marked "Secondary Storage VM" and
>>>>>> then
>>>>>>>>>> click
>>>>>>>>>>> on the Name of that VM on the left (it will be s-XXXX-VM).
>>>>> You'll
>>>>>>> see
>>>>>>>> a
>>>>>>>>>>> bunch of information about the VM. Find the Host line and also
>>>>> the
>>>>>>>> Link
>>>>>>>>>>> Local IP Address line and then follow the original instructions
>>>>> in
>>>>>>> my
>>>>>>>>>>> previous email.
>>>>>>>>>>>
>>>>>>>>>>> - Si
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> ________________________________________
>>>>>>>>>>> From: Shweta Agarwal <sh...@accelerite.com>
>>>>>>>>>>> Sent: Wednesday, March 16, 2016 12:16 AM
>>>>>>>>>>> To: users@cloudstack.apache.org
>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable
>>>>> to
>>>>>>>> start
>>>>>>>>>>> instance due to Unable to get answer that is of class
>>>>>>>>>>> com.cloud.agent.api.StartAnswer
>>>>>>>>>>>
>>>>>>>>>>> Hi Nezar,
>>>>>>>>>>>
>>>>>>>>>>> You can find host name on detail view of SSVM in UI .
>>>>>>>>>>> You can also use list api command listSystemVms to get the host
>>>>>>>> details
>>>>>>>>>> of
>>>>>>>>>>> system vms.
>>>>>>>>>>>
>>>>>>>>>>> Thanks
>>>>>>>>>>> Shweta
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> ________________________________________
>>>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>>>>>>>>>>> Sent: Wednesday, March 16, 2016 9:58 AM
>>>>>>>>>>> To: users@cloudstack.apache.org
>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable
>>>>> to
>>>>>>>> start
>>>>>>>>>>> instance due to Unable to get answer that is of class
>>>>>>>>>>> com.cloud.agent.api.StartAnswer
>>>>>>>>>>>
>>>>>>>>>>> Hi Simon,
>>>>>>>>>>>
>>>>>>>>>>> How can I find out which host the SSVM is on?
>>>>>>>>>>>
>>>>>>>>>>> Thanks & Regards,
>>>>>>>>>>> Nezar.
>>>>>>>>>>>
>>>>>>>>>>> On 16 March 2016 at 12:28, Simon Weller <sw...@ena.com>
>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Can you ssh into the Secondary Storage VM?
>>>>>>>>>>>>
>>>>>>>>>>>> This can be done by getting the link-local-ip address from the
>>>>>>>>>>>> infrastructure menu in the UI and then sshing into the host
>>>>> the
>>>>>>> SSVM
>>>>>>>>>> is
>>>>>>>>>>> on
>>>>>>>>>>>> and issuing this command as root: ssh -i
>>>>> /root/.ssh/id_rsa.cloud
>>>>>>> -p
>>>>>>>>>> 3922
>>>>>>>>>>>> root@link-local-ip (replace link-local-ip with the link
>>>>> local
>>>>>>> ip).
>>>>>>>>>>>>
>>>>>>>>>>>> Try running the SSVM diangnostic script:
>>>>>>>>>>>> usr/local/cloud/systemvm/ssvm-check.sh
>>>>>>>>>>>>
>>>>>>>>>>>> - Si
>>>>>>>>>>>> ________________________________________
>>>>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>>>>>>>>>>>> Sent: Tuesday, March 15, 2016 7:49 PM
>>>>>>>>>>>> To: users@cloudstack.apache.org
>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable
>>>>>> to
>>>>>>>>>> start
>>>>>>>>>>>> instance due to Unable to get answer that is of class
>>>>>>>>>>>> com.cloud.agent.api.StartAnswer
>>>>>>>>>>>>
>>>>>>>>>>>> Hi Simon,
>>>>>>>>>>>>
>>>>>>>>>>>> Please see details below:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> mysql> use cloud; select id, name, status, type from host
>>>>> where
>>>>>>> id =
>>>>>>>>>> 5;
>>>>>>>>>>>> Reading table information for completion of table and column
>>>>>> names
>>>>>>>>>>>> You can turn off this feature to get a quicker startup with -A
>>>>>>>>>>>>
>>>>>>>>>>>> Database changed
>>>>>>>>>>>> +----+------------------------------+--------+---------+
>>>>>>>>>>>> | id | name | status | type |
>>>>>>>>>>>> +----+------------------------------+--------+---------+
>>>>>>>>>>>> | 5 | eqx-cs-cmp-02.ipscape.com.au | up | routing |
>>>>>>>>>>>> +----+------------------------------+--------+---------+
>>>>>>>>>>>> 1 row in set (0.00 sec)
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> cat /var/log/cloudstack/agent/agent.log
>>>>>>>>>>>> com.cloud.utils.exception.CloudRuntimeException:
>>>>>>>>>>>> com.cloud.utils.exception.CloudRuntimeException: Can't find
>>>>>>>>>>>> volume:089ff3ca-9bff-402c-a14e-ea7a35aaad5c
>>>>>>>>>>>> at
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>> com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.getPhysicalDisk(KVMStoragePoolManager.java:262)
>>>>>>>>>>>> at
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.createVbd(LibvirtComputingResource.java:3921)
>>>>>>>>>>>> at
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(LibvirtComputingResource.java:3789)
>>>>>>>>>>>> at
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1333)
>>>>>>>>>>>> at
>>>>> com.cloud.agent.Agent.processRequest(Agent.java:501)
>>>>>>>>>>>> at
>>>>>>>>>>> com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:808)
>>>>>>>>>>>> at com.cloud.utils.nio.Task.run(Task.java:84)
>>>>>>>>>>>> at
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>>>>>>>> at
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>>>>>>> agent.log.2016-03-03.gz (END)
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> [root@eqx-cs-cmp-02 agent]# virsh pool-list
>>>>>>>>>>>> Name State Autostart
>>>>>>>>>>>> -----------------------------------------
>>>>>>>>>>>> 2d0967f8-50d6-4d6d-87b7-4da0596f6924 active no
>>>>>>>>>>>> b3647480-5709-36eb-9ddd-e2c666f30b57 active no
>>>>>>>>>>>> e3831246-ff55-4af4-9964-a17565edaccc active no
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> [Root @ eqx-cs-cmp-02 agent] # df -h
>>>>>>>>>>>> Filesystem Size Used Avail Use% Mounted on
>>>>>>>>>>>> /dev/mapper/vg_root-lv_root
>>>>>>>>>>>> 203G 215G 1.3G 1% /
>>>>>>>>>>>> tmpfs 48g 48g 0 0% / dev / shm
>>>>>>>>>>>> / Dev / sda1 477m 427m 25M 6% / boot
>>>>>>>>>>>> xx.xx.xx.xx:/Cloudstack/primary
>>>>>>>>>>>> 7.6T 4.1T 3.5T 54%
>>>>>>>>>>>> /mnt/b3647480-5709-36eb-9ddd-e2c666f30b57
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I also noticed the following in cloudstack-agent.err
>>>>>>>>>>>>
>>>>>>>>>>>> libvirt: QEMU Driver error : Domain not found: no domain with
>>>>>>>> matching
>>>>>>>>>>> name
>>>>>>>>>>>> 'i-11-600-VM'
>>>>>>>>>>>> libvirt: Storage Driver error : Storage volume not found: no
>>>>>>> storage
>>>>>>>>>> vol
>>>>>>>>>>>> with matching name '089ff3ca-9bff-402c-a14e-ea7a35aaad5c'
>>>>>>>>>>>>
>>>>>>>>>>>> Your help is much appreciated,
>>>>>>>>>>>> Nezar
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On 16 March 2016 at 11:03, Simon Weller <sw...@ena.com>
>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Nezar,
>>>>>>>>>>>>>
>>>>>>>>>>>>> ok, I just took a look at a pastebin you posted.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I see it complaining about the inability to find a volume
>>>>>> prior
>>>>>>> to
>>>>>>>>>>>>> throwing the host exception.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Firstly, let's find out who host 5 is. Can you log into your
>>>>>>> MySQL
>>>>>>>>>>> server
>>>>>>>>>>>>> and run this:
>>>>>>>>>>>>>
>>>>>>>>>>>>> use cloud;select id,name,status,type from host where id=5;
>>>>>>>>>>>>>
>>>>>>>>>>>>> Can you post the results?
>>>>>>>>>>>>>
>>>>>>>>>>>>> If it's a host, can you post the
>>>>>>>> /var/log/cloudstack/agent/agent.log
>>>>>>>>>>> from
>>>>>>>>>>>>> that host?
>>>>>>>>>>>>>
>>>>>>>>>>>>> And also run virsh pool-list on the host in question?
>>>>>>>>>>>>>
>>>>>>>>>>>>> - Si
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> ________________________________________
>>>>>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>>>>>>>>>>>>> Sent: Tuesday, March 15, 2016 6:11 PM
>>>>>>>>>>>>> To: users@cloudstack.apache.org
>>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message:
>>>>> Unable
>>>>>>> to
>>>>>>>>>> start
>>>>>>>>>>>>> instance due to Unable to get answer that is of class
>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Simon,
>>>>>>>>>>>>>
>>>>>>>>>>>>> When you say "double check your routers tied to the
>>>>> network",
>>>>>> do
>>>>>>>> you
>>>>>>>>>>> mean
>>>>>>>>>>>>> the physical routers or the virtual routers?
>>>>>>>>>>>>>
>>>>>>>>>>>>> When I try to create a VM in Prod Environment it fails. I
>>>>>> can't
>>>>>>>> ping
>>>>>>>>>>> the
>>>>>>>>>>>>> Prod virtual router from management server or Host.
>>>>>>>>>>>>>
>>>>>>>>>>>>> But I can ping Dev virtual router from management server and
>>>>>>> Host
>>>>>>>>>> and I
>>>>>>>>>>>> was
>>>>>>>>>>>>> able to create a VM in Dev Environment.
>>>>>>>>>>>>>
>>>>>>>>>>>>> How can I troubleshoot the Prod Virtual router. I launched
>>>>> the
>>>>>>>>>> console
>>>>>>>>>>>> but
>>>>>>>>>>>>> couldn't login.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Please note that we need to take into consideration that the
>>>>>>>>>> affected
>>>>>>>>>>>>> network is the Production network and we need to minimise
>>>>>>> service
>>>>>>>>>>> outage
>>>>>>>>>>>> as
>>>>>>>>>>>>> much as possible.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Your help is much appreciated,
>>>>>>>>>>>>> Nezar
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 16 March 2016 at 08:37, Simon Weller <sw...@ena.com>
>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> I've seen this before where a Virtual Router has crashed
>>>>> and
>>>>>>> the
>>>>>>>>>>>>>> management server can't communicate with it.
>>>>>>>>>>>>>> Are you using an advanced network setup?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> If so, double check your routers tied to the network where
>>>>>> you
>>>>>>>> are
>>>>>>>>>>>>>> attempting to start an instance.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> - Si
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ________________________________________
>>>>>>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>>>>>>>>>>>>>> Sent: Tuesday, March 15, 2016 4:02 PM
>>>>>>>>>>>>>> To: users@cloudstack.apache.org
>>>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message:
>>>>>> Unable
>>>>>>>> to
>>>>>>>>>>> start
>>>>>>>>>>>>>> instance due to Unable to get answer that is of class
>>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Glenn,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> We have 16 hosts, all KVM.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The error is not specific to Host 5. Every time I try to
>>>>>>> create
>>>>>>>> a
>>>>>>>>>> VM
>>>>>>>>>>> it
>>>>>>>>>>>>>> gives me the same error message but on different host.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> How do I restart the agent on the host. Is it service
>>>>>>> affecting?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks very much for your help.
>>>>>>>>>>>>>> Neza
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 15 March 2016 at 17:37, Glenn Wagner <
>>>>>>>>>> glenn.wagner@shapeblue.com>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Can you explain your environment first , how many hosts
>>>>> ,
>>>>>>> what
>>>>>>>>>> is
>>>>>>>>>>>> your
>>>>>>>>>>>>>>> hypervisors (KVM,Xenserver,VMware)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The error seems to be with your Host 5 Hypervisor
>>>>>>>>>>>>>>> Resource [Host:5] is unreachable:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Go to the infrastructure section and see what host5 is
>>>>>>>>>> reporting,
>>>>>>>>>>>>>>> Have you tried to restart the agent on the Host?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>> Glenn
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [image: ShapeBlue] <http://www.shapeblue.com>
>>>>>>>>>>>>>>> Glenn Wagner
>>>>>>>>>>>>>>> Senior Consultant , ShapeBlue
>>>>>>>>>>>>>>> d: * | s: +27 21 527 0091*
>>>>>>>>>> <%7C%20s:%20+27%2021%20527%200091> |
>>>>>>>>>>>> m:
>>>>>>>>>>>>>>> *+27 73 917 4111* <+27%2073%20917%204111>
>>>>>>>>>>>>>>> e: *glenn.wagner@shapeblue.com | t: *
>>>>>>>>>>>>>>> <glenn.wagner@shapeblue.com%20%7C%20t:> | w: *
>>>>>>>>>> www.shapeblue.com
>>>>>>>>>>> *
>>>>>>>>>>>>>>> <http://www.shapeblue.com>
>>>>>>>>>>>>>>> a:
>>>>>>>>>>>>>>> 2nd Floor, Oudehuis Centre, 122 Main Rd, Somerset West
>>>>>> Cape
>>>>>>>> Town
>>>>>>>>>>> 7130
>>>>>>>>>>>>>> South Africa
>>>>>>>>>>>>>>> Shape Blue Ltd is a company incorporated in England &
>>>>>> Wales.
>>>>>>>>>>>> ShapeBlue
>>>>>>>>>>>>>>> Services India LLP is a company incorporated in India
>>>>> and
>>>>>> is
>>>>>>>>>>> operated
>>>>>>>>>>>>>> under
>>>>>>>>>>>>>>> license from Shape Blue Ltd. Shape Blue Brasil
>>>>> Consultoria
>>>>>>>> Ltda
>>>>>>>>>> is
>>>>>>>>>>> a
>>>>>>>>>>>>>>> company incorporated in Brasil and is operated under
>>>>>> license
>>>>>>>>>> from
>>>>>>>>>>>> Shape
>>>>>>>>>>>>>>> Blue Ltd. ShapeBlue SA Pty Ltd is a company registered
>>>>> by
>>>>>>> The
>>>>>>>>>>>> Republic
>>>>>>>>>>>>> of
>>>>>>>>>>>>>>> South Africa and is traded under license from Shape Blue
>>>>>>> Ltd.
>>>>>>>>>>>> ShapeBlue
>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>> a registered trademark.
>>>>>>>>>>>>>>> This email and any attachments to it may be confidential
>>>>>> and
>>>>>>>> are
>>>>>>>>>>>>> intended
>>>>>>>>>>>>>>> solely for the use of the individual to whom it is
>>>>>>> addressed.
>>>>>>>>>> Any
>>>>>>>>>>>> views
>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>> opinions expressed are solely those of the author and do
>>>>>> not
>>>>>>>>>>>>> necessarily
>>>>>>>>>>>>>>> represent those of Shape Blue Ltd or related companies.
>>>>> If
>>>>>>> you
>>>>>>>>>> are
>>>>>>>>>>>> not
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> intended recipient of this email, you must neither take
>>>>>> any
>>>>>>>>>> action
>>>>>>>>>>>>> based
>>>>>>>>>>>>>>> upon its contents, nor copy or show it to anyone. Please
>>>>>>>> contact
>>>>>>>>>>> the
>>>>>>>>>>>>>> sender
>>>>>>>>>>>>>>> if you believe you have received this email in error.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>>>>>> From: Nezar Madbouh [mailto:
>>>>> nezar.madbouh@ipscape.com.au]
>>>>>>>>>>>>>>> Sent: Tuesday, 15 March 2016 5:45 AM
>>>>>>>>>>>>>>> To: users@cloudstack.apache.org
>>>>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message:
>>>>>>> Unable
>>>>>>>>>> to
>>>>>>>>>>>> start
>>>>>>>>>>>>>>> instance due to Unable to get answer that is of class
>>>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Jeremy,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I have uploaded the contents to
>>>>>>> http://pastebin.com/1J3mmgTh
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> your help is much appreciated,
>>>>>>>>>>>>>>> Nezar.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 15 March 2016 at 13:46, Jeremy Peterson <
>>>>>>>>>> jpeterson@acentek.net>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I don't see any attachments can you create a pastebin.
>>>>>>>> That's
>>>>>>>>>>>> easier
>>>>>>>>>>>>>>>> to look through as I think attachments are removed
>>>>>> before
>>>>>>>>>> being
>>>>>>>>>>>> sent
>>>>>>>>>>>>>>>> to user group.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Jeremy
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> -------- Original message --------
>>>>>>>>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>>>>>>>>>>>>>>>> Date: 3/14/2016 6:38 PM (GMT-06:00)
>>>>>>>>>>>>>>>> To: users@cloudstack.apache.org
>>>>>>>>>>>>>>>> Subject: Unable to create a New VM - Error message:
>>>>>> Unable
>>>>>>>> to
>>>>>>>>>>> start
>>>>>>>>>>>>>>>> instance due to Unable to get answer that is of class
>>>>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi Shweta and Sadhu,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Please find attached managerment-server.log which
>>>>>> contains
>>>>>>>> the
>>>>>>>>>>> logs
>>>>>>>>>>>>>>>> since I started creating a new VM (VM id 600).
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Please note that we don't have agent.log files
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Your help is much appreciated.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>> Nezar.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Find out more about ShapeBlue and our range of
>>>>> CloudStack
>>>>>>>>>> related
>>>>>>>>>>>>>> services:
>>>>>>>>>>>>>>> IaaS Cloud Design & Build
>>>>>>>>>>>>>>> <http://shapeblue.com/iaas-cloud-design-and-build//> |
>>>>>>>> CSForge
>>>>>>>>>> -
>>>>>>>>>>>> rapid
>>>>>>>>>>>>>>> IaaS deployment framework <
>>>>> http://shapeblue.com/csforge/>
>>>>>>>>>>>>>>> CloudStack Consulting <
>>>>>>>>>>> http://shapeblue.com/cloudstack-consultancy/>
>>>>>>>>>>>> |
>>>>>>>>>>>>>> CloudStack
>>>>>>>>>>>>>>> Software Engineering
>>>>>>>>>>>>>>> <http://shapeblue.com/cloudstack-software-engineering/>
>>>>>>>>>>>>>>> CloudStack Infrastructure Support
>>>>>>>>>>>>>>> <
>>>>> http://shapeblue.com/cloudstack-infrastructure-support/>
>>>>>> |
>>>>>>>>>>>> CloudStack
>>>>>>>>>>>>>>> Bootcamp Training Courses <
>>>>>>>>>>> http://shapeblue.com/cloudstack-training/
>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> DISCLAIMER
>>>>>>>>>>> ==========
>>>>>>>>>>> This e-mail may contain privileged and confidential information
>>>>>>> which
>>>>>>>> is
>>>>>>>>>>> the property of Accelerite, a Persistent Systems business. It is
>>>>>>>>>> intended
>>>>>>>>>>> only for the use of the individual or entity to which it is
>>>>>>> addressed.
>>>>>>>>>> If
>>>>>>>>>>> you are not the intended recipient, you are not authorized to
>>>>>> read,
>>>>>>>>>> retain,
>>>>>>>>>>> copy, print, distribute or use this message. If you have
>>>>> received
>>>>>>> this
>>>>>>>>>>> communication in error, please notify the sender and delete all
>>>>>>> copies
>>>>>>>>>> of
>>>>>>>>>>> this message. Accelerite, a Persistent Systems business does not
>>>>>>>> accept
>>>>>>>>>> any
>>>>>>>>>>> liability for virus infected mails.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>>
>
Re: Unable to create a New VM - Error message: Unable to start
instance due to Unable to get answer that is of class
com.cloud.agent.api.StartAnswer
Posted by Shweta Agarwal <sh...@accelerite.com>.
Hi Nezar,
When no VMs are running in the Vlan ,Router will only shut down but not deleted. If you want to delete the router delete it manually. And then when you will restart any VM in that Vlan a new Router will be created.
Thanks
Shweta
Principal Product Engineer, CloudPlatform
Accelerite
________________________________________
From: Nezar Madbouh <ne...@ipscape.com.au>
Sent: Friday, April 15, 2016 5:35 AM
To: users@cloudstack.apache.org
Subject: Re: Unable to create a New VM - Error message: Unable to start instance due to Unable to get answer that is of class com.cloud.agent.api.StartAnswer
Hi Ilya,
I shutdown all the VMs in Pre-prod Environmnet expecting the Pre-prod
Virtual router to be deleted automatically because no VM is running on this
vlan but it wasn't deleted.
Am I right to expect the VR to be deleted automatically if no VM is running
in that vlan?
Thanks,
Nezar.
On 15 April 2016 at 09:53, Nezar Madbouh <ne...@ipscape.com.au>
wrote:
> Hi Ilya,
>
> I have run the script and it didn't return any results:
>
> [root@eqx-cs-mgt-01 ~]# ./async_jobs.sh
> [root@eqx-cs-mgt-01 ~]#
>
>
> I also ran the command inside mysql:
>
> mysql> SELECT id, user_id, account_id, instance_type, instance_id,
> job_cmd, job_cmd_info, job_dispatcher, job_pending_signals FROM
> cloud.async_job where job_status=0 and job_dispatcher not like
> "pseudoJobDispatcher";
> Empty set (0.00 sec)
>
> your help is much appreciated,
> Nezar.
>
>
> On 14 April 2016 at 16:49, ilya <il...@gmail.com> wrote:
>
>> It appears you have pending jobs that are not letting you go through.
>>
>> Usually stop and starting management service should be enough.
>>
>> However, try running this script from your management server to show us
>> what you have running in async_job table.
>>
>>
>>
>> > #!/bin/bash
>> > DATESTAMP=$(date +%m%d%y-%H%M%S)
>> > DBPASS=$(java -classpath
>> /usr/share/cloudstack-common/lib/jasypt-1.9.0.jar
>> org.jasypt.intf.cli.JasyptPBEStringDecryptionCLI input="$(cat
>> /etc/cloudstack/management/db.properties | grep db.cloud.password | awk
>> -F'(' '{print $2}' | sed 's/)//g')" password="$(cat
>> /etc/cloudstack/management/key)" | grep -A2 OUTPUT | tail -1)
>> > DBHOST=$(cat /etc/cloudstack/management/db.properties | grep
>> db.cloud.host | awk -F'=' '{print $2}' | tail -1 )
>> > DBUSER=$(cat /etc/cloudstack/management/db.properties | grep
>> db.cloud.username | awk -F'=' '{print $2}')
>> > DB=$(cat /etc/cloudstack/management/db.properties | grep db.cloud.name
>> | awk -F'=' '{print $2}')
>> > DBPORT=$(cat /etc/cloudstack/management/db.properties | grep
>> db.cloud.port | awk -F'=' '{print $2}')
>> > MYSQLCMD="mysql -h $DBHOST -u $DBUSER -P $DBPORT -p$DBPASS $DB"
>> > echo 'SELECT id, user_id, account_id, instance_type, instance_id,
>> job_cmd, job_cmd_info, job_dispatcher, job_pending_signals FROM
>> cloud.async_job where job_status=0 and job_dispatcher not like
>> "pseudoJobDispatcher"' | $MYSQLCMD
>>
>>
>>
>>
>> On 4/13/16 5:16 PM, Nezar Madbouh wrote:
>> > Hi Simon,
>> >
>> > The following work was done but unfortunately the problem is not fixed:
>> >
>> >
>> > on management server:
>> >
>> > service cloudstack-management stop
>> >
>> >
>> > on the host:
>> >
>> > service cloudstack-agent stop
>> >
>> > service libvirtd restart
>> >
>> > service cloudstack-agent start
>> >
>> >
>> > on management server:
>> >
>> > service cloudstack-management start
>> >
>> > Regards,
>> > Nezar.
>> >
>> > On 24 March 2016 at 05:06, Simon Weller <sw...@ena.com> wrote:
>> >
>> >> Yes, I think that order is fine. Make sure the processes are indeed
>> >> shutdown before moving on to the next step.
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> ________________________________________
>> >> From: Nezar Madbouh <ne...@ipscape.com.au>
>> >> Sent: Tuesday, March 22, 2016 5:33 PM
>> >> To: users@cloudstack.apache.org
>> >> Subject: Re: Unable to create a New VM - Error message: Unable to start
>> >> instance due to Unable to get answer that is of class
>> >> com.cloud.agent.api.StartAnswer
>> >>
>> >> Hi Simon,
>> >>
>> >> Can you please check if the following sequence of action is correct:
>> >>
>> >> on management server:
>> >>
>> >> service cloudstack-management stop
>> >>
>> >>
>> >> on the host:
>> >>
>> >> service cloudstack-agent stop
>> >>
>> >> service libvirtd restart
>> >>
>> >> service cloudstack-agent start
>> >>
>> >>
>> >> on management server:
>> >>
>> >> service cloudstack-management start
>> >>
>> >> Thanks very much for your help.
>> >> Nezar.
>> >>
>> >> On 23 March 2016 at 08:39, Simon Weller <sw...@ena.com> wrote:
>> >>
>> >>> Shutting down all the management services should clear the jobs.
>> >>>
>> >>>
>> >>> ________________________________________
>> >>> From: Nezar Madbouh <ne...@ipscape.com.au>
>> >>> Sent: Tuesday, March 22, 2016 4:28 PM
>> >>> To: users@cloudstack.apache.org
>> >>> Subject: Re: Unable to create a New VM - Error message: Unable to
>> start
>> >>> instance due to Unable to get answer that is of class
>> >>> com.cloud.agent.api.StartAnswer
>> >>>
>> >>> Hi Simon,
>> >>>
>> >>> I will plan to restart agents, libvirtd and management services.
>> >>>
>> >>> How do I clear hung jobs.
>> >>>
>> >>> Your help is much appreciated,
>> >>> Nezar.
>> >>>
>> >>> On 22 March 2016 at 23:03, Simon Weller <sw...@ena.com> wrote:
>> >>>
>> >>>> Nezar,
>> >>>>
>> >>>> I've looked through the code related to your release and it seems
>> that
>> >> in
>> >>>> order to throw this exception, 10 attempts have been made to retrieve
>> >> the
>> >>>> disk information prior to the exception being thrown.
>> >>>>
>> >>>> It appears there is something really odd going on here. I'm starting
>> >> to
>> >>>> wonder whether it might be worth while for you to stop all of your
>> >>> agents,
>> >>>> plus management server processes, restart libvirtd on each of the
>> hosts
>> >>> and
>> >>>> then bring the agents back up prior to starting the management
>> servers.
>> >>>>
>> >>>> There are a bunch of logs the pastebin you posted that seems to
>> >> indicate
>> >>>> you have some jobs hung, for example:
>> >>>>
>> >>>> 2016-03-14 21:45:31,828 WARN [o.a.c.f.j.i.AsyncJobMonitor]
>> >>>> (Timer-1:ctx-0c090bd7) Task (job-9066) has been pending for 1660851
>> >>> seconds
>> >>>> 2016-03-14 21:45:31,828 WARN [o.a.c.f.j.i.AsyncJobMonitor]
>> >>>> (Timer-1:ctx-0c090bd7) Task (job-8133) has been pending for 4253003
>> >>> seconds
>> >>>>
>> >>>> Restarting the agents, libvirtd and management will not cause an
>> >> outage
>> >>>> to your cloud, but I would suggest, if you decided to do this, you
>> >> should
>> >>>> complete it in a maintenance window.
>> >>>>
>> >>>> - Si
>> >>>>
>> >>>> ________________________________________
>> >>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>> >>>> Sent: Sunday, March 20, 2016 10:27 PM
>> >>>> To: users@cloudstack.apache.org
>> >>>> Subject: Re: Unable to create a New VM - Error message: Unable to
>> start
>> >>>> instance due to Unable to get answer that is of class
>> >>>> com.cloud.agent.api.StartAnswer
>> >>>>
>> >>>> Hi Simon,
>> >>>>
>> >>>> Please see answers to your questions below. I am not sure if I
>> provided
>> >>> the
>> >>>> correct answer for the primary and secondary storage configuration.
>> >>> Please
>> >>>> let me know if this is not what you are after:
>> >>>>
>> >>>>
>> >>>> Can you screenshot and post somewhere your secondary storage
>> >>> configuration
>> >>>> and your primary storage configuration from the UI with the details?
>> >>>>
>> >>>> mysql> select * from configuration where name like '%secondary%'\G
>> >>>> *************************** 1. row ***************************
>> >>>> category: Account Defaults
>> >>>> instance: DEFAULT
>> >>>> component: management-server
>> >>>> name: max.account.secondary.storage
>> >>>> value: 400
>> >>>> description: The default maximum secondary storage space (in GiB)
>> >> that
>> >>>> can be used for an account
>> >>>> default_value: 400
>> >>>> updated: NULL
>> >>>> scope: NULL
>> >>>> is_dynamic: 0
>> >>>> *************************** 2. row ***************************
>> >>>> category: Project Defaults
>> >>>> instance: DEFAULT
>> >>>> component: management-server
>> >>>> name: max.project.secondary.storage
>> >>>> value: 400
>> >>>> description: The default maximum secondary storage space (in GiB)
>> >> that
>> >>>> can be used for a project
>> >>>> default_value: 400
>> >>>> updated: NULL
>> >>>> scope: NULL
>> >>>> is_dynamic: 0
>> >>>> *************************** 3. row ***************************
>> >>>> category: Hidden
>> >>>> instance: DEFAULT
>> >>>> component: management-server
>> >>>> name: secondary.storage.vm
>> >>>> value: +rwRhJOXPymbQOuAMLnhiw==
>> >>>> description: Deploys a VM per zone to manage secondary storage if
>> >> true,
>> >>>> otherwise secondary storage is mounted on management server
>> >>>> default_value: NULL
>> >>>> updated: NULL
>> >>>> scope: NULL
>> >>>> is_dynamic: 0
>> >>>> *************************** 4. row ***************************
>> >>>> category: Advanced
>> >>>> instance: DEFAULT
>> >>>> component: ExtensionRegistry
>> >>>> name: secondary.storage.vm.allocators.exclude
>> >>>> value: NULL
>> >>>> description: Extensions to exclude from being registered
>> >>>> default_value: NULL
>> >>>> updated: 2014-12-04 00:18:15
>> >>>> scope: Global
>> >>>> is_dynamic: 0
>> >>>> *************************** 5. row ***************************
>> >>>> category: NetworkManager
>> >>>> instance: DEFAULT
>> >>>> component: management-server
>> >>>> name: vm.network.nic.max.secondary.ipaddresses
>> >>>> value: NULL
>> >>>> description: Specify the number of secondary ip addresses per nic
>> per
>> >>> vm
>> >>>> default_value: 256
>> >>>> updated: NULL
>> >>>> scope: NULL
>> >>>> is_dynamic: 0
>> >>>> 5 rows in set (0.00 sec)
>> >>>>
>> >>>>
>> >>>> mysql> select * from configuration where name like '%primary%'\G
>> >>>> *************************** 1. row ***************************
>> >>>> category: Account Defaults
>> >>>> instance: DEFAULT
>> >>>> component: management-server
>> >>>> name: max.account.primary.storage
>> >>>> value: 200
>> >>>> description: The default maximum primary storage space (in GiB)
>> that
>> >>> can
>> >>>> be used for an account
>> >>>> default_value: 200
>> >>>> updated: NULL
>> >>>> scope: NULL
>> >>>> is_dynamic: 0
>> >>>> *************************** 2. row ***************************
>> >>>> category: Project Defaults
>> >>>> instance: DEFAULT
>> >>>> component: management-server
>> >>>> name: max.project.primary.storage
>> >>>> value: 200
>> >>>> description: The default maximum primary storage space (in GiB)
>> that
>> >>> can
>> >>>> be used for a project
>> >>>> default_value: 200
>> >>>> updated: NULL
>> >>>> scope: NULL
>> >>>> is_dynamic: 0
>> >>>> *************************** 3. row ***************************
>> >>>> category: Storage
>> >>>> instance: DEFAULT
>> >>>> component: TemplateManager
>> >>>> name: primary.storage.download.wait
>> >>>> value: 10800
>> >>>> description: In second, timeout for download template to primary
>> >>> storage
>> >>>> default_value: NULL
>> >>>> updated: NULL
>> >>>> scope: NULL
>> >>>> is_dynamic: 0
>> >>>> 3 rows in set (0.00 sec)
>> >>>>
>> >>>>
>> >>>>
>> >>>> Also, what is the Cloudstack version you are running and what is the
>> OS
>> >>>> you're running it on?
>> >>>>
>> >>>> Cloudstack 4.4.1
>> >>>>
>> >>>> Centos 6.6
>> >>>>
>> >>>> Are you using actimeo=0 in your NFS mount line?
>> >>>>
>> >>>> Primary storage on Hypervisors has the following parameters
>> >>>> 10.61.154.10:/Cloudstack/primary on
>> >>>> /mnt/b3647480-5709-36eb-9ddd-e2c666f30b57 type nfs
>> >>>> (rw,noatime,nfsvers=3,proto=tcp,hard,nolock,intr,addr=10.61.154.10)
>> >>>>
>> >>>>
>> >>>> Secondary storage on SSVM has the following parameters
>> >>>> 10.61.154.10:/Cloudstack/secondary on
>> >>>> /mnt/SecStorage/d11d5c92-6950-30e8-a0b5-3cd46a17ec14 type nfs4
>> >>>>
>> >>>>
>> >>>
>> >>
>> (rw,relatime,vers=4,rsize=262144,wsize=262144,namlen=255,acdirmin=0,acdirmax=0,soft,proto=tcp,port=0,timeo=133,retrans=
>> 2147483647
>> ,sec=sys,clientaddr=10.61.154.70,minorversion=0,local_lock=none,addr=10.61.154.10)
>> >>>>
>> >>>>
>> >>>> Thanks & Regards,
>> >>>> Nezar.
>> >>>>
>> >>>> On 18 March 2016 at 23:50, Simon Weller <sw...@ena.com> wrote:
>> >>>>
>> >>>>> Also, what is the Cloudstack version you are running and what is the
>> >> OS
>> >>>>> you're running it on?
>> >>>>>
>> >>>>> Are you using actimeo=0 in your NFS mount line?
>> >>>>>
>> >>>>> ________________________________________
>> >>>>> From: Simon Weller <sw...@ena.com>
>> >>>>> Sent: Friday, March 18, 2016 7:40 AM
>> >>>>> To: users@cloudstack.apache.org
>> >>>>> Subject: Re: Unable to create a New VM - Error message: Unable to
>> >> start
>> >>>>> instance due to Unable to get answer that is of class
>> >>>>> com.cloud.agent.api.StartAnswer
>> >>>>>
>> >>>>> Nexar,
>> >>>>>
>> >>>>> That's normal behaviour. You can only ssh to the link local on the
>> >> host
>> >>>>> where the VM resides.
>> >>>>>
>> >>>>> Can you screenshot and post somewhere your secondary storage
>> >>>> configuration
>> >>>>> and your primary storage configuration from the UI with the details?
>> >>>>>
>> >>>>> - Si
>> >>>>>
>> >>>>> ________________________________________
>> >>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>> >>>>> Sent: Thursday, March 17, 2016 8:19 PM
>> >>>>> To: users@cloudstack.apache.org
>> >>>>> Subject: Re: Unable to create a New VM - Error message: Unable to
>> >> start
>> >>>>> instance due to Unable to get answer that is of class
>> >>>>> com.cloud.agent.api.StartAnswer
>> >>>>>
>> >>>>> Hi Simon,
>> >>>>>
>> >>>>> I just noticed that from other Hosts I can't ssh into the SSVM:
>> >>>>>
>> >>>>> [root@eqx-cs-cmp-01 ~]# ssh -i /root/.ssh/id_rsa.cloud -p 3922
>> >>>>> root@169.254.3.161
>> >>>>> ssh: connect to host 169.254.3.161 port 3922: No route to host
>> >>>>>
>> >>>>>
>> >>>>> I can only ssh from one Host:
>> >>>>> [root@eqx-cs-cmp-06 ~]# ssh -i /root/.ssh/id_rsa.cloud -p 3922
>> >>>>> root@169.254.3.161
>> >>>>> Linux s-11-VM 3.2.0-4-amd64 #1 SMP Debian 3.2.57-3 x86_64
>> >>>>>
>> >>>>> Thanks & Regards,
>> >>>>> Nezar.
>> >>>>>
>> >>>>> On 18 March 2016 at 10:36, Nezar Madbouh <
>> >> nezar.madbouh@ipscape.com.au
>> >>>>
>> >>>>> wrote:
>> >>>>>
>> >>>>>> Hi Simon,
>> >>>>>>
>> >>>>>> Yes existing VMs are successfully running. Yesterday I was able to
>> >>>> create
>> >>>>>> a VM in Dev Environment but I can't create VMs in Prod environment.
>> >>>>>>
>> >>>>>> Thanks,
>> >>>>>> Nezar.
>> >>>>>>
>> >>>>>> On 18 March 2016 at 09:12, Simon Weller <sw...@ena.com> wrote:
>> >>>>>>
>> >>>>>>> I'm assuming you have other VMs successfully running, correct? So
>> >>> you
>> >>>>> can
>> >>>>>>> write to your primary storage.
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> ________________________________________
>> >>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>> >>>>>>> Sent: Thursday, March 17, 2016 4:43 PM
>> >>>>>>> To: users@cloudstack.apache.org
>> >>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable to
>> >>>> start
>> >>>>>>> instance due to Unable to get answer that is of class
>> >>>>>>> com.cloud.agent.api.StartAnswer
>> >>>>>>>
>> >>>>>>> Hi Simon and Shweta,
>> >>>>>>>
>> >>>>>>> Please see the result below:
>> >>>>>>>
>> >>>>>>> root@s-11-VM:~# /usr/local/cloud/systemvm/ssvm-check.sh
>> >>>>>>> ================================================
>> >>>>>>> First DNS server is 8.8.8.8
>> >>>>>>> PING 8.8.8.8 (8.8.8.8): 48 data bytes
>> >>>>>>> 56 bytes from 8.8.8.8: icmp_seq=0 ttl=57 time=5.209 ms
>> >>>>>>> 56 bytes from 8.8.8.8: icmp_seq=1 ttl=57 time=0.936 ms
>> >>>>>>> --- 8.8.8.8 ping statistics ---
>> >>>>>>> 2 packets transmitted, 2 packets received, 0% packet loss
>> >>>>>>> round-trip min/avg/max/stddev = 0.936/3.072/5.209/2.137 ms
>> >>>>>>> Good: Can ping DNS server
>> >>>>>>> ================================================
>> >>>>>>> Good: DNS resolves download.cloud.com
>> >>>>>>> ================================================
>> >>>>>>> nfs is currently mounted
>> >>>>>>> Mount point is
>> >> /mnt/SecStorage/d11d5c92-6950-30e8-a0b5-3cd46a17ec14
>> >>>>>>> Good: Can write to mount point
>> >>>>>>> ================================================
>> >>>>>>> Management server is 10.61.153.20. Checking connectivity.
>> >>>>>>> Good: Can connect to management server port 8250
>> >>>>>>> ================================================
>> >>>>>>> Good: Java process is running
>> >>>>>>> ================================================
>> >>>>>>> Tests Complete. Look for ERROR or WARNING above.
>> >>>>>>> root@s-11-VM:~#
>> >>>>>>>
>> >>>>>>> Thanks & Regards,
>> >>>>>>> Nezar.
>> >>>>>>>
>> >>>>>>> On 17 March 2016 at 09:42, Simon Weller <sw...@ena.com> wrote:
>> >>>>>>>
>> >>>>>>>> Nezar,
>> >>>>>>>>
>> >>>>>>>> Log into the Cloudstack UI.
>> >>>>>>>>
>> >>>>>>>> Click on "Infrastructure" on the left menu. Click on View All
>> >>> below
>> >>>>>>>> "System VMs". find the item marked "Secondary Storage VM" and
>> >>> then
>> >>>>>>> click
>> >>>>>>>> on the Name of that VM on the left (it will be s-XXXX-VM).
>> >> You'll
>> >>>> see
>> >>>>> a
>> >>>>>>>> bunch of information about the VM. Find the Host line and also
>> >> the
>> >>>>> Link
>> >>>>>>>> Local IP Address line and then follow the original instructions
>> >> in
>> >>>> my
>> >>>>>>>> previous email.
>> >>>>>>>>
>> >>>>>>>> - Si
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> ________________________________________
>> >>>>>>>> From: Shweta Agarwal <sh...@accelerite.com>
>> >>>>>>>> Sent: Wednesday, March 16, 2016 12:16 AM
>> >>>>>>>> To: users@cloudstack.apache.org
>> >>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable
>> >> to
>> >>>>> start
>> >>>>>>>> instance due to Unable to get answer that is of class
>> >>>>>>>> com.cloud.agent.api.StartAnswer
>> >>>>>>>>
>> >>>>>>>> Hi Nezar,
>> >>>>>>>>
>> >>>>>>>> You can find host name on detail view of SSVM in UI .
>> >>>>>>>> You can also use list api command listSystemVms to get the host
>> >>>>> details
>> >>>>>>> of
>> >>>>>>>> system vms.
>> >>>>>>>>
>> >>>>>>>> Thanks
>> >>>>>>>> Shweta
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> ________________________________________
>> >>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>> >>>>>>>> Sent: Wednesday, March 16, 2016 9:58 AM
>> >>>>>>>> To: users@cloudstack.apache.org
>> >>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable
>> >> to
>> >>>>> start
>> >>>>>>>> instance due to Unable to get answer that is of class
>> >>>>>>>> com.cloud.agent.api.StartAnswer
>> >>>>>>>>
>> >>>>>>>> Hi Simon,
>> >>>>>>>>
>> >>>>>>>> How can I find out which host the SSVM is on?
>> >>>>>>>>
>> >>>>>>>> Thanks & Regards,
>> >>>>>>>> Nezar.
>> >>>>>>>>
>> >>>>>>>> On 16 March 2016 at 12:28, Simon Weller <sw...@ena.com>
>> >> wrote:
>> >>>>>>>>
>> >>>>>>>>> Can you ssh into the Secondary Storage VM?
>> >>>>>>>>>
>> >>>>>>>>> This can be done by getting the link-local-ip address from the
>> >>>>>>>>> infrastructure menu in the UI and then sshing into the host
>> >> the
>> >>>> SSVM
>> >>>>>>> is
>> >>>>>>>> on
>> >>>>>>>>> and issuing this command as root: ssh -i
>> >> /root/.ssh/id_rsa.cloud
>> >>>> -p
>> >>>>>>> 3922
>> >>>>>>>>> root@link-local-ip (replace link-local-ip with the link
>> >> local
>> >>>> ip).
>> >>>>>>>>>
>> >>>>>>>>> Try running the SSVM diangnostic script:
>> >>>>>>>>> usr/local/cloud/systemvm/ssvm-check.sh
>> >>>>>>>>>
>> >>>>>>>>> - Si
>> >>>>>>>>> ________________________________________
>> >>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>> >>>>>>>>> Sent: Tuesday, March 15, 2016 7:49 PM
>> >>>>>>>>> To: users@cloudstack.apache.org
>> >>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable
>> >>> to
>> >>>>>>> start
>> >>>>>>>>> instance due to Unable to get answer that is of class
>> >>>>>>>>> com.cloud.agent.api.StartAnswer
>> >>>>>>>>>
>> >>>>>>>>> Hi Simon,
>> >>>>>>>>>
>> >>>>>>>>> Please see details below:
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>> mysql> use cloud; select id, name, status, type from host
>> >> where
>> >>>> id =
>> >>>>>>> 5;
>> >>>>>>>>> Reading table information for completion of table and column
>> >>> names
>> >>>>>>>>> You can turn off this feature to get a quicker startup with -A
>> >>>>>>>>>
>> >>>>>>>>> Database changed
>> >>>>>>>>> +----+------------------------------+--------+---------+
>> >>>>>>>>> | id | name | status | type |
>> >>>>>>>>> +----+------------------------------+--------+---------+
>> >>>>>>>>> | 5 | eqx-cs-cmp-02.ipscape.com.au | up | routing |
>> >>>>>>>>> +----+------------------------------+--------+---------+
>> >>>>>>>>> 1 row in set (0.00 sec)
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>> cat /var/log/cloudstack/agent/agent.log
>> >>>>>>>>> com.cloud.utils.exception.CloudRuntimeException:
>> >>>>>>>>> com.cloud.utils.exception.CloudRuntimeException: Can't find
>> >>>>>>>>> volume:089ff3ca-9bff-402c-a14e-ea7a35aaad5c
>> >>>>>>>>> at
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>
>> com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.getPhysicalDisk(KVMStoragePoolManager.java:262)
>> >>>>>>>>> at
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>
>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.createVbd(LibvirtComputingResource.java:3921)
>> >>>>>>>>> at
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>
>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(LibvirtComputingResource.java:3789)
>> >>>>>>>>> at
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>
>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1333)
>> >>>>>>>>> at
>> >> com.cloud.agent.Agent.processRequest(Agent.java:501)
>> >>>>>>>>> at
>> >>>>>>>> com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:808)
>> >>>>>>>>> at com.cloud.utils.nio.Task.run(Task.java:84)
>> >>>>>>>>> at
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> >>>>>>>>> at
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> >>>>>>>>> at java.lang.Thread.run(Thread.java:745)
>> >>>>>>>>> agent.log.2016-03-03.gz (END)
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>> [root@eqx-cs-cmp-02 agent]# virsh pool-list
>> >>>>>>>>> Name State Autostart
>> >>>>>>>>> -----------------------------------------
>> >>>>>>>>> 2d0967f8-50d6-4d6d-87b7-4da0596f6924 active no
>> >>>>>>>>> b3647480-5709-36eb-9ddd-e2c666f30b57 active no
>> >>>>>>>>> e3831246-ff55-4af4-9964-a17565edaccc active no
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>> [Root @ eqx-cs-cmp-02 agent] # df -h
>> >>>>>>>>> Filesystem Size Used Avail Use% Mounted on
>> >>>>>>>>> /dev/mapper/vg_root-lv_root
>> >>>>>>>>> 203G 215G 1.3G 1% /
>> >>>>>>>>> tmpfs 48g 48g 0 0% / dev / shm
>> >>>>>>>>> / Dev / sda1 477m 427m 25M 6% / boot
>> >>>>>>>>> xx.xx.xx.xx:/Cloudstack/primary
>> >>>>>>>>> 7.6T 4.1T 3.5T 54%
>> >>>>>>>>> /mnt/b3647480-5709-36eb-9ddd-e2c666f30b57
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>> I also noticed the following in cloudstack-agent.err
>> >>>>>>>>>
>> >>>>>>>>> libvirt: QEMU Driver error : Domain not found: no domain with
>> >>>>> matching
>> >>>>>>>> name
>> >>>>>>>>> 'i-11-600-VM'
>> >>>>>>>>> libvirt: Storage Driver error : Storage volume not found: no
>> >>>> storage
>> >>>>>>> vol
>> >>>>>>>>> with matching name '089ff3ca-9bff-402c-a14e-ea7a35aaad5c'
>> >>>>>>>>>
>> >>>>>>>>> Your help is much appreciated,
>> >>>>>>>>> Nezar
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>> On 16 March 2016 at 11:03, Simon Weller <sw...@ena.com>
>> >>> wrote:
>> >>>>>>>>>
>> >>>>>>>>>> Nezar,
>> >>>>>>>>>>
>> >>>>>>>>>> ok, I just took a look at a pastebin you posted.
>> >>>>>>>>>>
>> >>>>>>>>>> I see it complaining about the inability to find a volume
>> >>> prior
>> >>>> to
>> >>>>>>>>>> throwing the host exception.
>> >>>>>>>>>>
>> >>>>>>>>>> Firstly, let's find out who host 5 is. Can you log into your
>> >>>> MySQL
>> >>>>>>>> server
>> >>>>>>>>>> and run this:
>> >>>>>>>>>>
>> >>>>>>>>>> use cloud;select id,name,status,type from host where id=5;
>> >>>>>>>>>>
>> >>>>>>>>>> Can you post the results?
>> >>>>>>>>>>
>> >>>>>>>>>> If it's a host, can you post the
>> >>>>> /var/log/cloudstack/agent/agent.log
>> >>>>>>>> from
>> >>>>>>>>>> that host?
>> >>>>>>>>>>
>> >>>>>>>>>> And also run virsh pool-list on the host in question?
>> >>>>>>>>>>
>> >>>>>>>>>> - Si
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> ________________________________________
>> >>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>> >>>>>>>>>> Sent: Tuesday, March 15, 2016 6:11 PM
>> >>>>>>>>>> To: users@cloudstack.apache.org
>> >>>>>>>>>> Subject: Re: Unable to create a New VM - Error message:
>> >> Unable
>> >>>> to
>> >>>>>>> start
>> >>>>>>>>>> instance due to Unable to get answer that is of class
>> >>>>>>>>>> com.cloud.agent.api.StartAnswer
>> >>>>>>>>>>
>> >>>>>>>>>> Hi Simon,
>> >>>>>>>>>>
>> >>>>>>>>>> When you say "double check your routers tied to the
>> >> network",
>> >>> do
>> >>>>> you
>> >>>>>>>> mean
>> >>>>>>>>>> the physical routers or the virtual routers?
>> >>>>>>>>>>
>> >>>>>>>>>> When I try to create a VM in Prod Environment it fails. I
>> >>> can't
>> >>>>> ping
>> >>>>>>>> the
>> >>>>>>>>>> Prod virtual router from management server or Host.
>> >>>>>>>>>>
>> >>>>>>>>>> But I can ping Dev virtual router from management server and
>> >>>> Host
>> >>>>>>> and I
>> >>>>>>>>> was
>> >>>>>>>>>> able to create a VM in Dev Environment.
>> >>>>>>>>>>
>> >>>>>>>>>> How can I troubleshoot the Prod Virtual router. I launched
>> >> the
>> >>>>>>> console
>> >>>>>>>>> but
>> >>>>>>>>>> couldn't login.
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> Please note that we need to take into consideration that the
>> >>>>>>> affected
>> >>>>>>>>>> network is the Production network and we need to minimise
>> >>>> service
>> >>>>>>>> outage
>> >>>>>>>>> as
>> >>>>>>>>>> much as possible.
>> >>>>>>>>>>
>> >>>>>>>>>> Your help is much appreciated,
>> >>>>>>>>>> Nezar
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> On 16 March 2016 at 08:37, Simon Weller <sw...@ena.com>
>> >>>> wrote:
>> >>>>>>>>>>
>> >>>>>>>>>>> I've seen this before where a Virtual Router has crashed
>> >> and
>> >>>> the
>> >>>>>>>>>>> management server can't communicate with it.
>> >>>>>>>>>>> Are you using an advanced network setup?
>> >>>>>>>>>>>
>> >>>>>>>>>>> If so, double check your routers tied to the network where
>> >>> you
>> >>>>> are
>> >>>>>>>>>>> attempting to start an instance.
>> >>>>>>>>>>>
>> >>>>>>>>>>> - Si
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> ________________________________________
>> >>>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>> >>>>>>>>>>> Sent: Tuesday, March 15, 2016 4:02 PM
>> >>>>>>>>>>> To: users@cloudstack.apache.org
>> >>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message:
>> >>> Unable
>> >>>>> to
>> >>>>>>>> start
>> >>>>>>>>>>> instance due to Unable to get answer that is of class
>> >>>>>>>>>>> com.cloud.agent.api.StartAnswer
>> >>>>>>>>>>>
>> >>>>>>>>>>> Hi Glenn,
>> >>>>>>>>>>>
>> >>>>>>>>>>> We have 16 hosts, all KVM.
>> >>>>>>>>>>>
>> >>>>>>>>>>> The error is not specific to Host 5. Every time I try to
>> >>>> create
>> >>>>> a
>> >>>>>>> VM
>> >>>>>>>> it
>> >>>>>>>>>>> gives me the same error message but on different host.
>> >>>>>>>>>>>
>> >>>>>>>>>>> How do I restart the agent on the host. Is it service
>> >>>> affecting?
>> >>>>>>>>>>>
>> >>>>>>>>>>> Thanks very much for your help.
>> >>>>>>>>>>> Neza
>> >>>>>>>>>>>
>> >>>>>>>>>>> On 15 March 2016 at 17:37, Glenn Wagner <
>> >>>>>>> glenn.wagner@shapeblue.com>
>> >>>>>>>>>>> wrote:
>> >>>>>>>>>>>
>> >>>>>>>>>>>> Hi,
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> Can you explain your environment first , how many hosts
>> >> ,
>> >>>> what
>> >>>>>>> is
>> >>>>>>>>> your
>> >>>>>>>>>>>> hypervisors (KVM,Xenserver,VMware)
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> The error seems to be with your Host 5 Hypervisor
>> >>>>>>>>>>>> Resource [Host:5] is unreachable:
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> Go to the infrastructure section and see what host5 is
>> >>>>>>> reporting,
>> >>>>>>>>>>>> Have you tried to restart the agent on the Host?
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> Thanks
>> >>>>>>>>>>>> Glenn
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> [image: ShapeBlue] <http://www.shapeblue.com>
>> >>>>>>>>>>>> Glenn Wagner
>> >>>>>>>>>>>> Senior Consultant , ShapeBlue
>> >>>>>>>>>>>> d: * | s: +27 21 527 0091*
>> >>>>>>> <%7C%20s:%20+27%2021%20527%200091> |
>> >>>>>>>>> m:
>> >>>>>>>>>>>> *+27 73 917 4111* <+27%2073%20917%204111>
>> >>>>>>>>>>>> e: *glenn.wagner@shapeblue.com | t: *
>> >>>>>>>>>>>> <glenn.wagner@shapeblue.com%20%7C%20t:> | w: *
>> >>>>>>> www.shapeblue.com
>> >>>>>>>> *
>> >>>>>>>>>>>> <http://www.shapeblue.com>
>> >>>>>>>>>>>> a:
>> >>>>>>>>>>>> 2nd Floor, Oudehuis Centre, 122 Main Rd, Somerset West
>> >>> Cape
>> >>>>> Town
>> >>>>>>>> 7130
>> >>>>>>>>>>> South Africa
>> >>>>>>>>>>>> Shape Blue Ltd is a company incorporated in England &
>> >>> Wales.
>> >>>>>>>>> ShapeBlue
>> >>>>>>>>>>>> Services India LLP is a company incorporated in India
>> >> and
>> >>> is
>> >>>>>>>> operated
>> >>>>>>>>>>> under
>> >>>>>>>>>>>> license from Shape Blue Ltd. Shape Blue Brasil
>> >> Consultoria
>> >>>>> Ltda
>> >>>>>>> is
>> >>>>>>>> a
>> >>>>>>>>>>>> company incorporated in Brasil and is operated under
>> >>> license
>> >>>>>>> from
>> >>>>>>>>> Shape
>> >>>>>>>>>>>> Blue Ltd. ShapeBlue SA Pty Ltd is a company registered
>> >> by
>> >>>> The
>> >>>>>>>>> Republic
>> >>>>>>>>>> of
>> >>>>>>>>>>>> South Africa and is traded under license from Shape Blue
>> >>>> Ltd.
>> >>>>>>>>> ShapeBlue
>> >>>>>>>>>>> is
>> >>>>>>>>>>>> a registered trademark.
>> >>>>>>>>>>>> This email and any attachments to it may be confidential
>> >>> and
>> >>>>> are
>> >>>>>>>>>> intended
>> >>>>>>>>>>>> solely for the use of the individual to whom it is
>> >>>> addressed.
>> >>>>>>> Any
>> >>>>>>>>> views
>> >>>>>>>>>>> or
>> >>>>>>>>>>>> opinions expressed are solely those of the author and do
>> >>> not
>> >>>>>>>>>> necessarily
>> >>>>>>>>>>>> represent those of Shape Blue Ltd or related companies.
>> >> If
>> >>>> you
>> >>>>>>> are
>> >>>>>>>>> not
>> >>>>>>>>>>> the
>> >>>>>>>>>>>> intended recipient of this email, you must neither take
>> >>> any
>> >>>>>>> action
>> >>>>>>>>>> based
>> >>>>>>>>>>>> upon its contents, nor copy or show it to anyone. Please
>> >>>>> contact
>> >>>>>>>> the
>> >>>>>>>>>>> sender
>> >>>>>>>>>>>> if you believe you have received this email in error.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> -----Original Message-----
>> >>>>>>>>>>>> From: Nezar Madbouh [mailto:
>> >> nezar.madbouh@ipscape.com.au]
>> >>>>>>>>>>>> Sent: Tuesday, 15 March 2016 5:45 AM
>> >>>>>>>>>>>> To: users@cloudstack.apache.org
>> >>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message:
>> >>>> Unable
>> >>>>>>> to
>> >>>>>>>>> start
>> >>>>>>>>>>>> instance due to Unable to get answer that is of class
>> >>>>>>>>>>>> com.cloud.agent.api.StartAnswer
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> Hi Jeremy,
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> I have uploaded the contents to
>> >>>> http://pastebin.com/1J3mmgTh
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> your help is much appreciated,
>> >>>>>>>>>>>> Nezar.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> On 15 March 2016 at 13:46, Jeremy Peterson <
>> >>>>>>> jpeterson@acentek.net>
>> >>>>>>>>>>> wrote:
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>> I don't see any attachments can you create a pastebin.
>> >>>>> That's
>> >>>>>>>>> easier
>> >>>>>>>>>>>>> to look through as I think attachments are removed
>> >>> before
>> >>>>>>> being
>> >>>>>>>>> sent
>> >>>>>>>>>>>>> to user group.
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Jeremy
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> -------- Original message --------
>> >>>>>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>> >>>>>>>>>>>>> Date: 3/14/2016 6:38 PM (GMT-06:00)
>> >>>>>>>>>>>>> To: users@cloudstack.apache.org
>> >>>>>>>>>>>>> Subject: Unable to create a New VM - Error message:
>> >>> Unable
>> >>>>> to
>> >>>>>>>> start
>> >>>>>>>>>>>>> instance due to Unable to get answer that is of class
>> >>>>>>>>>>>>> com.cloud.agent.api.StartAnswer
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Hi Shweta and Sadhu,
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Please find attached managerment-server.log which
>> >>> contains
>> >>>>> the
>> >>>>>>>> logs
>> >>>>>>>>>>>>> since I started creating a new VM (VM id 600).
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Please note that we don't have agent.log files
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Your help is much appreciated.
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Regards,
>> >>>>>>>>>>>>> Nezar.
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>> Find out more about ShapeBlue and our range of
>> >> CloudStack
>> >>>>>>> related
>> >>>>>>>>>>> services:
>> >>>>>>>>>>>> IaaS Cloud Design & Build
>> >>>>>>>>>>>> <http://shapeblue.com/iaas-cloud-design-and-build//> |
>> >>>>> CSForge
>> >>>>>>> -
>> >>>>>>>>> rapid
>> >>>>>>>>>>>> IaaS deployment framework <
>> >> http://shapeblue.com/csforge/>
>> >>>>>>>>>>>> CloudStack Consulting <
>> >>>>>>>> http://shapeblue.com/cloudstack-consultancy/>
>> >>>>>>>>> |
>> >>>>>>>>>>> CloudStack
>> >>>>>>>>>>>> Software Engineering
>> >>>>>>>>>>>> <http://shapeblue.com/cloudstack-software-engineering/>
>> >>>>>>>>>>>> CloudStack Infrastructure Support
>> >>>>>>>>>>>> <
>> >> http://shapeblue.com/cloudstack-infrastructure-support/>
>> >>> |
>> >>>>>>>>> CloudStack
>> >>>>>>>>>>>> Bootcamp Training Courses <
>> >>>>>>>> http://shapeblue.com/cloudstack-training/
>> >>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> DISCLAIMER
>> >>>>>>>> ==========
>> >>>>>>>> This e-mail may contain privileged and confidential information
>> >>>> which
>> >>>>> is
>> >>>>>>>> the property of Accelerite, a Persistent Systems business. It is
>> >>>>>>> intended
>> >>>>>>>> only for the use of the individual or entity to which it is
>> >>>> addressed.
>> >>>>>>> If
>> >>>>>>>> you are not the intended recipient, you are not authorized to
>> >>> read,
>> >>>>>>> retain,
>> >>>>>>>> copy, print, distribute or use this message. If you have
>> >> received
>> >>>> this
>> >>>>>>>> communication in error, please notify the sender and delete all
>> >>>> copies
>> >>>>>>> of
>> >>>>>>>> this message. Accelerite, a Persistent Systems business does not
>> >>>>> accept
>> >>>>>>> any
>> >>>>>>>> liability for virus infected mails.
>> >>>>>>>>
>> >>>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>
>> >>
>> >
>>
>
>
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Accelerite, a Persistent Systems business. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent Systems business does not accept any liability for virus infected mails.
Re: Unable to create a New VM - Error message: Unable to start
instance due to Unable to get answer that is of class com.cloud.agent.api.StartAnswer
Posted by Nezar Madbouh <ne...@ipscape.com.au>.
Hi Ilya,
I shutdown all the VMs in Pre-prod Environmnet expecting the Pre-prod
Virtual router to be deleted automatically because no VM is running on this
vlan but it wasn't deleted.
Am I right to expect the VR to be deleted automatically if no VM is running
in that vlan?
Thanks,
Nezar.
On 15 April 2016 at 09:53, Nezar Madbouh <ne...@ipscape.com.au>
wrote:
> Hi Ilya,
>
> I have run the script and it didn't return any results:
>
> [root@eqx-cs-mgt-01 ~]# ./async_jobs.sh
> [root@eqx-cs-mgt-01 ~]#
>
>
> I also ran the command inside mysql:
>
> mysql> SELECT id, user_id, account_id, instance_type, instance_id,
> job_cmd, job_cmd_info, job_dispatcher, job_pending_signals FROM
> cloud.async_job where job_status=0 and job_dispatcher not like
> "pseudoJobDispatcher";
> Empty set (0.00 sec)
>
> your help is much appreciated,
> Nezar.
>
>
> On 14 April 2016 at 16:49, ilya <il...@gmail.com> wrote:
>
>> It appears you have pending jobs that are not letting you go through.
>>
>> Usually stop and starting management service should be enough.
>>
>> However, try running this script from your management server to show us
>> what you have running in async_job table.
>>
>>
>>
>> > #!/bin/bash
>> > DATESTAMP=$(date +%m%d%y-%H%M%S)
>> > DBPASS=$(java -classpath
>> /usr/share/cloudstack-common/lib/jasypt-1.9.0.jar
>> org.jasypt.intf.cli.JasyptPBEStringDecryptionCLI input="$(cat
>> /etc/cloudstack/management/db.properties | grep db.cloud.password | awk
>> -F'(' '{print $2}' | sed 's/)//g')" password="$(cat
>> /etc/cloudstack/management/key)" | grep -A2 OUTPUT | tail -1)
>> > DBHOST=$(cat /etc/cloudstack/management/db.properties | grep
>> db.cloud.host | awk -F'=' '{print $2}' | tail -1 )
>> > DBUSER=$(cat /etc/cloudstack/management/db.properties | grep
>> db.cloud.username | awk -F'=' '{print $2}')
>> > DB=$(cat /etc/cloudstack/management/db.properties | grep db.cloud.name
>> | awk -F'=' '{print $2}')
>> > DBPORT=$(cat /etc/cloudstack/management/db.properties | grep
>> db.cloud.port | awk -F'=' '{print $2}')
>> > MYSQLCMD="mysql -h $DBHOST -u $DBUSER -P $DBPORT -p$DBPASS $DB"
>> > echo 'SELECT id, user_id, account_id, instance_type, instance_id,
>> job_cmd, job_cmd_info, job_dispatcher, job_pending_signals FROM
>> cloud.async_job where job_status=0 and job_dispatcher not like
>> "pseudoJobDispatcher"' | $MYSQLCMD
>>
>>
>>
>>
>> On 4/13/16 5:16 PM, Nezar Madbouh wrote:
>> > Hi Simon,
>> >
>> > The following work was done but unfortunately the problem is not fixed:
>> >
>> >
>> > on management server:
>> >
>> > service cloudstack-management stop
>> >
>> >
>> > on the host:
>> >
>> > service cloudstack-agent stop
>> >
>> > service libvirtd restart
>> >
>> > service cloudstack-agent start
>> >
>> >
>> > on management server:
>> >
>> > service cloudstack-management start
>> >
>> > Regards,
>> > Nezar.
>> >
>> > On 24 March 2016 at 05:06, Simon Weller <sw...@ena.com> wrote:
>> >
>> >> Yes, I think that order is fine. Make sure the processes are indeed
>> >> shutdown before moving on to the next step.
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> ________________________________________
>> >> From: Nezar Madbouh <ne...@ipscape.com.au>
>> >> Sent: Tuesday, March 22, 2016 5:33 PM
>> >> To: users@cloudstack.apache.org
>> >> Subject: Re: Unable to create a New VM - Error message: Unable to start
>> >> instance due to Unable to get answer that is of class
>> >> com.cloud.agent.api.StartAnswer
>> >>
>> >> Hi Simon,
>> >>
>> >> Can you please check if the following sequence of action is correct:
>> >>
>> >> on management server:
>> >>
>> >> service cloudstack-management stop
>> >>
>> >>
>> >> on the host:
>> >>
>> >> service cloudstack-agent stop
>> >>
>> >> service libvirtd restart
>> >>
>> >> service cloudstack-agent start
>> >>
>> >>
>> >> on management server:
>> >>
>> >> service cloudstack-management start
>> >>
>> >> Thanks very much for your help.
>> >> Nezar.
>> >>
>> >> On 23 March 2016 at 08:39, Simon Weller <sw...@ena.com> wrote:
>> >>
>> >>> Shutting down all the management services should clear the jobs.
>> >>>
>> >>>
>> >>> ________________________________________
>> >>> From: Nezar Madbouh <ne...@ipscape.com.au>
>> >>> Sent: Tuesday, March 22, 2016 4:28 PM
>> >>> To: users@cloudstack.apache.org
>> >>> Subject: Re: Unable to create a New VM - Error message: Unable to
>> start
>> >>> instance due to Unable to get answer that is of class
>> >>> com.cloud.agent.api.StartAnswer
>> >>>
>> >>> Hi Simon,
>> >>>
>> >>> I will plan to restart agents, libvirtd and management services.
>> >>>
>> >>> How do I clear hung jobs.
>> >>>
>> >>> Your help is much appreciated,
>> >>> Nezar.
>> >>>
>> >>> On 22 March 2016 at 23:03, Simon Weller <sw...@ena.com> wrote:
>> >>>
>> >>>> Nezar,
>> >>>>
>> >>>> I've looked through the code related to your release and it seems
>> that
>> >> in
>> >>>> order to throw this exception, 10 attempts have been made to retrieve
>> >> the
>> >>>> disk information prior to the exception being thrown.
>> >>>>
>> >>>> It appears there is something really odd going on here. I'm starting
>> >> to
>> >>>> wonder whether it might be worth while for you to stop all of your
>> >>> agents,
>> >>>> plus management server processes, restart libvirtd on each of the
>> hosts
>> >>> and
>> >>>> then bring the agents back up prior to starting the management
>> servers.
>> >>>>
>> >>>> There are a bunch of logs the pastebin you posted that seems to
>> >> indicate
>> >>>> you have some jobs hung, for example:
>> >>>>
>> >>>> 2016-03-14 21:45:31,828 WARN [o.a.c.f.j.i.AsyncJobMonitor]
>> >>>> (Timer-1:ctx-0c090bd7) Task (job-9066) has been pending for 1660851
>> >>> seconds
>> >>>> 2016-03-14 21:45:31,828 WARN [o.a.c.f.j.i.AsyncJobMonitor]
>> >>>> (Timer-1:ctx-0c090bd7) Task (job-8133) has been pending for 4253003
>> >>> seconds
>> >>>>
>> >>>> Restarting the agents, libvirtd and management will not cause an
>> >> outage
>> >>>> to your cloud, but I would suggest, if you decided to do this, you
>> >> should
>> >>>> complete it in a maintenance window.
>> >>>>
>> >>>> - Si
>> >>>>
>> >>>> ________________________________________
>> >>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>> >>>> Sent: Sunday, March 20, 2016 10:27 PM
>> >>>> To: users@cloudstack.apache.org
>> >>>> Subject: Re: Unable to create a New VM - Error message: Unable to
>> start
>> >>>> instance due to Unable to get answer that is of class
>> >>>> com.cloud.agent.api.StartAnswer
>> >>>>
>> >>>> Hi Simon,
>> >>>>
>> >>>> Please see answers to your questions below. I am not sure if I
>> provided
>> >>> the
>> >>>> correct answer for the primary and secondary storage configuration.
>> >>> Please
>> >>>> let me know if this is not what you are after:
>> >>>>
>> >>>>
>> >>>> Can you screenshot and post somewhere your secondary storage
>> >>> configuration
>> >>>> and your primary storage configuration from the UI with the details?
>> >>>>
>> >>>> mysql> select * from configuration where name like '%secondary%'\G
>> >>>> *************************** 1. row ***************************
>> >>>> category: Account Defaults
>> >>>> instance: DEFAULT
>> >>>> component: management-server
>> >>>> name: max.account.secondary.storage
>> >>>> value: 400
>> >>>> description: The default maximum secondary storage space (in GiB)
>> >> that
>> >>>> can be used for an account
>> >>>> default_value: 400
>> >>>> updated: NULL
>> >>>> scope: NULL
>> >>>> is_dynamic: 0
>> >>>> *************************** 2. row ***************************
>> >>>> category: Project Defaults
>> >>>> instance: DEFAULT
>> >>>> component: management-server
>> >>>> name: max.project.secondary.storage
>> >>>> value: 400
>> >>>> description: The default maximum secondary storage space (in GiB)
>> >> that
>> >>>> can be used for a project
>> >>>> default_value: 400
>> >>>> updated: NULL
>> >>>> scope: NULL
>> >>>> is_dynamic: 0
>> >>>> *************************** 3. row ***************************
>> >>>> category: Hidden
>> >>>> instance: DEFAULT
>> >>>> component: management-server
>> >>>> name: secondary.storage.vm
>> >>>> value: +rwRhJOXPymbQOuAMLnhiw==
>> >>>> description: Deploys a VM per zone to manage secondary storage if
>> >> true,
>> >>>> otherwise secondary storage is mounted on management server
>> >>>> default_value: NULL
>> >>>> updated: NULL
>> >>>> scope: NULL
>> >>>> is_dynamic: 0
>> >>>> *************************** 4. row ***************************
>> >>>> category: Advanced
>> >>>> instance: DEFAULT
>> >>>> component: ExtensionRegistry
>> >>>> name: secondary.storage.vm.allocators.exclude
>> >>>> value: NULL
>> >>>> description: Extensions to exclude from being registered
>> >>>> default_value: NULL
>> >>>> updated: 2014-12-04 00:18:15
>> >>>> scope: Global
>> >>>> is_dynamic: 0
>> >>>> *************************** 5. row ***************************
>> >>>> category: NetworkManager
>> >>>> instance: DEFAULT
>> >>>> component: management-server
>> >>>> name: vm.network.nic.max.secondary.ipaddresses
>> >>>> value: NULL
>> >>>> description: Specify the number of secondary ip addresses per nic
>> per
>> >>> vm
>> >>>> default_value: 256
>> >>>> updated: NULL
>> >>>> scope: NULL
>> >>>> is_dynamic: 0
>> >>>> 5 rows in set (0.00 sec)
>> >>>>
>> >>>>
>> >>>> mysql> select * from configuration where name like '%primary%'\G
>> >>>> *************************** 1. row ***************************
>> >>>> category: Account Defaults
>> >>>> instance: DEFAULT
>> >>>> component: management-server
>> >>>> name: max.account.primary.storage
>> >>>> value: 200
>> >>>> description: The default maximum primary storage space (in GiB)
>> that
>> >>> can
>> >>>> be used for an account
>> >>>> default_value: 200
>> >>>> updated: NULL
>> >>>> scope: NULL
>> >>>> is_dynamic: 0
>> >>>> *************************** 2. row ***************************
>> >>>> category: Project Defaults
>> >>>> instance: DEFAULT
>> >>>> component: management-server
>> >>>> name: max.project.primary.storage
>> >>>> value: 200
>> >>>> description: The default maximum primary storage space (in GiB)
>> that
>> >>> can
>> >>>> be used for a project
>> >>>> default_value: 200
>> >>>> updated: NULL
>> >>>> scope: NULL
>> >>>> is_dynamic: 0
>> >>>> *************************** 3. row ***************************
>> >>>> category: Storage
>> >>>> instance: DEFAULT
>> >>>> component: TemplateManager
>> >>>> name: primary.storage.download.wait
>> >>>> value: 10800
>> >>>> description: In second, timeout for download template to primary
>> >>> storage
>> >>>> default_value: NULL
>> >>>> updated: NULL
>> >>>> scope: NULL
>> >>>> is_dynamic: 0
>> >>>> 3 rows in set (0.00 sec)
>> >>>>
>> >>>>
>> >>>>
>> >>>> Also, what is the Cloudstack version you are running and what is the
>> OS
>> >>>> you're running it on?
>> >>>>
>> >>>> Cloudstack 4.4.1
>> >>>>
>> >>>> Centos 6.6
>> >>>>
>> >>>> Are you using actimeo=0 in your NFS mount line?
>> >>>>
>> >>>> Primary storage on Hypervisors has the following parameters
>> >>>> 10.61.154.10:/Cloudstack/primary on
>> >>>> /mnt/b3647480-5709-36eb-9ddd-e2c666f30b57 type nfs
>> >>>> (rw,noatime,nfsvers=3,proto=tcp,hard,nolock,intr,addr=10.61.154.10)
>> >>>>
>> >>>>
>> >>>> Secondary storage on SSVM has the following parameters
>> >>>> 10.61.154.10:/Cloudstack/secondary on
>> >>>> /mnt/SecStorage/d11d5c92-6950-30e8-a0b5-3cd46a17ec14 type nfs4
>> >>>>
>> >>>>
>> >>>
>> >>
>> (rw,relatime,vers=4,rsize=262144,wsize=262144,namlen=255,acdirmin=0,acdirmax=0,soft,proto=tcp,port=0,timeo=133,retrans=
>> 2147483647
>> ,sec=sys,clientaddr=10.61.154.70,minorversion=0,local_lock=none,addr=10.61.154.10)
>> >>>>
>> >>>>
>> >>>> Thanks & Regards,
>> >>>> Nezar.
>> >>>>
>> >>>> On 18 March 2016 at 23:50, Simon Weller <sw...@ena.com> wrote:
>> >>>>
>> >>>>> Also, what is the Cloudstack version you are running and what is the
>> >> OS
>> >>>>> you're running it on?
>> >>>>>
>> >>>>> Are you using actimeo=0 in your NFS mount line?
>> >>>>>
>> >>>>> ________________________________________
>> >>>>> From: Simon Weller <sw...@ena.com>
>> >>>>> Sent: Friday, March 18, 2016 7:40 AM
>> >>>>> To: users@cloudstack.apache.org
>> >>>>> Subject: Re: Unable to create a New VM - Error message: Unable to
>> >> start
>> >>>>> instance due to Unable to get answer that is of class
>> >>>>> com.cloud.agent.api.StartAnswer
>> >>>>>
>> >>>>> Nexar,
>> >>>>>
>> >>>>> That's normal behaviour. You can only ssh to the link local on the
>> >> host
>> >>>>> where the VM resides.
>> >>>>>
>> >>>>> Can you screenshot and post somewhere your secondary storage
>> >>>> configuration
>> >>>>> and your primary storage configuration from the UI with the details?
>> >>>>>
>> >>>>> - Si
>> >>>>>
>> >>>>> ________________________________________
>> >>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>> >>>>> Sent: Thursday, March 17, 2016 8:19 PM
>> >>>>> To: users@cloudstack.apache.org
>> >>>>> Subject: Re: Unable to create a New VM - Error message: Unable to
>> >> start
>> >>>>> instance due to Unable to get answer that is of class
>> >>>>> com.cloud.agent.api.StartAnswer
>> >>>>>
>> >>>>> Hi Simon,
>> >>>>>
>> >>>>> I just noticed that from other Hosts I can't ssh into the SSVM:
>> >>>>>
>> >>>>> [root@eqx-cs-cmp-01 ~]# ssh -i /root/.ssh/id_rsa.cloud -p 3922
>> >>>>> root@169.254.3.161
>> >>>>> ssh: connect to host 169.254.3.161 port 3922: No route to host
>> >>>>>
>> >>>>>
>> >>>>> I can only ssh from one Host:
>> >>>>> [root@eqx-cs-cmp-06 ~]# ssh -i /root/.ssh/id_rsa.cloud -p 3922
>> >>>>> root@169.254.3.161
>> >>>>> Linux s-11-VM 3.2.0-4-amd64 #1 SMP Debian 3.2.57-3 x86_64
>> >>>>>
>> >>>>> Thanks & Regards,
>> >>>>> Nezar.
>> >>>>>
>> >>>>> On 18 March 2016 at 10:36, Nezar Madbouh <
>> >> nezar.madbouh@ipscape.com.au
>> >>>>
>> >>>>> wrote:
>> >>>>>
>> >>>>>> Hi Simon,
>> >>>>>>
>> >>>>>> Yes existing VMs are successfully running. Yesterday I was able to
>> >>>> create
>> >>>>>> a VM in Dev Environment but I can't create VMs in Prod environment.
>> >>>>>>
>> >>>>>> Thanks,
>> >>>>>> Nezar.
>> >>>>>>
>> >>>>>> On 18 March 2016 at 09:12, Simon Weller <sw...@ena.com> wrote:
>> >>>>>>
>> >>>>>>> I'm assuming you have other VMs successfully running, correct? So
>> >>> you
>> >>>>> can
>> >>>>>>> write to your primary storage.
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> ________________________________________
>> >>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>> >>>>>>> Sent: Thursday, March 17, 2016 4:43 PM
>> >>>>>>> To: users@cloudstack.apache.org
>> >>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable to
>> >>>> start
>> >>>>>>> instance due to Unable to get answer that is of class
>> >>>>>>> com.cloud.agent.api.StartAnswer
>> >>>>>>>
>> >>>>>>> Hi Simon and Shweta,
>> >>>>>>>
>> >>>>>>> Please see the result below:
>> >>>>>>>
>> >>>>>>> root@s-11-VM:~# /usr/local/cloud/systemvm/ssvm-check.sh
>> >>>>>>> ================================================
>> >>>>>>> First DNS server is 8.8.8.8
>> >>>>>>> PING 8.8.8.8 (8.8.8.8): 48 data bytes
>> >>>>>>> 56 bytes from 8.8.8.8: icmp_seq=0 ttl=57 time=5.209 ms
>> >>>>>>> 56 bytes from 8.8.8.8: icmp_seq=1 ttl=57 time=0.936 ms
>> >>>>>>> --- 8.8.8.8 ping statistics ---
>> >>>>>>> 2 packets transmitted, 2 packets received, 0% packet loss
>> >>>>>>> round-trip min/avg/max/stddev = 0.936/3.072/5.209/2.137 ms
>> >>>>>>> Good: Can ping DNS server
>> >>>>>>> ================================================
>> >>>>>>> Good: DNS resolves download.cloud.com
>> >>>>>>> ================================================
>> >>>>>>> nfs is currently mounted
>> >>>>>>> Mount point is
>> >> /mnt/SecStorage/d11d5c92-6950-30e8-a0b5-3cd46a17ec14
>> >>>>>>> Good: Can write to mount point
>> >>>>>>> ================================================
>> >>>>>>> Management server is 10.61.153.20. Checking connectivity.
>> >>>>>>> Good: Can connect to management server port 8250
>> >>>>>>> ================================================
>> >>>>>>> Good: Java process is running
>> >>>>>>> ================================================
>> >>>>>>> Tests Complete. Look for ERROR or WARNING above.
>> >>>>>>> root@s-11-VM:~#
>> >>>>>>>
>> >>>>>>> Thanks & Regards,
>> >>>>>>> Nezar.
>> >>>>>>>
>> >>>>>>> On 17 March 2016 at 09:42, Simon Weller <sw...@ena.com> wrote:
>> >>>>>>>
>> >>>>>>>> Nezar,
>> >>>>>>>>
>> >>>>>>>> Log into the Cloudstack UI.
>> >>>>>>>>
>> >>>>>>>> Click on "Infrastructure" on the left menu. Click on View All
>> >>> below
>> >>>>>>>> "System VMs". find the item marked "Secondary Storage VM" and
>> >>> then
>> >>>>>>> click
>> >>>>>>>> on the Name of that VM on the left (it will be s-XXXX-VM).
>> >> You'll
>> >>>> see
>> >>>>> a
>> >>>>>>>> bunch of information about the VM. Find the Host line and also
>> >> the
>> >>>>> Link
>> >>>>>>>> Local IP Address line and then follow the original instructions
>> >> in
>> >>>> my
>> >>>>>>>> previous email.
>> >>>>>>>>
>> >>>>>>>> - Si
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> ________________________________________
>> >>>>>>>> From: Shweta Agarwal <sh...@accelerite.com>
>> >>>>>>>> Sent: Wednesday, March 16, 2016 12:16 AM
>> >>>>>>>> To: users@cloudstack.apache.org
>> >>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable
>> >> to
>> >>>>> start
>> >>>>>>>> instance due to Unable to get answer that is of class
>> >>>>>>>> com.cloud.agent.api.StartAnswer
>> >>>>>>>>
>> >>>>>>>> Hi Nezar,
>> >>>>>>>>
>> >>>>>>>> You can find host name on detail view of SSVM in UI .
>> >>>>>>>> You can also use list api command listSystemVms to get the host
>> >>>>> details
>> >>>>>>> of
>> >>>>>>>> system vms.
>> >>>>>>>>
>> >>>>>>>> Thanks
>> >>>>>>>> Shweta
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> ________________________________________
>> >>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>> >>>>>>>> Sent: Wednesday, March 16, 2016 9:58 AM
>> >>>>>>>> To: users@cloudstack.apache.org
>> >>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable
>> >> to
>> >>>>> start
>> >>>>>>>> instance due to Unable to get answer that is of class
>> >>>>>>>> com.cloud.agent.api.StartAnswer
>> >>>>>>>>
>> >>>>>>>> Hi Simon,
>> >>>>>>>>
>> >>>>>>>> How can I find out which host the SSVM is on?
>> >>>>>>>>
>> >>>>>>>> Thanks & Regards,
>> >>>>>>>> Nezar.
>> >>>>>>>>
>> >>>>>>>> On 16 March 2016 at 12:28, Simon Weller <sw...@ena.com>
>> >> wrote:
>> >>>>>>>>
>> >>>>>>>>> Can you ssh into the Secondary Storage VM?
>> >>>>>>>>>
>> >>>>>>>>> This can be done by getting the link-local-ip address from the
>> >>>>>>>>> infrastructure menu in the UI and then sshing into the host
>> >> the
>> >>>> SSVM
>> >>>>>>> is
>> >>>>>>>> on
>> >>>>>>>>> and issuing this command as root: ssh -i
>> >> /root/.ssh/id_rsa.cloud
>> >>>> -p
>> >>>>>>> 3922
>> >>>>>>>>> root@link-local-ip (replace link-local-ip with the link
>> >> local
>> >>>> ip).
>> >>>>>>>>>
>> >>>>>>>>> Try running the SSVM diangnostic script:
>> >>>>>>>>> usr/local/cloud/systemvm/ssvm-check.sh
>> >>>>>>>>>
>> >>>>>>>>> - Si
>> >>>>>>>>> ________________________________________
>> >>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>> >>>>>>>>> Sent: Tuesday, March 15, 2016 7:49 PM
>> >>>>>>>>> To: users@cloudstack.apache.org
>> >>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable
>> >>> to
>> >>>>>>> start
>> >>>>>>>>> instance due to Unable to get answer that is of class
>> >>>>>>>>> com.cloud.agent.api.StartAnswer
>> >>>>>>>>>
>> >>>>>>>>> Hi Simon,
>> >>>>>>>>>
>> >>>>>>>>> Please see details below:
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>> mysql> use cloud; select id, name, status, type from host
>> >> where
>> >>>> id =
>> >>>>>>> 5;
>> >>>>>>>>> Reading table information for completion of table and column
>> >>> names
>> >>>>>>>>> You can turn off this feature to get a quicker startup with -A
>> >>>>>>>>>
>> >>>>>>>>> Database changed
>> >>>>>>>>> +----+------------------------------+--------+---------+
>> >>>>>>>>> | id | name | status | type |
>> >>>>>>>>> +----+------------------------------+--------+---------+
>> >>>>>>>>> | 5 | eqx-cs-cmp-02.ipscape.com.au | up | routing |
>> >>>>>>>>> +----+------------------------------+--------+---------+
>> >>>>>>>>> 1 row in set (0.00 sec)
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>> cat /var/log/cloudstack/agent/agent.log
>> >>>>>>>>> com.cloud.utils.exception.CloudRuntimeException:
>> >>>>>>>>> com.cloud.utils.exception.CloudRuntimeException: Can't find
>> >>>>>>>>> volume:089ff3ca-9bff-402c-a14e-ea7a35aaad5c
>> >>>>>>>>> at
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>
>> com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.getPhysicalDisk(KVMStoragePoolManager.java:262)
>> >>>>>>>>> at
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>
>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.createVbd(LibvirtComputingResource.java:3921)
>> >>>>>>>>> at
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>
>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(LibvirtComputingResource.java:3789)
>> >>>>>>>>> at
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>
>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1333)
>> >>>>>>>>> at
>> >> com.cloud.agent.Agent.processRequest(Agent.java:501)
>> >>>>>>>>> at
>> >>>>>>>> com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:808)
>> >>>>>>>>> at com.cloud.utils.nio.Task.run(Task.java:84)
>> >>>>>>>>> at
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> >>>>>>>>> at
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> >>>>>>>>> at java.lang.Thread.run(Thread.java:745)
>> >>>>>>>>> agent.log.2016-03-03.gz (END)
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>> [root@eqx-cs-cmp-02 agent]# virsh pool-list
>> >>>>>>>>> Name State Autostart
>> >>>>>>>>> -----------------------------------------
>> >>>>>>>>> 2d0967f8-50d6-4d6d-87b7-4da0596f6924 active no
>> >>>>>>>>> b3647480-5709-36eb-9ddd-e2c666f30b57 active no
>> >>>>>>>>> e3831246-ff55-4af4-9964-a17565edaccc active no
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>> [Root @ eqx-cs-cmp-02 agent] # df -h
>> >>>>>>>>> Filesystem Size Used Avail Use% Mounted on
>> >>>>>>>>> /dev/mapper/vg_root-lv_root
>> >>>>>>>>> 203G 215G 1.3G 1% /
>> >>>>>>>>> tmpfs 48g 48g 0 0% / dev / shm
>> >>>>>>>>> / Dev / sda1 477m 427m 25M 6% / boot
>> >>>>>>>>> xx.xx.xx.xx:/Cloudstack/primary
>> >>>>>>>>> 7.6T 4.1T 3.5T 54%
>> >>>>>>>>> /mnt/b3647480-5709-36eb-9ddd-e2c666f30b57
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>> I also noticed the following in cloudstack-agent.err
>> >>>>>>>>>
>> >>>>>>>>> libvirt: QEMU Driver error : Domain not found: no domain with
>> >>>>> matching
>> >>>>>>>> name
>> >>>>>>>>> 'i-11-600-VM'
>> >>>>>>>>> libvirt: Storage Driver error : Storage volume not found: no
>> >>>> storage
>> >>>>>>> vol
>> >>>>>>>>> with matching name '089ff3ca-9bff-402c-a14e-ea7a35aaad5c'
>> >>>>>>>>>
>> >>>>>>>>> Your help is much appreciated,
>> >>>>>>>>> Nezar
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>> On 16 March 2016 at 11:03, Simon Weller <sw...@ena.com>
>> >>> wrote:
>> >>>>>>>>>
>> >>>>>>>>>> Nezar,
>> >>>>>>>>>>
>> >>>>>>>>>> ok, I just took a look at a pastebin you posted.
>> >>>>>>>>>>
>> >>>>>>>>>> I see it complaining about the inability to find a volume
>> >>> prior
>> >>>> to
>> >>>>>>>>>> throwing the host exception.
>> >>>>>>>>>>
>> >>>>>>>>>> Firstly, let's find out who host 5 is. Can you log into your
>> >>>> MySQL
>> >>>>>>>> server
>> >>>>>>>>>> and run this:
>> >>>>>>>>>>
>> >>>>>>>>>> use cloud;select id,name,status,type from host where id=5;
>> >>>>>>>>>>
>> >>>>>>>>>> Can you post the results?
>> >>>>>>>>>>
>> >>>>>>>>>> If it's a host, can you post the
>> >>>>> /var/log/cloudstack/agent/agent.log
>> >>>>>>>> from
>> >>>>>>>>>> that host?
>> >>>>>>>>>>
>> >>>>>>>>>> And also run virsh pool-list on the host in question?
>> >>>>>>>>>>
>> >>>>>>>>>> - Si
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> ________________________________________
>> >>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>> >>>>>>>>>> Sent: Tuesday, March 15, 2016 6:11 PM
>> >>>>>>>>>> To: users@cloudstack.apache.org
>> >>>>>>>>>> Subject: Re: Unable to create a New VM - Error message:
>> >> Unable
>> >>>> to
>> >>>>>>> start
>> >>>>>>>>>> instance due to Unable to get answer that is of class
>> >>>>>>>>>> com.cloud.agent.api.StartAnswer
>> >>>>>>>>>>
>> >>>>>>>>>> Hi Simon,
>> >>>>>>>>>>
>> >>>>>>>>>> When you say "double check your routers tied to the
>> >> network",
>> >>> do
>> >>>>> you
>> >>>>>>>> mean
>> >>>>>>>>>> the physical routers or the virtual routers?
>> >>>>>>>>>>
>> >>>>>>>>>> When I try to create a VM in Prod Environment it fails. I
>> >>> can't
>> >>>>> ping
>> >>>>>>>> the
>> >>>>>>>>>> Prod virtual router from management server or Host.
>> >>>>>>>>>>
>> >>>>>>>>>> But I can ping Dev virtual router from management server and
>> >>>> Host
>> >>>>>>> and I
>> >>>>>>>>> was
>> >>>>>>>>>> able to create a VM in Dev Environment.
>> >>>>>>>>>>
>> >>>>>>>>>> How can I troubleshoot the Prod Virtual router. I launched
>> >> the
>> >>>>>>> console
>> >>>>>>>>> but
>> >>>>>>>>>> couldn't login.
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> Please note that we need to take into consideration that the
>> >>>>>>> affected
>> >>>>>>>>>> network is the Production network and we need to minimise
>> >>>> service
>> >>>>>>>> outage
>> >>>>>>>>> as
>> >>>>>>>>>> much as possible.
>> >>>>>>>>>>
>> >>>>>>>>>> Your help is much appreciated,
>> >>>>>>>>>> Nezar
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> On 16 March 2016 at 08:37, Simon Weller <sw...@ena.com>
>> >>>> wrote:
>> >>>>>>>>>>
>> >>>>>>>>>>> I've seen this before where a Virtual Router has crashed
>> >> and
>> >>>> the
>> >>>>>>>>>>> management server can't communicate with it.
>> >>>>>>>>>>> Are you using an advanced network setup?
>> >>>>>>>>>>>
>> >>>>>>>>>>> If so, double check your routers tied to the network where
>> >>> you
>> >>>>> are
>> >>>>>>>>>>> attempting to start an instance.
>> >>>>>>>>>>>
>> >>>>>>>>>>> - Si
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> ________________________________________
>> >>>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>> >>>>>>>>>>> Sent: Tuesday, March 15, 2016 4:02 PM
>> >>>>>>>>>>> To: users@cloudstack.apache.org
>> >>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message:
>> >>> Unable
>> >>>>> to
>> >>>>>>>> start
>> >>>>>>>>>>> instance due to Unable to get answer that is of class
>> >>>>>>>>>>> com.cloud.agent.api.StartAnswer
>> >>>>>>>>>>>
>> >>>>>>>>>>> Hi Glenn,
>> >>>>>>>>>>>
>> >>>>>>>>>>> We have 16 hosts, all KVM.
>> >>>>>>>>>>>
>> >>>>>>>>>>> The error is not specific to Host 5. Every time I try to
>> >>>> create
>> >>>>> a
>> >>>>>>> VM
>> >>>>>>>> it
>> >>>>>>>>>>> gives me the same error message but on different host.
>> >>>>>>>>>>>
>> >>>>>>>>>>> How do I restart the agent on the host. Is it service
>> >>>> affecting?
>> >>>>>>>>>>>
>> >>>>>>>>>>> Thanks very much for your help.
>> >>>>>>>>>>> Neza
>> >>>>>>>>>>>
>> >>>>>>>>>>> On 15 March 2016 at 17:37, Glenn Wagner <
>> >>>>>>> glenn.wagner@shapeblue.com>
>> >>>>>>>>>>> wrote:
>> >>>>>>>>>>>
>> >>>>>>>>>>>> Hi,
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> Can you explain your environment first , how many hosts
>> >> ,
>> >>>> what
>> >>>>>>> is
>> >>>>>>>>> your
>> >>>>>>>>>>>> hypervisors (KVM,Xenserver,VMware)
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> The error seems to be with your Host 5 Hypervisor
>> >>>>>>>>>>>> Resource [Host:5] is unreachable:
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> Go to the infrastructure section and see what host5 is
>> >>>>>>> reporting,
>> >>>>>>>>>>>> Have you tried to restart the agent on the Host?
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> Thanks
>> >>>>>>>>>>>> Glenn
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> [image: ShapeBlue] <http://www.shapeblue.com>
>> >>>>>>>>>>>> Glenn Wagner
>> >>>>>>>>>>>> Senior Consultant , ShapeBlue
>> >>>>>>>>>>>> d: * | s: +27 21 527 0091*
>> >>>>>>> <%7C%20s:%20+27%2021%20527%200091> |
>> >>>>>>>>> m:
>> >>>>>>>>>>>> *+27 73 917 4111* <+27%2073%20917%204111>
>> >>>>>>>>>>>> e: *glenn.wagner@shapeblue.com | t: *
>> >>>>>>>>>>>> <glenn.wagner@shapeblue.com%20%7C%20t:> | w: *
>> >>>>>>> www.shapeblue.com
>> >>>>>>>> *
>> >>>>>>>>>>>> <http://www.shapeblue.com>
>> >>>>>>>>>>>> a:
>> >>>>>>>>>>>> 2nd Floor, Oudehuis Centre, 122 Main Rd, Somerset West
>> >>> Cape
>> >>>>> Town
>> >>>>>>>> 7130
>> >>>>>>>>>>> South Africa
>> >>>>>>>>>>>> Shape Blue Ltd is a company incorporated in England &
>> >>> Wales.
>> >>>>>>>>> ShapeBlue
>> >>>>>>>>>>>> Services India LLP is a company incorporated in India
>> >> and
>> >>> is
>> >>>>>>>> operated
>> >>>>>>>>>>> under
>> >>>>>>>>>>>> license from Shape Blue Ltd. Shape Blue Brasil
>> >> Consultoria
>> >>>>> Ltda
>> >>>>>>> is
>> >>>>>>>> a
>> >>>>>>>>>>>> company incorporated in Brasil and is operated under
>> >>> license
>> >>>>>>> from
>> >>>>>>>>> Shape
>> >>>>>>>>>>>> Blue Ltd. ShapeBlue SA Pty Ltd is a company registered
>> >> by
>> >>>> The
>> >>>>>>>>> Republic
>> >>>>>>>>>> of
>> >>>>>>>>>>>> South Africa and is traded under license from Shape Blue
>> >>>> Ltd.
>> >>>>>>>>> ShapeBlue
>> >>>>>>>>>>> is
>> >>>>>>>>>>>> a registered trademark.
>> >>>>>>>>>>>> This email and any attachments to it may be confidential
>> >>> and
>> >>>>> are
>> >>>>>>>>>> intended
>> >>>>>>>>>>>> solely for the use of the individual to whom it is
>> >>>> addressed.
>> >>>>>>> Any
>> >>>>>>>>> views
>> >>>>>>>>>>> or
>> >>>>>>>>>>>> opinions expressed are solely those of the author and do
>> >>> not
>> >>>>>>>>>> necessarily
>> >>>>>>>>>>>> represent those of Shape Blue Ltd or related companies.
>> >> If
>> >>>> you
>> >>>>>>> are
>> >>>>>>>>> not
>> >>>>>>>>>>> the
>> >>>>>>>>>>>> intended recipient of this email, you must neither take
>> >>> any
>> >>>>>>> action
>> >>>>>>>>>> based
>> >>>>>>>>>>>> upon its contents, nor copy or show it to anyone. Please
>> >>>>> contact
>> >>>>>>>> the
>> >>>>>>>>>>> sender
>> >>>>>>>>>>>> if you believe you have received this email in error.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> -----Original Message-----
>> >>>>>>>>>>>> From: Nezar Madbouh [mailto:
>> >> nezar.madbouh@ipscape.com.au]
>> >>>>>>>>>>>> Sent: Tuesday, 15 March 2016 5:45 AM
>> >>>>>>>>>>>> To: users@cloudstack.apache.org
>> >>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message:
>> >>>> Unable
>> >>>>>>> to
>> >>>>>>>>> start
>> >>>>>>>>>>>> instance due to Unable to get answer that is of class
>> >>>>>>>>>>>> com.cloud.agent.api.StartAnswer
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> Hi Jeremy,
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> I have uploaded the contents to
>> >>>> http://pastebin.com/1J3mmgTh
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> your help is much appreciated,
>> >>>>>>>>>>>> Nezar.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> On 15 March 2016 at 13:46, Jeremy Peterson <
>> >>>>>>> jpeterson@acentek.net>
>> >>>>>>>>>>> wrote:
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>> I don't see any attachments can you create a pastebin.
>> >>>>> That's
>> >>>>>>>>> easier
>> >>>>>>>>>>>>> to look through as I think attachments are removed
>> >>> before
>> >>>>>>> being
>> >>>>>>>>> sent
>> >>>>>>>>>>>>> to user group.
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Jeremy
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> -------- Original message --------
>> >>>>>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>> >>>>>>>>>>>>> Date: 3/14/2016 6:38 PM (GMT-06:00)
>> >>>>>>>>>>>>> To: users@cloudstack.apache.org
>> >>>>>>>>>>>>> Subject: Unable to create a New VM - Error message:
>> >>> Unable
>> >>>>> to
>> >>>>>>>> start
>> >>>>>>>>>>>>> instance due to Unable to get answer that is of class
>> >>>>>>>>>>>>> com.cloud.agent.api.StartAnswer
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Hi Shweta and Sadhu,
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Please find attached managerment-server.log which
>> >>> contains
>> >>>>> the
>> >>>>>>>> logs
>> >>>>>>>>>>>>> since I started creating a new VM (VM id 600).
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Please note that we don't have agent.log files
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Your help is much appreciated.
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Regards,
>> >>>>>>>>>>>>> Nezar.
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>> Find out more about ShapeBlue and our range of
>> >> CloudStack
>> >>>>>>> related
>> >>>>>>>>>>> services:
>> >>>>>>>>>>>> IaaS Cloud Design & Build
>> >>>>>>>>>>>> <http://shapeblue.com/iaas-cloud-design-and-build//> |
>> >>>>> CSForge
>> >>>>>>> -
>> >>>>>>>>> rapid
>> >>>>>>>>>>>> IaaS deployment framework <
>> >> http://shapeblue.com/csforge/>
>> >>>>>>>>>>>> CloudStack Consulting <
>> >>>>>>>> http://shapeblue.com/cloudstack-consultancy/>
>> >>>>>>>>> |
>> >>>>>>>>>>> CloudStack
>> >>>>>>>>>>>> Software Engineering
>> >>>>>>>>>>>> <http://shapeblue.com/cloudstack-software-engineering/>
>> >>>>>>>>>>>> CloudStack Infrastructure Support
>> >>>>>>>>>>>> <
>> >> http://shapeblue.com/cloudstack-infrastructure-support/>
>> >>> |
>> >>>>>>>>> CloudStack
>> >>>>>>>>>>>> Bootcamp Training Courses <
>> >>>>>>>> http://shapeblue.com/cloudstack-training/
>> >>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> DISCLAIMER
>> >>>>>>>> ==========
>> >>>>>>>> This e-mail may contain privileged and confidential information
>> >>>> which
>> >>>>> is
>> >>>>>>>> the property of Accelerite, a Persistent Systems business. It is
>> >>>>>>> intended
>> >>>>>>>> only for the use of the individual or entity to which it is
>> >>>> addressed.
>> >>>>>>> If
>> >>>>>>>> you are not the intended recipient, you are not authorized to
>> >>> read,
>> >>>>>>> retain,
>> >>>>>>>> copy, print, distribute or use this message. If you have
>> >> received
>> >>>> this
>> >>>>>>>> communication in error, please notify the sender and delete all
>> >>>> copies
>> >>>>>>> of
>> >>>>>>>> this message. Accelerite, a Persistent Systems business does not
>> >>>>> accept
>> >>>>>>> any
>> >>>>>>>> liability for virus infected mails.
>> >>>>>>>>
>> >>>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>
>> >>
>> >
>>
>
>
Re: Unable to create a New VM - Error message: Unable to start
instance due to Unable to get answer that is of class com.cloud.agent.api.StartAnswer
Posted by Nezar Madbouh <ne...@ipscape.com.au>.
Hi Ilya,
I have run the script and it didn't return any results:
[root@eqx-cs-mgt-01 ~]# ./async_jobs.sh
[root@eqx-cs-mgt-01 ~]#
I also ran the command inside mysql:
mysql> SELECT id, user_id, account_id, instance_type, instance_id, job_cmd,
job_cmd_info, job_dispatcher, job_pending_signals FROM cloud.async_job
where job_status=0 and job_dispatcher not like "pseudoJobDispatcher";
Empty set (0.00 sec)
your help is much appreciated,
Nezar.
On 14 April 2016 at 16:49, ilya <il...@gmail.com> wrote:
> It appears you have pending jobs that are not letting you go through.
>
> Usually stop and starting management service should be enough.
>
> However, try running this script from your management server to show us
> what you have running in async_job table.
>
>
>
> > #!/bin/bash
> > DATESTAMP=$(date +%m%d%y-%H%M%S)
> > DBPASS=$(java -classpath
> /usr/share/cloudstack-common/lib/jasypt-1.9.0.jar
> org.jasypt.intf.cli.JasyptPBEStringDecryptionCLI input="$(cat
> /etc/cloudstack/management/db.properties | grep db.cloud.password | awk
> -F'(' '{print $2}' | sed 's/)//g')" password="$(cat
> /etc/cloudstack/management/key)" | grep -A2 OUTPUT | tail -1)
> > DBHOST=$(cat /etc/cloudstack/management/db.properties | grep
> db.cloud.host | awk -F'=' '{print $2}' | tail -1 )
> > DBUSER=$(cat /etc/cloudstack/management/db.properties | grep
> db.cloud.username | awk -F'=' '{print $2}')
> > DB=$(cat /etc/cloudstack/management/db.properties | grep db.cloud.name
> | awk -F'=' '{print $2}')
> > DBPORT=$(cat /etc/cloudstack/management/db.properties | grep
> db.cloud.port | awk -F'=' '{print $2}')
> > MYSQLCMD="mysql -h $DBHOST -u $DBUSER -P $DBPORT -p$DBPASS $DB"
> > echo 'SELECT id, user_id, account_id, instance_type, instance_id,
> job_cmd, job_cmd_info, job_dispatcher, job_pending_signals FROM
> cloud.async_job where job_status=0 and job_dispatcher not like
> "pseudoJobDispatcher"' | $MYSQLCMD
>
>
>
>
> On 4/13/16 5:16 PM, Nezar Madbouh wrote:
> > Hi Simon,
> >
> > The following work was done but unfortunately the problem is not fixed:
> >
> >
> > on management server:
> >
> > service cloudstack-management stop
> >
> >
> > on the host:
> >
> > service cloudstack-agent stop
> >
> > service libvirtd restart
> >
> > service cloudstack-agent start
> >
> >
> > on management server:
> >
> > service cloudstack-management start
> >
> > Regards,
> > Nezar.
> >
> > On 24 March 2016 at 05:06, Simon Weller <sw...@ena.com> wrote:
> >
> >> Yes, I think that order is fine. Make sure the processes are indeed
> >> shutdown before moving on to the next step.
> >>
> >>
> >>
> >>
> >>
> >> ________________________________________
> >> From: Nezar Madbouh <ne...@ipscape.com.au>
> >> Sent: Tuesday, March 22, 2016 5:33 PM
> >> To: users@cloudstack.apache.org
> >> Subject: Re: Unable to create a New VM - Error message: Unable to start
> >> instance due to Unable to get answer that is of class
> >> com.cloud.agent.api.StartAnswer
> >>
> >> Hi Simon,
> >>
> >> Can you please check if the following sequence of action is correct:
> >>
> >> on management server:
> >>
> >> service cloudstack-management stop
> >>
> >>
> >> on the host:
> >>
> >> service cloudstack-agent stop
> >>
> >> service libvirtd restart
> >>
> >> service cloudstack-agent start
> >>
> >>
> >> on management server:
> >>
> >> service cloudstack-management start
> >>
> >> Thanks very much for your help.
> >> Nezar.
> >>
> >> On 23 March 2016 at 08:39, Simon Weller <sw...@ena.com> wrote:
> >>
> >>> Shutting down all the management services should clear the jobs.
> >>>
> >>>
> >>> ________________________________________
> >>> From: Nezar Madbouh <ne...@ipscape.com.au>
> >>> Sent: Tuesday, March 22, 2016 4:28 PM
> >>> To: users@cloudstack.apache.org
> >>> Subject: Re: Unable to create a New VM - Error message: Unable to start
> >>> instance due to Unable to get answer that is of class
> >>> com.cloud.agent.api.StartAnswer
> >>>
> >>> Hi Simon,
> >>>
> >>> I will plan to restart agents, libvirtd and management services.
> >>>
> >>> How do I clear hung jobs.
> >>>
> >>> Your help is much appreciated,
> >>> Nezar.
> >>>
> >>> On 22 March 2016 at 23:03, Simon Weller <sw...@ena.com> wrote:
> >>>
> >>>> Nezar,
> >>>>
> >>>> I've looked through the code related to your release and it seems that
> >> in
> >>>> order to throw this exception, 10 attempts have been made to retrieve
> >> the
> >>>> disk information prior to the exception being thrown.
> >>>>
> >>>> It appears there is something really odd going on here. I'm starting
> >> to
> >>>> wonder whether it might be worth while for you to stop all of your
> >>> agents,
> >>>> plus management server processes, restart libvirtd on each of the
> hosts
> >>> and
> >>>> then bring the agents back up prior to starting the management
> servers.
> >>>>
> >>>> There are a bunch of logs the pastebin you posted that seems to
> >> indicate
> >>>> you have some jobs hung, for example:
> >>>>
> >>>> 2016-03-14 21:45:31,828 WARN [o.a.c.f.j.i.AsyncJobMonitor]
> >>>> (Timer-1:ctx-0c090bd7) Task (job-9066) has been pending for 1660851
> >>> seconds
> >>>> 2016-03-14 21:45:31,828 WARN [o.a.c.f.j.i.AsyncJobMonitor]
> >>>> (Timer-1:ctx-0c090bd7) Task (job-8133) has been pending for 4253003
> >>> seconds
> >>>>
> >>>> Restarting the agents, libvirtd and management will not cause an
> >> outage
> >>>> to your cloud, but I would suggest, if you decided to do this, you
> >> should
> >>>> complete it in a maintenance window.
> >>>>
> >>>> - Si
> >>>>
> >>>> ________________________________________
> >>>> From: Nezar Madbouh <ne...@ipscape.com.au>
> >>>> Sent: Sunday, March 20, 2016 10:27 PM
> >>>> To: users@cloudstack.apache.org
> >>>> Subject: Re: Unable to create a New VM - Error message: Unable to
> start
> >>>> instance due to Unable to get answer that is of class
> >>>> com.cloud.agent.api.StartAnswer
> >>>>
> >>>> Hi Simon,
> >>>>
> >>>> Please see answers to your questions below. I am not sure if I
> provided
> >>> the
> >>>> correct answer for the primary and secondary storage configuration.
> >>> Please
> >>>> let me know if this is not what you are after:
> >>>>
> >>>>
> >>>> Can you screenshot and post somewhere your secondary storage
> >>> configuration
> >>>> and your primary storage configuration from the UI with the details?
> >>>>
> >>>> mysql> select * from configuration where name like '%secondary%'\G
> >>>> *************************** 1. row ***************************
> >>>> category: Account Defaults
> >>>> instance: DEFAULT
> >>>> component: management-server
> >>>> name: max.account.secondary.storage
> >>>> value: 400
> >>>> description: The default maximum secondary storage space (in GiB)
> >> that
> >>>> can be used for an account
> >>>> default_value: 400
> >>>> updated: NULL
> >>>> scope: NULL
> >>>> is_dynamic: 0
> >>>> *************************** 2. row ***************************
> >>>> category: Project Defaults
> >>>> instance: DEFAULT
> >>>> component: management-server
> >>>> name: max.project.secondary.storage
> >>>> value: 400
> >>>> description: The default maximum secondary storage space (in GiB)
> >> that
> >>>> can be used for a project
> >>>> default_value: 400
> >>>> updated: NULL
> >>>> scope: NULL
> >>>> is_dynamic: 0
> >>>> *************************** 3. row ***************************
> >>>> category: Hidden
> >>>> instance: DEFAULT
> >>>> component: management-server
> >>>> name: secondary.storage.vm
> >>>> value: +rwRhJOXPymbQOuAMLnhiw==
> >>>> description: Deploys a VM per zone to manage secondary storage if
> >> true,
> >>>> otherwise secondary storage is mounted on management server
> >>>> default_value: NULL
> >>>> updated: NULL
> >>>> scope: NULL
> >>>> is_dynamic: 0
> >>>> *************************** 4. row ***************************
> >>>> category: Advanced
> >>>> instance: DEFAULT
> >>>> component: ExtensionRegistry
> >>>> name: secondary.storage.vm.allocators.exclude
> >>>> value: NULL
> >>>> description: Extensions to exclude from being registered
> >>>> default_value: NULL
> >>>> updated: 2014-12-04 00:18:15
> >>>> scope: Global
> >>>> is_dynamic: 0
> >>>> *************************** 5. row ***************************
> >>>> category: NetworkManager
> >>>> instance: DEFAULT
> >>>> component: management-server
> >>>> name: vm.network.nic.max.secondary.ipaddresses
> >>>> value: NULL
> >>>> description: Specify the number of secondary ip addresses per nic
> per
> >>> vm
> >>>> default_value: 256
> >>>> updated: NULL
> >>>> scope: NULL
> >>>> is_dynamic: 0
> >>>> 5 rows in set (0.00 sec)
> >>>>
> >>>>
> >>>> mysql> select * from configuration where name like '%primary%'\G
> >>>> *************************** 1. row ***************************
> >>>> category: Account Defaults
> >>>> instance: DEFAULT
> >>>> component: management-server
> >>>> name: max.account.primary.storage
> >>>> value: 200
> >>>> description: The default maximum primary storage space (in GiB) that
> >>> can
> >>>> be used for an account
> >>>> default_value: 200
> >>>> updated: NULL
> >>>> scope: NULL
> >>>> is_dynamic: 0
> >>>> *************************** 2. row ***************************
> >>>> category: Project Defaults
> >>>> instance: DEFAULT
> >>>> component: management-server
> >>>> name: max.project.primary.storage
> >>>> value: 200
> >>>> description: The default maximum primary storage space (in GiB) that
> >>> can
> >>>> be used for a project
> >>>> default_value: 200
> >>>> updated: NULL
> >>>> scope: NULL
> >>>> is_dynamic: 0
> >>>> *************************** 3. row ***************************
> >>>> category: Storage
> >>>> instance: DEFAULT
> >>>> component: TemplateManager
> >>>> name: primary.storage.download.wait
> >>>> value: 10800
> >>>> description: In second, timeout for download template to primary
> >>> storage
> >>>> default_value: NULL
> >>>> updated: NULL
> >>>> scope: NULL
> >>>> is_dynamic: 0
> >>>> 3 rows in set (0.00 sec)
> >>>>
> >>>>
> >>>>
> >>>> Also, what is the Cloudstack version you are running and what is the
> OS
> >>>> you're running it on?
> >>>>
> >>>> Cloudstack 4.4.1
> >>>>
> >>>> Centos 6.6
> >>>>
> >>>> Are you using actimeo=0 in your NFS mount line?
> >>>>
> >>>> Primary storage on Hypervisors has the following parameters
> >>>> 10.61.154.10:/Cloudstack/primary on
> >>>> /mnt/b3647480-5709-36eb-9ddd-e2c666f30b57 type nfs
> >>>> (rw,noatime,nfsvers=3,proto=tcp,hard,nolock,intr,addr=10.61.154.10)
> >>>>
> >>>>
> >>>> Secondary storage on SSVM has the following parameters
> >>>> 10.61.154.10:/Cloudstack/secondary on
> >>>> /mnt/SecStorage/d11d5c92-6950-30e8-a0b5-3cd46a17ec14 type nfs4
> >>>>
> >>>>
> >>>
> >>
> (rw,relatime,vers=4,rsize=262144,wsize=262144,namlen=255,acdirmin=0,acdirmax=0,soft,proto=tcp,port=0,timeo=133,retrans=2147483647,sec=sys,clientaddr=10.61.154.70,minorversion=0,local_lock=none,addr=10.61.154.10)
> >>>>
> >>>>
> >>>> Thanks & Regards,
> >>>> Nezar.
> >>>>
> >>>> On 18 March 2016 at 23:50, Simon Weller <sw...@ena.com> wrote:
> >>>>
> >>>>> Also, what is the Cloudstack version you are running and what is the
> >> OS
> >>>>> you're running it on?
> >>>>>
> >>>>> Are you using actimeo=0 in your NFS mount line?
> >>>>>
> >>>>> ________________________________________
> >>>>> From: Simon Weller <sw...@ena.com>
> >>>>> Sent: Friday, March 18, 2016 7:40 AM
> >>>>> To: users@cloudstack.apache.org
> >>>>> Subject: Re: Unable to create a New VM - Error message: Unable to
> >> start
> >>>>> instance due to Unable to get answer that is of class
> >>>>> com.cloud.agent.api.StartAnswer
> >>>>>
> >>>>> Nexar,
> >>>>>
> >>>>> That's normal behaviour. You can only ssh to the link local on the
> >> host
> >>>>> where the VM resides.
> >>>>>
> >>>>> Can you screenshot and post somewhere your secondary storage
> >>>> configuration
> >>>>> and your primary storage configuration from the UI with the details?
> >>>>>
> >>>>> - Si
> >>>>>
> >>>>> ________________________________________
> >>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
> >>>>> Sent: Thursday, March 17, 2016 8:19 PM
> >>>>> To: users@cloudstack.apache.org
> >>>>> Subject: Re: Unable to create a New VM - Error message: Unable to
> >> start
> >>>>> instance due to Unable to get answer that is of class
> >>>>> com.cloud.agent.api.StartAnswer
> >>>>>
> >>>>> Hi Simon,
> >>>>>
> >>>>> I just noticed that from other Hosts I can't ssh into the SSVM:
> >>>>>
> >>>>> [root@eqx-cs-cmp-01 ~]# ssh -i /root/.ssh/id_rsa.cloud -p 3922
> >>>>> root@169.254.3.161
> >>>>> ssh: connect to host 169.254.3.161 port 3922: No route to host
> >>>>>
> >>>>>
> >>>>> I can only ssh from one Host:
> >>>>> [root@eqx-cs-cmp-06 ~]# ssh -i /root/.ssh/id_rsa.cloud -p 3922
> >>>>> root@169.254.3.161
> >>>>> Linux s-11-VM 3.2.0-4-amd64 #1 SMP Debian 3.2.57-3 x86_64
> >>>>>
> >>>>> Thanks & Regards,
> >>>>> Nezar.
> >>>>>
> >>>>> On 18 March 2016 at 10:36, Nezar Madbouh <
> >> nezar.madbouh@ipscape.com.au
> >>>>
> >>>>> wrote:
> >>>>>
> >>>>>> Hi Simon,
> >>>>>>
> >>>>>> Yes existing VMs are successfully running. Yesterday I was able to
> >>>> create
> >>>>>> a VM in Dev Environment but I can't create VMs in Prod environment.
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Nezar.
> >>>>>>
> >>>>>> On 18 March 2016 at 09:12, Simon Weller <sw...@ena.com> wrote:
> >>>>>>
> >>>>>>> I'm assuming you have other VMs successfully running, correct? So
> >>> you
> >>>>> can
> >>>>>>> write to your primary storage.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> ________________________________________
> >>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
> >>>>>>> Sent: Thursday, March 17, 2016 4:43 PM
> >>>>>>> To: users@cloudstack.apache.org
> >>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable to
> >>>> start
> >>>>>>> instance due to Unable to get answer that is of class
> >>>>>>> com.cloud.agent.api.StartAnswer
> >>>>>>>
> >>>>>>> Hi Simon and Shweta,
> >>>>>>>
> >>>>>>> Please see the result below:
> >>>>>>>
> >>>>>>> root@s-11-VM:~# /usr/local/cloud/systemvm/ssvm-check.sh
> >>>>>>> ================================================
> >>>>>>> First DNS server is 8.8.8.8
> >>>>>>> PING 8.8.8.8 (8.8.8.8): 48 data bytes
> >>>>>>> 56 bytes from 8.8.8.8: icmp_seq=0 ttl=57 time=5.209 ms
> >>>>>>> 56 bytes from 8.8.8.8: icmp_seq=1 ttl=57 time=0.936 ms
> >>>>>>> --- 8.8.8.8 ping statistics ---
> >>>>>>> 2 packets transmitted, 2 packets received, 0% packet loss
> >>>>>>> round-trip min/avg/max/stddev = 0.936/3.072/5.209/2.137 ms
> >>>>>>> Good: Can ping DNS server
> >>>>>>> ================================================
> >>>>>>> Good: DNS resolves download.cloud.com
> >>>>>>> ================================================
> >>>>>>> nfs is currently mounted
> >>>>>>> Mount point is
> >> /mnt/SecStorage/d11d5c92-6950-30e8-a0b5-3cd46a17ec14
> >>>>>>> Good: Can write to mount point
> >>>>>>> ================================================
> >>>>>>> Management server is 10.61.153.20. Checking connectivity.
> >>>>>>> Good: Can connect to management server port 8250
> >>>>>>> ================================================
> >>>>>>> Good: Java process is running
> >>>>>>> ================================================
> >>>>>>> Tests Complete. Look for ERROR or WARNING above.
> >>>>>>> root@s-11-VM:~#
> >>>>>>>
> >>>>>>> Thanks & Regards,
> >>>>>>> Nezar.
> >>>>>>>
> >>>>>>> On 17 March 2016 at 09:42, Simon Weller <sw...@ena.com> wrote:
> >>>>>>>
> >>>>>>>> Nezar,
> >>>>>>>>
> >>>>>>>> Log into the Cloudstack UI.
> >>>>>>>>
> >>>>>>>> Click on "Infrastructure" on the left menu. Click on View All
> >>> below
> >>>>>>>> "System VMs". find the item marked "Secondary Storage VM" and
> >>> then
> >>>>>>> click
> >>>>>>>> on the Name of that VM on the left (it will be s-XXXX-VM).
> >> You'll
> >>>> see
> >>>>> a
> >>>>>>>> bunch of information about the VM. Find the Host line and also
> >> the
> >>>>> Link
> >>>>>>>> Local IP Address line and then follow the original instructions
> >> in
> >>>> my
> >>>>>>>> previous email.
> >>>>>>>>
> >>>>>>>> - Si
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> ________________________________________
> >>>>>>>> From: Shweta Agarwal <sh...@accelerite.com>
> >>>>>>>> Sent: Wednesday, March 16, 2016 12:16 AM
> >>>>>>>> To: users@cloudstack.apache.org
> >>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable
> >> to
> >>>>> start
> >>>>>>>> instance due to Unable to get answer that is of class
> >>>>>>>> com.cloud.agent.api.StartAnswer
> >>>>>>>>
> >>>>>>>> Hi Nezar,
> >>>>>>>>
> >>>>>>>> You can find host name on detail view of SSVM in UI .
> >>>>>>>> You can also use list api command listSystemVms to get the host
> >>>>> details
> >>>>>>> of
> >>>>>>>> system vms.
> >>>>>>>>
> >>>>>>>> Thanks
> >>>>>>>> Shweta
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> ________________________________________
> >>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
> >>>>>>>> Sent: Wednesday, March 16, 2016 9:58 AM
> >>>>>>>> To: users@cloudstack.apache.org
> >>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable
> >> to
> >>>>> start
> >>>>>>>> instance due to Unable to get answer that is of class
> >>>>>>>> com.cloud.agent.api.StartAnswer
> >>>>>>>>
> >>>>>>>> Hi Simon,
> >>>>>>>>
> >>>>>>>> How can I find out which host the SSVM is on?
> >>>>>>>>
> >>>>>>>> Thanks & Regards,
> >>>>>>>> Nezar.
> >>>>>>>>
> >>>>>>>> On 16 March 2016 at 12:28, Simon Weller <sw...@ena.com>
> >> wrote:
> >>>>>>>>
> >>>>>>>>> Can you ssh into the Secondary Storage VM?
> >>>>>>>>>
> >>>>>>>>> This can be done by getting the link-local-ip address from the
> >>>>>>>>> infrastructure menu in the UI and then sshing into the host
> >> the
> >>>> SSVM
> >>>>>>> is
> >>>>>>>> on
> >>>>>>>>> and issuing this command as root: ssh -i
> >> /root/.ssh/id_rsa.cloud
> >>>> -p
> >>>>>>> 3922
> >>>>>>>>> root@link-local-ip (replace link-local-ip with the link
> >> local
> >>>> ip).
> >>>>>>>>>
> >>>>>>>>> Try running the SSVM diangnostic script:
> >>>>>>>>> usr/local/cloud/systemvm/ssvm-check.sh
> >>>>>>>>>
> >>>>>>>>> - Si
> >>>>>>>>> ________________________________________
> >>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
> >>>>>>>>> Sent: Tuesday, March 15, 2016 7:49 PM
> >>>>>>>>> To: users@cloudstack.apache.org
> >>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable
> >>> to
> >>>>>>> start
> >>>>>>>>> instance due to Unable to get answer that is of class
> >>>>>>>>> com.cloud.agent.api.StartAnswer
> >>>>>>>>>
> >>>>>>>>> Hi Simon,
> >>>>>>>>>
> >>>>>>>>> Please see details below:
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> mysql> use cloud; select id, name, status, type from host
> >> where
> >>>> id =
> >>>>>>> 5;
> >>>>>>>>> Reading table information for completion of table and column
> >>> names
> >>>>>>>>> You can turn off this feature to get a quicker startup with -A
> >>>>>>>>>
> >>>>>>>>> Database changed
> >>>>>>>>> +----+------------------------------+--------+---------+
> >>>>>>>>> | id | name | status | type |
> >>>>>>>>> +----+------------------------------+--------+---------+
> >>>>>>>>> | 5 | eqx-cs-cmp-02.ipscape.com.au | up | routing |
> >>>>>>>>> +----+------------------------------+--------+---------+
> >>>>>>>>> 1 row in set (0.00 sec)
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> cat /var/log/cloudstack/agent/agent.log
> >>>>>>>>> com.cloud.utils.exception.CloudRuntimeException:
> >>>>>>>>> com.cloud.utils.exception.CloudRuntimeException: Can't find
> >>>>>>>>> volume:089ff3ca-9bff-402c-a14e-ea7a35aaad5c
> >>>>>>>>> at
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>>
> >>>
> >>
> com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.getPhysicalDisk(KVMStoragePoolManager.java:262)
> >>>>>>>>> at
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>>
> >>>
> >>
> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.createVbd(LibvirtComputingResource.java:3921)
> >>>>>>>>> at
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>>
> >>>
> >>
> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(LibvirtComputingResource.java:3789)
> >>>>>>>>> at
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>>
> >>>
> >>
> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1333)
> >>>>>>>>> at
> >> com.cloud.agent.Agent.processRequest(Agent.java:501)
> >>>>>>>>> at
> >>>>>>>> com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:808)
> >>>>>>>>> at com.cloud.utils.nio.Task.run(Task.java:84)
> >>>>>>>>> at
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>>
> >>>
> >>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> >>>>>>>>> at
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>>
> >>>
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> >>>>>>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>>>>> agent.log.2016-03-03.gz (END)
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> [root@eqx-cs-cmp-02 agent]# virsh pool-list
> >>>>>>>>> Name State Autostart
> >>>>>>>>> -----------------------------------------
> >>>>>>>>> 2d0967f8-50d6-4d6d-87b7-4da0596f6924 active no
> >>>>>>>>> b3647480-5709-36eb-9ddd-e2c666f30b57 active no
> >>>>>>>>> e3831246-ff55-4af4-9964-a17565edaccc active no
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> [Root @ eqx-cs-cmp-02 agent] # df -h
> >>>>>>>>> Filesystem Size Used Avail Use% Mounted on
> >>>>>>>>> /dev/mapper/vg_root-lv_root
> >>>>>>>>> 203G 215G 1.3G 1% /
> >>>>>>>>> tmpfs 48g 48g 0 0% / dev / shm
> >>>>>>>>> / Dev / sda1 477m 427m 25M 6% / boot
> >>>>>>>>> xx.xx.xx.xx:/Cloudstack/primary
> >>>>>>>>> 7.6T 4.1T 3.5T 54%
> >>>>>>>>> /mnt/b3647480-5709-36eb-9ddd-e2c666f30b57
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> I also noticed the following in cloudstack-agent.err
> >>>>>>>>>
> >>>>>>>>> libvirt: QEMU Driver error : Domain not found: no domain with
> >>>>> matching
> >>>>>>>> name
> >>>>>>>>> 'i-11-600-VM'
> >>>>>>>>> libvirt: Storage Driver error : Storage volume not found: no
> >>>> storage
> >>>>>>> vol
> >>>>>>>>> with matching name '089ff3ca-9bff-402c-a14e-ea7a35aaad5c'
> >>>>>>>>>
> >>>>>>>>> Your help is much appreciated,
> >>>>>>>>> Nezar
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On 16 March 2016 at 11:03, Simon Weller <sw...@ena.com>
> >>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Nezar,
> >>>>>>>>>>
> >>>>>>>>>> ok, I just took a look at a pastebin you posted.
> >>>>>>>>>>
> >>>>>>>>>> I see it complaining about the inability to find a volume
> >>> prior
> >>>> to
> >>>>>>>>>> throwing the host exception.
> >>>>>>>>>>
> >>>>>>>>>> Firstly, let's find out who host 5 is. Can you log into your
> >>>> MySQL
> >>>>>>>> server
> >>>>>>>>>> and run this:
> >>>>>>>>>>
> >>>>>>>>>> use cloud;select id,name,status,type from host where id=5;
> >>>>>>>>>>
> >>>>>>>>>> Can you post the results?
> >>>>>>>>>>
> >>>>>>>>>> If it's a host, can you post the
> >>>>> /var/log/cloudstack/agent/agent.log
> >>>>>>>> from
> >>>>>>>>>> that host?
> >>>>>>>>>>
> >>>>>>>>>> And also run virsh pool-list on the host in question?
> >>>>>>>>>>
> >>>>>>>>>> - Si
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> ________________________________________
> >>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
> >>>>>>>>>> Sent: Tuesday, March 15, 2016 6:11 PM
> >>>>>>>>>> To: users@cloudstack.apache.org
> >>>>>>>>>> Subject: Re: Unable to create a New VM - Error message:
> >> Unable
> >>>> to
> >>>>>>> start
> >>>>>>>>>> instance due to Unable to get answer that is of class
> >>>>>>>>>> com.cloud.agent.api.StartAnswer
> >>>>>>>>>>
> >>>>>>>>>> Hi Simon,
> >>>>>>>>>>
> >>>>>>>>>> When you say "double check your routers tied to the
> >> network",
> >>> do
> >>>>> you
> >>>>>>>> mean
> >>>>>>>>>> the physical routers or the virtual routers?
> >>>>>>>>>>
> >>>>>>>>>> When I try to create a VM in Prod Environment it fails. I
> >>> can't
> >>>>> ping
> >>>>>>>> the
> >>>>>>>>>> Prod virtual router from management server or Host.
> >>>>>>>>>>
> >>>>>>>>>> But I can ping Dev virtual router from management server and
> >>>> Host
> >>>>>>> and I
> >>>>>>>>> was
> >>>>>>>>>> able to create a VM in Dev Environment.
> >>>>>>>>>>
> >>>>>>>>>> How can I troubleshoot the Prod Virtual router. I launched
> >> the
> >>>>>>> console
> >>>>>>>>> but
> >>>>>>>>>> couldn't login.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Please note that we need to take into consideration that the
> >>>>>>> affected
> >>>>>>>>>> network is the Production network and we need to minimise
> >>>> service
> >>>>>>>> outage
> >>>>>>>>> as
> >>>>>>>>>> much as possible.
> >>>>>>>>>>
> >>>>>>>>>> Your help is much appreciated,
> >>>>>>>>>> Nezar
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On 16 March 2016 at 08:37, Simon Weller <sw...@ena.com>
> >>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> I've seen this before where a Virtual Router has crashed
> >> and
> >>>> the
> >>>>>>>>>>> management server can't communicate with it.
> >>>>>>>>>>> Are you using an advanced network setup?
> >>>>>>>>>>>
> >>>>>>>>>>> If so, double check your routers tied to the network where
> >>> you
> >>>>> are
> >>>>>>>>>>> attempting to start an instance.
> >>>>>>>>>>>
> >>>>>>>>>>> - Si
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> ________________________________________
> >>>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
> >>>>>>>>>>> Sent: Tuesday, March 15, 2016 4:02 PM
> >>>>>>>>>>> To: users@cloudstack.apache.org
> >>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message:
> >>> Unable
> >>>>> to
> >>>>>>>> start
> >>>>>>>>>>> instance due to Unable to get answer that is of class
> >>>>>>>>>>> com.cloud.agent.api.StartAnswer
> >>>>>>>>>>>
> >>>>>>>>>>> Hi Glenn,
> >>>>>>>>>>>
> >>>>>>>>>>> We have 16 hosts, all KVM.
> >>>>>>>>>>>
> >>>>>>>>>>> The error is not specific to Host 5. Every time I try to
> >>>> create
> >>>>> a
> >>>>>>> VM
> >>>>>>>> it
> >>>>>>>>>>> gives me the same error message but on different host.
> >>>>>>>>>>>
> >>>>>>>>>>> How do I restart the agent on the host. Is it service
> >>>> affecting?
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks very much for your help.
> >>>>>>>>>>> Neza
> >>>>>>>>>>>
> >>>>>>>>>>> On 15 March 2016 at 17:37, Glenn Wagner <
> >>>>>>> glenn.wagner@shapeblue.com>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Hi,
> >>>>>>>>>>>>
> >>>>>>>>>>>> Can you explain your environment first , how many hosts
> >> ,
> >>>> what
> >>>>>>> is
> >>>>>>>>> your
> >>>>>>>>>>>> hypervisors (KVM,Xenserver,VMware)
> >>>>>>>>>>>>
> >>>>>>>>>>>> The error seems to be with your Host 5 Hypervisor
> >>>>>>>>>>>> Resource [Host:5] is unreachable:
> >>>>>>>>>>>>
> >>>>>>>>>>>> Go to the infrastructure section and see what host5 is
> >>>>>>> reporting,
> >>>>>>>>>>>> Have you tried to restart the agent on the Host?
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks
> >>>>>>>>>>>> Glenn
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> [image: ShapeBlue] <http://www.shapeblue.com>
> >>>>>>>>>>>> Glenn Wagner
> >>>>>>>>>>>> Senior Consultant , ShapeBlue
> >>>>>>>>>>>> d: * | s: +27 21 527 0091*
> >>>>>>> <%7C%20s:%20+27%2021%20527%200091> |
> >>>>>>>>> m:
> >>>>>>>>>>>> *+27 73 917 4111* <+27%2073%20917%204111>
> >>>>>>>>>>>> e: *glenn.wagner@shapeblue.com | t: *
> >>>>>>>>>>>> <glenn.wagner@shapeblue.com%20%7C%20t:> | w: *
> >>>>>>> www.shapeblue.com
> >>>>>>>> *
> >>>>>>>>>>>> <http://www.shapeblue.com>
> >>>>>>>>>>>> a:
> >>>>>>>>>>>> 2nd Floor, Oudehuis Centre, 122 Main Rd, Somerset West
> >>> Cape
> >>>>> Town
> >>>>>>>> 7130
> >>>>>>>>>>> South Africa
> >>>>>>>>>>>> Shape Blue Ltd is a company incorporated in England &
> >>> Wales.
> >>>>>>>>> ShapeBlue
> >>>>>>>>>>>> Services India LLP is a company incorporated in India
> >> and
> >>> is
> >>>>>>>> operated
> >>>>>>>>>>> under
> >>>>>>>>>>>> license from Shape Blue Ltd. Shape Blue Brasil
> >> Consultoria
> >>>>> Ltda
> >>>>>>> is
> >>>>>>>> a
> >>>>>>>>>>>> company incorporated in Brasil and is operated under
> >>> license
> >>>>>>> from
> >>>>>>>>> Shape
> >>>>>>>>>>>> Blue Ltd. ShapeBlue SA Pty Ltd is a company registered
> >> by
> >>>> The
> >>>>>>>>> Republic
> >>>>>>>>>> of
> >>>>>>>>>>>> South Africa and is traded under license from Shape Blue
> >>>> Ltd.
> >>>>>>>>> ShapeBlue
> >>>>>>>>>>> is
> >>>>>>>>>>>> a registered trademark.
> >>>>>>>>>>>> This email and any attachments to it may be confidential
> >>> and
> >>>>> are
> >>>>>>>>>> intended
> >>>>>>>>>>>> solely for the use of the individual to whom it is
> >>>> addressed.
> >>>>>>> Any
> >>>>>>>>> views
> >>>>>>>>>>> or
> >>>>>>>>>>>> opinions expressed are solely those of the author and do
> >>> not
> >>>>>>>>>> necessarily
> >>>>>>>>>>>> represent those of Shape Blue Ltd or related companies.
> >> If
> >>>> you
> >>>>>>> are
> >>>>>>>>> not
> >>>>>>>>>>> the
> >>>>>>>>>>>> intended recipient of this email, you must neither take
> >>> any
> >>>>>>> action
> >>>>>>>>>> based
> >>>>>>>>>>>> upon its contents, nor copy or show it to anyone. Please
> >>>>> contact
> >>>>>>>> the
> >>>>>>>>>>> sender
> >>>>>>>>>>>> if you believe you have received this email in error.
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> -----Original Message-----
> >>>>>>>>>>>> From: Nezar Madbouh [mailto:
> >> nezar.madbouh@ipscape.com.au]
> >>>>>>>>>>>> Sent: Tuesday, 15 March 2016 5:45 AM
> >>>>>>>>>>>> To: users@cloudstack.apache.org
> >>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message:
> >>>> Unable
> >>>>>>> to
> >>>>>>>>> start
> >>>>>>>>>>>> instance due to Unable to get answer that is of class
> >>>>>>>>>>>> com.cloud.agent.api.StartAnswer
> >>>>>>>>>>>>
> >>>>>>>>>>>> Hi Jeremy,
> >>>>>>>>>>>>
> >>>>>>>>>>>> I have uploaded the contents to
> >>>> http://pastebin.com/1J3mmgTh
> >>>>>>>>>>>>
> >>>>>>>>>>>> your help is much appreciated,
> >>>>>>>>>>>> Nezar.
> >>>>>>>>>>>>
> >>>>>>>>>>>> On 15 March 2016 at 13:46, Jeremy Peterson <
> >>>>>>> jpeterson@acentek.net>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> I don't see any attachments can you create a pastebin.
> >>>>> That's
> >>>>>>>>> easier
> >>>>>>>>>>>>> to look through as I think attachments are removed
> >>> before
> >>>>>>> being
> >>>>>>>>> sent
> >>>>>>>>>>>>> to user group.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Jeremy
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> -------- Original message --------
> >>>>>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
> >>>>>>>>>>>>> Date: 3/14/2016 6:38 PM (GMT-06:00)
> >>>>>>>>>>>>> To: users@cloudstack.apache.org
> >>>>>>>>>>>>> Subject: Unable to create a New VM - Error message:
> >>> Unable
> >>>>> to
> >>>>>>>> start
> >>>>>>>>>>>>> instance due to Unable to get answer that is of class
> >>>>>>>>>>>>> com.cloud.agent.api.StartAnswer
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Hi Shweta and Sadhu,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Please find attached managerment-server.log which
> >>> contains
> >>>>> the
> >>>>>>>> logs
> >>>>>>>>>>>>> since I started creating a new VM (VM id 600).
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Please note that we don't have agent.log files
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Your help is much appreciated.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Regards,
> >>>>>>>>>>>>> Nezar.
> >>>>>>>>>>>>>
> >>>>>>>>>>>> Find out more about ShapeBlue and our range of
> >> CloudStack
> >>>>>>> related
> >>>>>>>>>>> services:
> >>>>>>>>>>>> IaaS Cloud Design & Build
> >>>>>>>>>>>> <http://shapeblue.com/iaas-cloud-design-and-build//> |
> >>>>> CSForge
> >>>>>>> -
> >>>>>>>>> rapid
> >>>>>>>>>>>> IaaS deployment framework <
> >> http://shapeblue.com/csforge/>
> >>>>>>>>>>>> CloudStack Consulting <
> >>>>>>>> http://shapeblue.com/cloudstack-consultancy/>
> >>>>>>>>> |
> >>>>>>>>>>> CloudStack
> >>>>>>>>>>>> Software Engineering
> >>>>>>>>>>>> <http://shapeblue.com/cloudstack-software-engineering/>
> >>>>>>>>>>>> CloudStack Infrastructure Support
> >>>>>>>>>>>> <
> >> http://shapeblue.com/cloudstack-infrastructure-support/>
> >>> |
> >>>>>>>>> CloudStack
> >>>>>>>>>>>> Bootcamp Training Courses <
> >>>>>>>> http://shapeblue.com/cloudstack-training/
> >>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> DISCLAIMER
> >>>>>>>> ==========
> >>>>>>>> This e-mail may contain privileged and confidential information
> >>>> which
> >>>>> is
> >>>>>>>> the property of Accelerite, a Persistent Systems business. It is
> >>>>>>> intended
> >>>>>>>> only for the use of the individual or entity to which it is
> >>>> addressed.
> >>>>>>> If
> >>>>>>>> you are not the intended recipient, you are not authorized to
> >>> read,
> >>>>>>> retain,
> >>>>>>>> copy, print, distribute or use this message. If you have
> >> received
> >>>> this
> >>>>>>>> communication in error, please notify the sender and delete all
> >>>> copies
> >>>>>>> of
> >>>>>>>> this message. Accelerite, a Persistent Systems business does not
> >>>>> accept
> >>>>>>> any
> >>>>>>>> liability for virus infected mails.
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> >>
> >
>
Re: Unable to create a New VM - Error message: Unable to start instance
due to Unable to get answer that is of class com.cloud.agent.api.StartAnswer
Posted by ilya <il...@gmail.com>.
It appears you have pending jobs that are not letting you go through.
Usually stop and starting management service should be enough.
However, try running this script from your management server to show us
what you have running in async_job table.
> #!/bin/bash
> DATESTAMP=$(date +%m%d%y-%H%M%S)
> DBPASS=$(java -classpath /usr/share/cloudstack-common/lib/jasypt-1.9.0.jar org.jasypt.intf.cli.JasyptPBEStringDecryptionCLI input="$(cat /etc/cloudstack/management/db.properties | grep db.cloud.password | awk -F'(' '{print $2}' | sed 's/)//g')" password="$(cat /etc/cloudstack/management/key)" | grep -A2 OUTPUT | tail -1)
> DBHOST=$(cat /etc/cloudstack/management/db.properties | grep db.cloud.host | awk -F'=' '{print $2}' | tail -1 )
> DBUSER=$(cat /etc/cloudstack/management/db.properties | grep db.cloud.username | awk -F'=' '{print $2}')
> DB=$(cat /etc/cloudstack/management/db.properties | grep db.cloud.name | awk -F'=' '{print $2}')
> DBPORT=$(cat /etc/cloudstack/management/db.properties | grep db.cloud.port | awk -F'=' '{print $2}')
> MYSQLCMD="mysql -h $DBHOST -u $DBUSER -P $DBPORT -p$DBPASS $DB"
> echo 'SELECT id, user_id, account_id, instance_type, instance_id, job_cmd, job_cmd_info, job_dispatcher, job_pending_signals FROM cloud.async_job where job_status=0 and job_dispatcher not like "pseudoJobDispatcher"' | $MYSQLCMD
On 4/13/16 5:16 PM, Nezar Madbouh wrote:
> Hi Simon,
>
> The following work was done but unfortunately the problem is not fixed:
>
>
> on management server:
>
> service cloudstack-management stop
>
>
> on the host:
>
> service cloudstack-agent stop
>
> service libvirtd restart
>
> service cloudstack-agent start
>
>
> on management server:
>
> service cloudstack-management start
>
> Regards,
> Nezar.
>
> On 24 March 2016 at 05:06, Simon Weller <sw...@ena.com> wrote:
>
>> Yes, I think that order is fine. Make sure the processes are indeed
>> shutdown before moving on to the next step.
>>
>>
>>
>>
>>
>> ________________________________________
>> From: Nezar Madbouh <ne...@ipscape.com.au>
>> Sent: Tuesday, March 22, 2016 5:33 PM
>> To: users@cloudstack.apache.org
>> Subject: Re: Unable to create a New VM - Error message: Unable to start
>> instance due to Unable to get answer that is of class
>> com.cloud.agent.api.StartAnswer
>>
>> Hi Simon,
>>
>> Can you please check if the following sequence of action is correct:
>>
>> on management server:
>>
>> service cloudstack-management stop
>>
>>
>> on the host:
>>
>> service cloudstack-agent stop
>>
>> service libvirtd restart
>>
>> service cloudstack-agent start
>>
>>
>> on management server:
>>
>> service cloudstack-management start
>>
>> Thanks very much for your help.
>> Nezar.
>>
>> On 23 March 2016 at 08:39, Simon Weller <sw...@ena.com> wrote:
>>
>>> Shutting down all the management services should clear the jobs.
>>>
>>>
>>> ________________________________________
>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>>> Sent: Tuesday, March 22, 2016 4:28 PM
>>> To: users@cloudstack.apache.org
>>> Subject: Re: Unable to create a New VM - Error message: Unable to start
>>> instance due to Unable to get answer that is of class
>>> com.cloud.agent.api.StartAnswer
>>>
>>> Hi Simon,
>>>
>>> I will plan to restart agents, libvirtd and management services.
>>>
>>> How do I clear hung jobs.
>>>
>>> Your help is much appreciated,
>>> Nezar.
>>>
>>> On 22 March 2016 at 23:03, Simon Weller <sw...@ena.com> wrote:
>>>
>>>> Nezar,
>>>>
>>>> I've looked through the code related to your release and it seems that
>> in
>>>> order to throw this exception, 10 attempts have been made to retrieve
>> the
>>>> disk information prior to the exception being thrown.
>>>>
>>>> It appears there is something really odd going on here. I'm starting
>> to
>>>> wonder whether it might be worth while for you to stop all of your
>>> agents,
>>>> plus management server processes, restart libvirtd on each of the hosts
>>> and
>>>> then bring the agents back up prior to starting the management servers.
>>>>
>>>> There are a bunch of logs the pastebin you posted that seems to
>> indicate
>>>> you have some jobs hung, for example:
>>>>
>>>> 2016-03-14 21:45:31,828 WARN [o.a.c.f.j.i.AsyncJobMonitor]
>>>> (Timer-1:ctx-0c090bd7) Task (job-9066) has been pending for 1660851
>>> seconds
>>>> 2016-03-14 21:45:31,828 WARN [o.a.c.f.j.i.AsyncJobMonitor]
>>>> (Timer-1:ctx-0c090bd7) Task (job-8133) has been pending for 4253003
>>> seconds
>>>>
>>>> Restarting the agents, libvirtd and management will not cause an
>> outage
>>>> to your cloud, but I would suggest, if you decided to do this, you
>> should
>>>> complete it in a maintenance window.
>>>>
>>>> - Si
>>>>
>>>> ________________________________________
>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>>>> Sent: Sunday, March 20, 2016 10:27 PM
>>>> To: users@cloudstack.apache.org
>>>> Subject: Re: Unable to create a New VM - Error message: Unable to start
>>>> instance due to Unable to get answer that is of class
>>>> com.cloud.agent.api.StartAnswer
>>>>
>>>> Hi Simon,
>>>>
>>>> Please see answers to your questions below. I am not sure if I provided
>>> the
>>>> correct answer for the primary and secondary storage configuration.
>>> Please
>>>> let me know if this is not what you are after:
>>>>
>>>>
>>>> Can you screenshot and post somewhere your secondary storage
>>> configuration
>>>> and your primary storage configuration from the UI with the details?
>>>>
>>>> mysql> select * from configuration where name like '%secondary%'\G
>>>> *************************** 1. row ***************************
>>>> category: Account Defaults
>>>> instance: DEFAULT
>>>> component: management-server
>>>> name: max.account.secondary.storage
>>>> value: 400
>>>> description: The default maximum secondary storage space (in GiB)
>> that
>>>> can be used for an account
>>>> default_value: 400
>>>> updated: NULL
>>>> scope: NULL
>>>> is_dynamic: 0
>>>> *************************** 2. row ***************************
>>>> category: Project Defaults
>>>> instance: DEFAULT
>>>> component: management-server
>>>> name: max.project.secondary.storage
>>>> value: 400
>>>> description: The default maximum secondary storage space (in GiB)
>> that
>>>> can be used for a project
>>>> default_value: 400
>>>> updated: NULL
>>>> scope: NULL
>>>> is_dynamic: 0
>>>> *************************** 3. row ***************************
>>>> category: Hidden
>>>> instance: DEFAULT
>>>> component: management-server
>>>> name: secondary.storage.vm
>>>> value: +rwRhJOXPymbQOuAMLnhiw==
>>>> description: Deploys a VM per zone to manage secondary storage if
>> true,
>>>> otherwise secondary storage is mounted on management server
>>>> default_value: NULL
>>>> updated: NULL
>>>> scope: NULL
>>>> is_dynamic: 0
>>>> *************************** 4. row ***************************
>>>> category: Advanced
>>>> instance: DEFAULT
>>>> component: ExtensionRegistry
>>>> name: secondary.storage.vm.allocators.exclude
>>>> value: NULL
>>>> description: Extensions to exclude from being registered
>>>> default_value: NULL
>>>> updated: 2014-12-04 00:18:15
>>>> scope: Global
>>>> is_dynamic: 0
>>>> *************************** 5. row ***************************
>>>> category: NetworkManager
>>>> instance: DEFAULT
>>>> component: management-server
>>>> name: vm.network.nic.max.secondary.ipaddresses
>>>> value: NULL
>>>> description: Specify the number of secondary ip addresses per nic per
>>> vm
>>>> default_value: 256
>>>> updated: NULL
>>>> scope: NULL
>>>> is_dynamic: 0
>>>> 5 rows in set (0.00 sec)
>>>>
>>>>
>>>> mysql> select * from configuration where name like '%primary%'\G
>>>> *************************** 1. row ***************************
>>>> category: Account Defaults
>>>> instance: DEFAULT
>>>> component: management-server
>>>> name: max.account.primary.storage
>>>> value: 200
>>>> description: The default maximum primary storage space (in GiB) that
>>> can
>>>> be used for an account
>>>> default_value: 200
>>>> updated: NULL
>>>> scope: NULL
>>>> is_dynamic: 0
>>>> *************************** 2. row ***************************
>>>> category: Project Defaults
>>>> instance: DEFAULT
>>>> component: management-server
>>>> name: max.project.primary.storage
>>>> value: 200
>>>> description: The default maximum primary storage space (in GiB) that
>>> can
>>>> be used for a project
>>>> default_value: 200
>>>> updated: NULL
>>>> scope: NULL
>>>> is_dynamic: 0
>>>> *************************** 3. row ***************************
>>>> category: Storage
>>>> instance: DEFAULT
>>>> component: TemplateManager
>>>> name: primary.storage.download.wait
>>>> value: 10800
>>>> description: In second, timeout for download template to primary
>>> storage
>>>> default_value: NULL
>>>> updated: NULL
>>>> scope: NULL
>>>> is_dynamic: 0
>>>> 3 rows in set (0.00 sec)
>>>>
>>>>
>>>>
>>>> Also, what is the Cloudstack version you are running and what is the OS
>>>> you're running it on?
>>>>
>>>> Cloudstack 4.4.1
>>>>
>>>> Centos 6.6
>>>>
>>>> Are you using actimeo=0 in your NFS mount line?
>>>>
>>>> Primary storage on Hypervisors has the following parameters
>>>> 10.61.154.10:/Cloudstack/primary on
>>>> /mnt/b3647480-5709-36eb-9ddd-e2c666f30b57 type nfs
>>>> (rw,noatime,nfsvers=3,proto=tcp,hard,nolock,intr,addr=10.61.154.10)
>>>>
>>>>
>>>> Secondary storage on SSVM has the following parameters
>>>> 10.61.154.10:/Cloudstack/secondary on
>>>> /mnt/SecStorage/d11d5c92-6950-30e8-a0b5-3cd46a17ec14 type nfs4
>>>>
>>>>
>>>
>> (rw,relatime,vers=4,rsize=262144,wsize=262144,namlen=255,acdirmin=0,acdirmax=0,soft,proto=tcp,port=0,timeo=133,retrans=2147483647,sec=sys,clientaddr=10.61.154.70,minorversion=0,local_lock=none,addr=10.61.154.10)
>>>>
>>>>
>>>> Thanks & Regards,
>>>> Nezar.
>>>>
>>>> On 18 March 2016 at 23:50, Simon Weller <sw...@ena.com> wrote:
>>>>
>>>>> Also, what is the Cloudstack version you are running and what is the
>> OS
>>>>> you're running it on?
>>>>>
>>>>> Are you using actimeo=0 in your NFS mount line?
>>>>>
>>>>> ________________________________________
>>>>> From: Simon Weller <sw...@ena.com>
>>>>> Sent: Friday, March 18, 2016 7:40 AM
>>>>> To: users@cloudstack.apache.org
>>>>> Subject: Re: Unable to create a New VM - Error message: Unable to
>> start
>>>>> instance due to Unable to get answer that is of class
>>>>> com.cloud.agent.api.StartAnswer
>>>>>
>>>>> Nexar,
>>>>>
>>>>> That's normal behaviour. You can only ssh to the link local on the
>> host
>>>>> where the VM resides.
>>>>>
>>>>> Can you screenshot and post somewhere your secondary storage
>>>> configuration
>>>>> and your primary storage configuration from the UI with the details?
>>>>>
>>>>> - Si
>>>>>
>>>>> ________________________________________
>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>>>>> Sent: Thursday, March 17, 2016 8:19 PM
>>>>> To: users@cloudstack.apache.org
>>>>> Subject: Re: Unable to create a New VM - Error message: Unable to
>> start
>>>>> instance due to Unable to get answer that is of class
>>>>> com.cloud.agent.api.StartAnswer
>>>>>
>>>>> Hi Simon,
>>>>>
>>>>> I just noticed that from other Hosts I can't ssh into the SSVM:
>>>>>
>>>>> [root@eqx-cs-cmp-01 ~]# ssh -i /root/.ssh/id_rsa.cloud -p 3922
>>>>> root@169.254.3.161
>>>>> ssh: connect to host 169.254.3.161 port 3922: No route to host
>>>>>
>>>>>
>>>>> I can only ssh from one Host:
>>>>> [root@eqx-cs-cmp-06 ~]# ssh -i /root/.ssh/id_rsa.cloud -p 3922
>>>>> root@169.254.3.161
>>>>> Linux s-11-VM 3.2.0-4-amd64 #1 SMP Debian 3.2.57-3 x86_64
>>>>>
>>>>> Thanks & Regards,
>>>>> Nezar.
>>>>>
>>>>> On 18 March 2016 at 10:36, Nezar Madbouh <
>> nezar.madbouh@ipscape.com.au
>>>>
>>>>> wrote:
>>>>>
>>>>>> Hi Simon,
>>>>>>
>>>>>> Yes existing VMs are successfully running. Yesterday I was able to
>>>> create
>>>>>> a VM in Dev Environment but I can't create VMs in Prod environment.
>>>>>>
>>>>>> Thanks,
>>>>>> Nezar.
>>>>>>
>>>>>> On 18 March 2016 at 09:12, Simon Weller <sw...@ena.com> wrote:
>>>>>>
>>>>>>> I'm assuming you have other VMs successfully running, correct? So
>>> you
>>>>> can
>>>>>>> write to your primary storage.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ________________________________________
>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>>>>>>> Sent: Thursday, March 17, 2016 4:43 PM
>>>>>>> To: users@cloudstack.apache.org
>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable to
>>>> start
>>>>>>> instance due to Unable to get answer that is of class
>>>>>>> com.cloud.agent.api.StartAnswer
>>>>>>>
>>>>>>> Hi Simon and Shweta,
>>>>>>>
>>>>>>> Please see the result below:
>>>>>>>
>>>>>>> root@s-11-VM:~# /usr/local/cloud/systemvm/ssvm-check.sh
>>>>>>> ================================================
>>>>>>> First DNS server is 8.8.8.8
>>>>>>> PING 8.8.8.8 (8.8.8.8): 48 data bytes
>>>>>>> 56 bytes from 8.8.8.8: icmp_seq=0 ttl=57 time=5.209 ms
>>>>>>> 56 bytes from 8.8.8.8: icmp_seq=1 ttl=57 time=0.936 ms
>>>>>>> --- 8.8.8.8 ping statistics ---
>>>>>>> 2 packets transmitted, 2 packets received, 0% packet loss
>>>>>>> round-trip min/avg/max/stddev = 0.936/3.072/5.209/2.137 ms
>>>>>>> Good: Can ping DNS server
>>>>>>> ================================================
>>>>>>> Good: DNS resolves download.cloud.com
>>>>>>> ================================================
>>>>>>> nfs is currently mounted
>>>>>>> Mount point is
>> /mnt/SecStorage/d11d5c92-6950-30e8-a0b5-3cd46a17ec14
>>>>>>> Good: Can write to mount point
>>>>>>> ================================================
>>>>>>> Management server is 10.61.153.20. Checking connectivity.
>>>>>>> Good: Can connect to management server port 8250
>>>>>>> ================================================
>>>>>>> Good: Java process is running
>>>>>>> ================================================
>>>>>>> Tests Complete. Look for ERROR or WARNING above.
>>>>>>> root@s-11-VM:~#
>>>>>>>
>>>>>>> Thanks & Regards,
>>>>>>> Nezar.
>>>>>>>
>>>>>>> On 17 March 2016 at 09:42, Simon Weller <sw...@ena.com> wrote:
>>>>>>>
>>>>>>>> Nezar,
>>>>>>>>
>>>>>>>> Log into the Cloudstack UI.
>>>>>>>>
>>>>>>>> Click on "Infrastructure" on the left menu. Click on View All
>>> below
>>>>>>>> "System VMs". find the item marked "Secondary Storage VM" and
>>> then
>>>>>>> click
>>>>>>>> on the Name of that VM on the left (it will be s-XXXX-VM).
>> You'll
>>>> see
>>>>> a
>>>>>>>> bunch of information about the VM. Find the Host line and also
>> the
>>>>> Link
>>>>>>>> Local IP Address line and then follow the original instructions
>> in
>>>> my
>>>>>>>> previous email.
>>>>>>>>
>>>>>>>> - Si
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ________________________________________
>>>>>>>> From: Shweta Agarwal <sh...@accelerite.com>
>>>>>>>> Sent: Wednesday, March 16, 2016 12:16 AM
>>>>>>>> To: users@cloudstack.apache.org
>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable
>> to
>>>>> start
>>>>>>>> instance due to Unable to get answer that is of class
>>>>>>>> com.cloud.agent.api.StartAnswer
>>>>>>>>
>>>>>>>> Hi Nezar,
>>>>>>>>
>>>>>>>> You can find host name on detail view of SSVM in UI .
>>>>>>>> You can also use list api command listSystemVms to get the host
>>>>> details
>>>>>>> of
>>>>>>>> system vms.
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>> Shweta
>>>>>>>>
>>>>>>>>
>>>>>>>> ________________________________________
>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>>>>>>>> Sent: Wednesday, March 16, 2016 9:58 AM
>>>>>>>> To: users@cloudstack.apache.org
>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable
>> to
>>>>> start
>>>>>>>> instance due to Unable to get answer that is of class
>>>>>>>> com.cloud.agent.api.StartAnswer
>>>>>>>>
>>>>>>>> Hi Simon,
>>>>>>>>
>>>>>>>> How can I find out which host the SSVM is on?
>>>>>>>>
>>>>>>>> Thanks & Regards,
>>>>>>>> Nezar.
>>>>>>>>
>>>>>>>> On 16 March 2016 at 12:28, Simon Weller <sw...@ena.com>
>> wrote:
>>>>>>>>
>>>>>>>>> Can you ssh into the Secondary Storage VM?
>>>>>>>>>
>>>>>>>>> This can be done by getting the link-local-ip address from the
>>>>>>>>> infrastructure menu in the UI and then sshing into the host
>> the
>>>> SSVM
>>>>>>> is
>>>>>>>> on
>>>>>>>>> and issuing this command as root: ssh -i
>> /root/.ssh/id_rsa.cloud
>>>> -p
>>>>>>> 3922
>>>>>>>>> root@link-local-ip (replace link-local-ip with the link
>> local
>>>> ip).
>>>>>>>>>
>>>>>>>>> Try running the SSVM diangnostic script:
>>>>>>>>> usr/local/cloud/systemvm/ssvm-check.sh
>>>>>>>>>
>>>>>>>>> - Si
>>>>>>>>> ________________________________________
>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>>>>>>>>> Sent: Tuesday, March 15, 2016 7:49 PM
>>>>>>>>> To: users@cloudstack.apache.org
>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable
>>> to
>>>>>>> start
>>>>>>>>> instance due to Unable to get answer that is of class
>>>>>>>>> com.cloud.agent.api.StartAnswer
>>>>>>>>>
>>>>>>>>> Hi Simon,
>>>>>>>>>
>>>>>>>>> Please see details below:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> mysql> use cloud; select id, name, status, type from host
>> where
>>>> id =
>>>>>>> 5;
>>>>>>>>> Reading table information for completion of table and column
>>> names
>>>>>>>>> You can turn off this feature to get a quicker startup with -A
>>>>>>>>>
>>>>>>>>> Database changed
>>>>>>>>> +----+------------------------------+--------+---------+
>>>>>>>>> | id | name | status | type |
>>>>>>>>> +----+------------------------------+--------+---------+
>>>>>>>>> | 5 | eqx-cs-cmp-02.ipscape.com.au | up | routing |
>>>>>>>>> +----+------------------------------+--------+---------+
>>>>>>>>> 1 row in set (0.00 sec)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> cat /var/log/cloudstack/agent/agent.log
>>>>>>>>> com.cloud.utils.exception.CloudRuntimeException:
>>>>>>>>> com.cloud.utils.exception.CloudRuntimeException: Can't find
>>>>>>>>> volume:089ff3ca-9bff-402c-a14e-ea7a35aaad5c
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>>
>>>
>> com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.getPhysicalDisk(KVMStoragePoolManager.java:262)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>>
>>>
>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.createVbd(LibvirtComputingResource.java:3921)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>>
>>>
>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(LibvirtComputingResource.java:3789)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>>
>>>
>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1333)
>>>>>>>>> at
>> com.cloud.agent.Agent.processRequest(Agent.java:501)
>>>>>>>>> at
>>>>>>>> com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:808)
>>>>>>>>> at com.cloud.utils.nio.Task.run(Task.java:84)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>>
>>>
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>>
>>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>>>> agent.log.2016-03-03.gz (END)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> [root@eqx-cs-cmp-02 agent]# virsh pool-list
>>>>>>>>> Name State Autostart
>>>>>>>>> -----------------------------------------
>>>>>>>>> 2d0967f8-50d6-4d6d-87b7-4da0596f6924 active no
>>>>>>>>> b3647480-5709-36eb-9ddd-e2c666f30b57 active no
>>>>>>>>> e3831246-ff55-4af4-9964-a17565edaccc active no
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> [Root @ eqx-cs-cmp-02 agent] # df -h
>>>>>>>>> Filesystem Size Used Avail Use% Mounted on
>>>>>>>>> /dev/mapper/vg_root-lv_root
>>>>>>>>> 203G 215G 1.3G 1% /
>>>>>>>>> tmpfs 48g 48g 0 0% / dev / shm
>>>>>>>>> / Dev / sda1 477m 427m 25M 6% / boot
>>>>>>>>> xx.xx.xx.xx:/Cloudstack/primary
>>>>>>>>> 7.6T 4.1T 3.5T 54%
>>>>>>>>> /mnt/b3647480-5709-36eb-9ddd-e2c666f30b57
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I also noticed the following in cloudstack-agent.err
>>>>>>>>>
>>>>>>>>> libvirt: QEMU Driver error : Domain not found: no domain with
>>>>> matching
>>>>>>>> name
>>>>>>>>> 'i-11-600-VM'
>>>>>>>>> libvirt: Storage Driver error : Storage volume not found: no
>>>> storage
>>>>>>> vol
>>>>>>>>> with matching name '089ff3ca-9bff-402c-a14e-ea7a35aaad5c'
>>>>>>>>>
>>>>>>>>> Your help is much appreciated,
>>>>>>>>> Nezar
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 16 March 2016 at 11:03, Simon Weller <sw...@ena.com>
>>> wrote:
>>>>>>>>>
>>>>>>>>>> Nezar,
>>>>>>>>>>
>>>>>>>>>> ok, I just took a look at a pastebin you posted.
>>>>>>>>>>
>>>>>>>>>> I see it complaining about the inability to find a volume
>>> prior
>>>> to
>>>>>>>>>> throwing the host exception.
>>>>>>>>>>
>>>>>>>>>> Firstly, let's find out who host 5 is. Can you log into your
>>>> MySQL
>>>>>>>> server
>>>>>>>>>> and run this:
>>>>>>>>>>
>>>>>>>>>> use cloud;select id,name,status,type from host where id=5;
>>>>>>>>>>
>>>>>>>>>> Can you post the results?
>>>>>>>>>>
>>>>>>>>>> If it's a host, can you post the
>>>>> /var/log/cloudstack/agent/agent.log
>>>>>>>> from
>>>>>>>>>> that host?
>>>>>>>>>>
>>>>>>>>>> And also run virsh pool-list on the host in question?
>>>>>>>>>>
>>>>>>>>>> - Si
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ________________________________________
>>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>>>>>>>>>> Sent: Tuesday, March 15, 2016 6:11 PM
>>>>>>>>>> To: users@cloudstack.apache.org
>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message:
>> Unable
>>>> to
>>>>>>> start
>>>>>>>>>> instance due to Unable to get answer that is of class
>>>>>>>>>> com.cloud.agent.api.StartAnswer
>>>>>>>>>>
>>>>>>>>>> Hi Simon,
>>>>>>>>>>
>>>>>>>>>> When you say "double check your routers tied to the
>> network",
>>> do
>>>>> you
>>>>>>>> mean
>>>>>>>>>> the physical routers or the virtual routers?
>>>>>>>>>>
>>>>>>>>>> When I try to create a VM in Prod Environment it fails. I
>>> can't
>>>>> ping
>>>>>>>> the
>>>>>>>>>> Prod virtual router from management server or Host.
>>>>>>>>>>
>>>>>>>>>> But I can ping Dev virtual router from management server and
>>>> Host
>>>>>>> and I
>>>>>>>>> was
>>>>>>>>>> able to create a VM in Dev Environment.
>>>>>>>>>>
>>>>>>>>>> How can I troubleshoot the Prod Virtual router. I launched
>> the
>>>>>>> console
>>>>>>>>> but
>>>>>>>>>> couldn't login.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Please note that we need to take into consideration that the
>>>>>>> affected
>>>>>>>>>> network is the Production network and we need to minimise
>>>> service
>>>>>>>> outage
>>>>>>>>> as
>>>>>>>>>> much as possible.
>>>>>>>>>>
>>>>>>>>>> Your help is much appreciated,
>>>>>>>>>> Nezar
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 16 March 2016 at 08:37, Simon Weller <sw...@ena.com>
>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> I've seen this before where a Virtual Router has crashed
>> and
>>>> the
>>>>>>>>>>> management server can't communicate with it.
>>>>>>>>>>> Are you using an advanced network setup?
>>>>>>>>>>>
>>>>>>>>>>> If so, double check your routers tied to the network where
>>> you
>>>>> are
>>>>>>>>>>> attempting to start an instance.
>>>>>>>>>>>
>>>>>>>>>>> - Si
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> ________________________________________
>>>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>>>>>>>>>>> Sent: Tuesday, March 15, 2016 4:02 PM
>>>>>>>>>>> To: users@cloudstack.apache.org
>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message:
>>> Unable
>>>>> to
>>>>>>>> start
>>>>>>>>>>> instance due to Unable to get answer that is of class
>>>>>>>>>>> com.cloud.agent.api.StartAnswer
>>>>>>>>>>>
>>>>>>>>>>> Hi Glenn,
>>>>>>>>>>>
>>>>>>>>>>> We have 16 hosts, all KVM.
>>>>>>>>>>>
>>>>>>>>>>> The error is not specific to Host 5. Every time I try to
>>>> create
>>>>> a
>>>>>>> VM
>>>>>>>> it
>>>>>>>>>>> gives me the same error message but on different host.
>>>>>>>>>>>
>>>>>>>>>>> How do I restart the agent on the host. Is it service
>>>> affecting?
>>>>>>>>>>>
>>>>>>>>>>> Thanks very much for your help.
>>>>>>>>>>> Neza
>>>>>>>>>>>
>>>>>>>>>>> On 15 March 2016 at 17:37, Glenn Wagner <
>>>>>>> glenn.wagner@shapeblue.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> Can you explain your environment first , how many hosts
>> ,
>>>> what
>>>>>>> is
>>>>>>>>> your
>>>>>>>>>>>> hypervisors (KVM,Xenserver,VMware)
>>>>>>>>>>>>
>>>>>>>>>>>> The error seems to be with your Host 5 Hypervisor
>>>>>>>>>>>> Resource [Host:5] is unreachable:
>>>>>>>>>>>>
>>>>>>>>>>>> Go to the infrastructure section and see what host5 is
>>>>>>> reporting,
>>>>>>>>>>>> Have you tried to restart the agent on the Host?
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks
>>>>>>>>>>>> Glenn
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> [image: ShapeBlue] <http://www.shapeblue.com>
>>>>>>>>>>>> Glenn Wagner
>>>>>>>>>>>> Senior Consultant , ShapeBlue
>>>>>>>>>>>> d: * | s: +27 21 527 0091*
>>>>>>> <%7C%20s:%20+27%2021%20527%200091> |
>>>>>>>>> m:
>>>>>>>>>>>> *+27 73 917 4111* <+27%2073%20917%204111>
>>>>>>>>>>>> e: *glenn.wagner@shapeblue.com | t: *
>>>>>>>>>>>> <glenn.wagner@shapeblue.com%20%7C%20t:> | w: *
>>>>>>> www.shapeblue.com
>>>>>>>> *
>>>>>>>>>>>> <http://www.shapeblue.com>
>>>>>>>>>>>> a:
>>>>>>>>>>>> 2nd Floor, Oudehuis Centre, 122 Main Rd, Somerset West
>>> Cape
>>>>> Town
>>>>>>>> 7130
>>>>>>>>>>> South Africa
>>>>>>>>>>>> Shape Blue Ltd is a company incorporated in England &
>>> Wales.
>>>>>>>>> ShapeBlue
>>>>>>>>>>>> Services India LLP is a company incorporated in India
>> and
>>> is
>>>>>>>> operated
>>>>>>>>>>> under
>>>>>>>>>>>> license from Shape Blue Ltd. Shape Blue Brasil
>> Consultoria
>>>>> Ltda
>>>>>>> is
>>>>>>>> a
>>>>>>>>>>>> company incorporated in Brasil and is operated under
>>> license
>>>>>>> from
>>>>>>>>> Shape
>>>>>>>>>>>> Blue Ltd. ShapeBlue SA Pty Ltd is a company registered
>> by
>>>> The
>>>>>>>>> Republic
>>>>>>>>>> of
>>>>>>>>>>>> South Africa and is traded under license from Shape Blue
>>>> Ltd.
>>>>>>>>> ShapeBlue
>>>>>>>>>>> is
>>>>>>>>>>>> a registered trademark.
>>>>>>>>>>>> This email and any attachments to it may be confidential
>>> and
>>>>> are
>>>>>>>>>> intended
>>>>>>>>>>>> solely for the use of the individual to whom it is
>>>> addressed.
>>>>>>> Any
>>>>>>>>> views
>>>>>>>>>>> or
>>>>>>>>>>>> opinions expressed are solely those of the author and do
>>> not
>>>>>>>>>> necessarily
>>>>>>>>>>>> represent those of Shape Blue Ltd or related companies.
>> If
>>>> you
>>>>>>> are
>>>>>>>>> not
>>>>>>>>>>> the
>>>>>>>>>>>> intended recipient of this email, you must neither take
>>> any
>>>>>>> action
>>>>>>>>>> based
>>>>>>>>>>>> upon its contents, nor copy or show it to anyone. Please
>>>>> contact
>>>>>>>> the
>>>>>>>>>>> sender
>>>>>>>>>>>> if you believe you have received this email in error.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>>> From: Nezar Madbouh [mailto:
>> nezar.madbouh@ipscape.com.au]
>>>>>>>>>>>> Sent: Tuesday, 15 March 2016 5:45 AM
>>>>>>>>>>>> To: users@cloudstack.apache.org
>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message:
>>>> Unable
>>>>>>> to
>>>>>>>>> start
>>>>>>>>>>>> instance due to Unable to get answer that is of class
>>>>>>>>>>>> com.cloud.agent.api.StartAnswer
>>>>>>>>>>>>
>>>>>>>>>>>> Hi Jeremy,
>>>>>>>>>>>>
>>>>>>>>>>>> I have uploaded the contents to
>>>> http://pastebin.com/1J3mmgTh
>>>>>>>>>>>>
>>>>>>>>>>>> your help is much appreciated,
>>>>>>>>>>>> Nezar.
>>>>>>>>>>>>
>>>>>>>>>>>> On 15 March 2016 at 13:46, Jeremy Peterson <
>>>>>>> jpeterson@acentek.net>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> I don't see any attachments can you create a pastebin.
>>>>> That's
>>>>>>>>> easier
>>>>>>>>>>>>> to look through as I think attachments are removed
>>> before
>>>>>>> being
>>>>>>>>> sent
>>>>>>>>>>>>> to user group.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Jeremy
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> -------- Original message --------
>>>>>>>>>>>>> From: Nezar Madbouh <ne...@ipscape.com.au>
>>>>>>>>>>>>> Date: 3/14/2016 6:38 PM (GMT-06:00)
>>>>>>>>>>>>> To: users@cloudstack.apache.org
>>>>>>>>>>>>> Subject: Unable to create a New VM - Error message:
>>> Unable
>>>>> to
>>>>>>>> start
>>>>>>>>>>>>> instance due to Unable to get answer that is of class
>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Shweta and Sadhu,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Please find attached managerment-server.log which
>>> contains
>>>>> the
>>>>>>>> logs
>>>>>>>>>>>>> since I started creating a new VM (VM id 600).
>>>>>>>>>>>>>
>>>>>>>>>>>>> Please note that we don't have agent.log files
>>>>>>>>>>>>>
>>>>>>>>>>>>> Your help is much appreciated.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>> Nezar.
>>>>>>>>>>>>>
>>>>>>>>>>>> Find out more about ShapeBlue and our range of
>> CloudStack
>>>>>>> related
>>>>>>>>>>> services:
>>>>>>>>>>>> IaaS Cloud Design & Build
>>>>>>>>>>>> <http://shapeblue.com/iaas-cloud-design-and-build//> |
>>>>> CSForge
>>>>>>> -
>>>>>>>>> rapid
>>>>>>>>>>>> IaaS deployment framework <
>> http://shapeblue.com/csforge/>
>>>>>>>>>>>> CloudStack Consulting <
>>>>>>>> http://shapeblue.com/cloudstack-consultancy/>
>>>>>>>>> |
>>>>>>>>>>> CloudStack
>>>>>>>>>>>> Software Engineering
>>>>>>>>>>>> <http://shapeblue.com/cloudstack-software-engineering/>
>>>>>>>>>>>> CloudStack Infrastructure Support
>>>>>>>>>>>> <
>> http://shapeblue.com/cloudstack-infrastructure-support/>
>>> |
>>>>>>>>> CloudStack
>>>>>>>>>>>> Bootcamp Training Courses <
>>>>>>>> http://shapeblue.com/cloudstack-training/
>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> DISCLAIMER
>>>>>>>> ==========
>>>>>>>> This e-mail may contain privileged and confidential information
>>>> which
>>>>> is
>>>>>>>> the property of Accelerite, a Persistent Systems business. It is
>>>>>>> intended
>>>>>>>> only for the use of the individual or entity to which it is
>>>> addressed.
>>>>>>> If
>>>>>>>> you are not the intended recipient, you are not authorized to
>>> read,
>>>>>>> retain,
>>>>>>>> copy, print, distribute or use this message. If you have
>> received
>>>> this
>>>>>>>> communication in error, please notify the sender and delete all
>>>> copies
>>>>>>> of
>>>>>>>> this message. Accelerite, a Persistent Systems business does not
>>>>> accept
>>>>>>> any
>>>>>>>> liability for virus infected mails.
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>>
>