You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Kent Johnson <ke...@utah.edu> on 2013/08/24 00:53:48 UTC
Cs 4.1.0 plus Xen 6.1 No Instances or VR's starting up
Can anyone point me in the right direction on solving my instance startup problem? They won't start up, nor will my virtual router.
I am using CloudStack 4.1.0 on CentOS for my Management Server and XenServer 6.1 for my VM host.
I installed CS as per the documentation and have successfully added a zone, pod, cluster, host, primary, and secondary storage.
My System VM's start up and run correctly. My systemVM template and the default CentOS templates download correctly.
I can create instances but they always try to start up and then fail and end up in "Error" state. My Virtual Router tries to start up
But it always fails.
Are there any gotchas or quick suggestions anyone could give me to help me understand how to get my instances to run and possibly my virtual router?
I can provide more details from the logs if needed.
Kent Johnson
University of Utah
Graduate Student
RE: Cs 4.1.0 plus Xen 6.1 No Instances or VR's starting up
Posted by Kent Johnson <ke...@utah.edu>.
Wow guys, I am a little embarrassed. It looks like the whole problem I had must have been due to the file permissions problem on vhd-util. Like I said in my last message, the vhd-util file in /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver had permissions set to 0777. Also, that directory has a subdirectory called "xenserver60" which needed to have vhd-util copied there. As soon as I set the permissions to 0755 and made sure a copy was in both the parent and the sub-directory then my Virtual Router came up properly and my Instances came up properly. It is a bit embarrassing to be thwarted by something as small as a file permissions misconfiguration.
Nevertheless it is exciting to join the crew of those who have instances up and running! Now for configuring multiple hosts in my Xen cluster...
Thanks to all who helped me on my way! To see proof of my running VM's go to my blog (http://uofucloudcapstone.wordpress.com/)
and see this post: (http://uofucloudcapstone.wordpress.com/2013/08/24/finally-success/)
In Gratitude,
Kent Johnson
University of Utah
Graduate Student
MSIS Program
-----Original Message-----
From: Hongtu Zang [mailto:zanghongtu2006@gmail.com]
Sent: Saturday, August 24, 2013 7:14 PM
To: users@cloudstack.apache.org
Subject: Re: Cs 4.1.0 plus Xen 6.1 No Instances or VR's starting up
I have met such issue when I reuse an old xenserver which has been added to another cs before.
It is because that xenserver can not ssh into vrouter by authorizedkeys.
try to copy the systemvm.iso and id_rsa.pub from management server and replace which in xenserver now.
then you can start your vrouter.
If it doesn't work, you can try to re-install xenserver, i think.
2013/8/25 Kent Johnson <ke...@utah.edu>
> So I found something that could have caused my Virtual Router to not
> be able to start up.
>
> The permissions on my vhd-util may have been set wrong. They were set
> to
> 0777 and it looks like some things were looking for the permissions to
> be 0755. It doesn't make sense to me why that would be the problem but
> after I changed the permissions on the file in
> /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver/vhd-util
> and then copied that file to
> /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver/xenserver
> 60 then my virtual router started up properly. Why would the
> difference in permissions be the problem? Was it that I had not copied
> the file to the
> .../xenserver60 directory? Nothing in the docs told me about copying
> to
> .../xenserver60 and nothing mentioned particular permissions to be set.
>
> Whether or not that was the cause of my problem I do not know but I do
> know my Virtual Router is up and running now.
>
> Also, my instances used to go to an error state but now they don't.
> However, they stay in a perpetual starting state. I started a small
> instance and after 40 minutes it is still not running. It was
> successfully created and is now in the "Starting" state. How long
> should I give my instances before I start looking for issues?
>
> Kent Johnson
> University of Utah
> Graduate Student
> MSIS Program
>
> -----Original Message-----
> From: Kent Johnson [mailto:kent.johnson@utah.edu]
> Sent: Saturday, August 24, 2013 5:20 PM
> To: users@cloudstack.apache.org
> Subject: RE: Cs 4.1.0 plus Xen 6.1 No Instances or VR's starting up
>
> Which host are you referring to? My XenServer host mounts the
> secondary just fine. My secondary storage is a directory on my NAS
> using NFS. By the way, my NAS is on a separate subnet than all the
> rest of my devices. Would that cause any problems? My management
> server and my XenServer host mount it just fine and I haven't had any problems with it to date.
>
> Kent Johnson
> University of Utah
> Graduate Student
> MSIS Program
>
> -----Original Message-----
> From: Carlos Reátegui [mailto:creategui@gmail.com]
> Sent: Saturday, August 24, 2013 5:05 PM
> To: users@cloudstack.apache.org
> Subject: Re: Cs 4.1.0 plus Xen 6.1 No Instances or VR's starting up
>
> Is your host able to mount secondary storage?
>
> On Aug 24, 2013, at 3:34 PM, Kent Johnson <ke...@utah.edu> wrote:
>
> > Thank you Kirk.
> > As a preface to my response a virtual router is automatically
> > created
> every time I create my first virtual machine. I have not yet
> successfully been able to add an instance. The virtual router created
> when I make an instance is the VR I am having problems with.
> >
> > In answering your questions,
> > 1. The virtual router goes into the starting state and never makes
> > it to
> running. It errors out and then goes from starting state to stopping.
> I did see some suspicious lines on the log file:
> > 2013-08-23 18:47:58,361 DEBUG [xen.resource.CitrixResourceBase]
> > (DirectAgent-9:null) Trying to connect to 169.254.3.124
> > 2013-08-23 18:47:59,178 DEBUG [xen.resource.CitrixResourceBase]
> > (DirectAgent-9:null) Ping command port succeeded for vm r-11-VM
> > 2013-08-23 18:47:59,561 DEBUG [agent.manager.DirectAgentAttache]
> (DirectAgent-9:null) Seq 1-585236521: Cancelling because one of the
> answers is false and it is stop on error.
> >
> > Do you think the problem with my router not starting up is related
> > to
> the "stop on error" part of those log lines? A few lines later in the
> log it tells me the guru did not like the answers so is stopping the router:
> >
> > 2013-08-23 18:47:59,623 INFO [cloud.vm.VirtualMachineManagerImpl]
> > (Job-Executor-11:job-31) The guru did not like the answers so
> > stopping VM[DomainRouter|r-11-VM]
> > 2013-08-23 18:47:59,626 DEBUG [agent.transport.Request]
> > (Job-Executor-11:job-31) Seq 1-585236524: Sending { Cmd , MgmtId:
> > 166316981724, via: 1, Ver: v1, Flags: 100111,
> > [{"StopCommand":{"isProxy":false,"vmName":"r-11-VM","wait":0}}] }
> > 2013-08-23 18:47:59,627 DEBUG [agent.transport.Request]
> > (Job-Executor-11:job-31) Seq 1-585236524: Executing: { Cmd , MgmtId:
> > 166316981724, via: 1, Ver: v1, Flags: 100111,
> > [{"StopCommand":{"isProxy":false,"vmName":"r-11-VM","wait":0}}] }
> > 2013-08-23 18:47:59,627 DEBUG [agent.manager.DirectAgentAttache]
> > (DirectAgent-22:null) Seq 1-585236524: Executing request
> > 2013-08-23 18:47:59,774 DEBUG [xen.resource.CitrixResourceBase]
> > (DirectAgent-22:null) 9. The VM r-11-VM is in Stopping state
> > 2013-08-23 18:48:00,101 INFO [xen.resource.CitrixResourceBase]
> > (DirectAgent-22:null) Removed network rules for vm r-11-VM
> >
> > Later it says it is in stopped state and that stopping succeeded:
> > 2013-08-23 18:48:08,312 DEBUG [xen.resource.CitrixResourceBase]
> > (DirectAgent-22:null) 10. The VM r-11-VM is in Stopped state
> > 2013-08-23 18:48:08,313 DEBUG [agent.manager.AgentManagerImpl]
> > (Job-Executor-11:job-31) Details from executing class
> > com.cloud.agent.api.StopCommand: Stop VM r-11-VM Succeed
> >
> > Then it tells me the error was in finalizeStart:
> > 2013-08-23 18:48:08,313 ERROR [cloud.vm.VirtualMachineManagerImpl]
> > (Job-Executor-11:job-31) Failed to start instance
> > VM[DomainRouter|r-11-VM]
> > com.cloud.utils.exception.ExecutionException: Unable to start
> > VM[DomainRouter|r-11-VM] due to error in finalizeStart, not retrying
> >
> > Where can I see the finalizeStart method and the part of it that is
> failing?
> >
> > 2. I checked the /var/log/SMlog on the XenServer host and did not
> > find
> any lines that contained "getDomRVersion" How would I know if any of
> the entries are related to the getDomRVersion script?
> > I did notice that when I tried to start the VR then the SMlog showed
> > a
> failure after running router_proxy.sh and get_template_version.sh
> (shown in the log below):
> > [31327] 2013-08-24 16:14:20.346551 #### VMOPS enter routerProxy ####
> > [31327] 2013-08-24 16:14:20.346675 ['/bin/bash',
> '/opt/xensource/bin/router_proxy.sh', 'get_template_version.sh',
> '169.254.3.168']
> > [31327] 2013-08-24 16:14:20.464558 FAILED in util.pread: (rc 255)
> stdout: '', stderr: ''
> > [31327] 2013-08-24 16:14:20.464766 routerProxy command
> get_template_version.sh 169.254.3.168 failed
> > [31327] 2013-08-24 16:14:20.464872 #### VMOPS exit routerProxy ####
> >
> > 3. I checked for the /root/.ssh/id_rsa.cloud file on the XenServer
> > host
> and it did exist.
> >
> > 4. I tried forcing reconnect through the UI and that did not solve
> > the
> problem.
> > 5. I tried unmanaging and re-managing the cluster (I had to put my
> > host in maintenance mode first) 6. I tried clearing the host tags
> >
> > None of this solved my problem. I did see some entries in the SMlog
> > that
> made me wonder if vhd-util needs to be in my primary or secondary
> storage directory. To your knowledge, does it need to be? I thought
> vhd-util only needed to be in /usr/bin and /opt/xensource/bin. Is
> there somewhere else it needs to be?
> >
> >
> > I am starting to wonder if I should reinstall everything and use KVM
> with Libvirt. I really want to use Xen and CloudStack. This problem
> with my instances and the virtual router really is an impasse for my project.
> >
> > Thank you for your help, Kirk.
> >
> > Best,
> >
> > Kent Johnson
> > University of Utah
> > Graduate Student
> > MSIS Program
> > -----Original Message-----
> > From: Kirk Kosinski [mailto:kirkkosinski@gmail.com]
> > Sent: Friday, August 23, 2013 8:35 PM
> > To: users@cloudstack.apache.org
> > Cc: Ahmad Emneina
> > Subject: Re: Cs 4.1.0 plus Xen 6.1 No Instances or VR's starting up
> >
> > Does the virtual router start on XenServer and stop after a few
> > seconds
> or minutes? Or does it never start at all?
> >
> > Check /var/log/SMlog on the XenServer host for any entries or
> > (especially) errors related to the getDomRVersion script. If there
> > are
> no useful errors, find the script arguments in SMlog and try running
> it manually with bash -x to see where it is failing.
> >
> > One potential cause is a missing /root/.ssh/id_rsa.cloud on the
> XenServer host, so confirm it exists. Besides that, some general
> steps that might clear up the problem include:
> > 1. Force reconnect the host (CS API or UI).
> > 2. Unmanage and re-manage the cluster (CS API or UI).
> > 3. Unmanage cluster / clear host tags (xe host-param-clear
> > uuid=host_id
> > param-name=tags) / re-manage cluster
> >
> > Best regards,
> > Kirk
> >
> >
> > On 08/23/2013 04:24 PM, Ahmad Emneina wrote:
> >> looks like your management server reaches out to the router but
> >> barfs out
> >> here:
> >> 2013-08-23 17:04:53,938 DEBUG [agent.transport.Request]
> >> (Job-Executor-1:job-18) Seq 1-1363673129: Received: { Ans: , MgmtId:
> >> 166316981724, via: 1, Ver: v1, Flags: 110, { StartAnswer,
> >> CheckSshAnswer, GetDomRVersionAnswer } }
> >> 2013-08-23 17:04:53,983 WARN
> >> [network.router.VirtualNetworkApplianceManagerImpl]
> >> (Job-Executor-1:job-18) Unable to get the template/scripts version
> >> of router r-6-VM due to: getDomRVersionCmd failed
> >> 2013-08-23 17:04:53,984 INFO [cloud.vm.VirtualMachineManagerImpl]
> >> (Job-Executor-1:job-18) The guru did not like the answers so
> >> stopping VM[DomainRouter|r-6-VM]
> >> 2013-08-23 17:04:53,988 DEBUG [agent.transport.Request]
> >> (Job-Executor-1:job-18) Seq 1-1363673130: Sending { Cmd , MgmtId:
> >> 166316981724, via: 1, Ver: v1, Flags: 100111,
> >> [{"StopCommand":{"isProxy":false,"vmName":"r-6-VM","wait":0}}] }
> >>
> >> so the good news is your network config looks good, i just wonder
> >> if youre using the right template or if the system vm isnt patched
> properly.
> >>
> >>
> >> On Fri, Aug 23, 2013 at 4:16 PM, Kent Johnson
> >> <ke...@utah.edu>
> wrote:
> >>
> >>> Thank you Marty and Ahmad.
> >>>
> >>> Here is my management-server.log log: http://pastebin.com/TnRpsB8j
> >>>
> >>> Here is my catalina.out log: http://pastebin.com/index
> >>>
> >>> Here is my xensource.log from my xenserver host:
> >>> http://pastebin.com/RC6LhXQ4
> >>>
> >>> Best,
> >>>
> >>> Kent Johnson
> >>>
> >>> -----Original Message-----
> >>> From: Ahmad Emneina [mailto:aemneina@gmail.com]
> >>> Sent: Friday, August 23, 2013 5:00 PM
> >>> To: Cloudstack users mailing list
> >>> Subject: Re: Cs 4.1.0 plus Xen 6.1 No Instances or VR's starting
> >>> up
> >>>
> >>> probably best to post your logs to pastebin and have us sift over them.
> >>> see if we can spot anything.
> >>>
> >>>
> >>> On Fri, Aug 23, 2013 at 3:53 PM, Kent Johnson
> >>> <ke...@utah.edu>
> >>> wrote:
> >>>
> >>>> Can anyone point me in the right direction on solving my instance
> >>>> startup problem? They won't start up, nor will my virtual router.
> >>>>
> >>>> I am using CloudStack 4.1.0 on CentOS for my Management Server
> >>>> and XenServer 6.1 for my VM host.
> >>>>
> >>>> I installed CS as per the documentation and have successfully
> >>>> added a zone, pod, cluster, host, primary, and secondary storage.
> >>>> My System VM's start up and run correctly. My systemVM template
> >>>> and the default CentOS templates download correctly.
> >>>> I can create instances but they always try to start up and then
> >>>> fail and end up in "Error" state. My Virtual Router tries to
> >>>> start up But it always fails.
> >>>>
> >>>> Are there any gotchas or quick suggestions anyone could give me
> >>>> to help me understand how to get my instances to run and possibly
> >>>> my
> >>> virtual router?
> >>>> I can provide more details from the logs if needed.
> >>>>
> >>>> Kent Johnson
> >>>> University of Utah
> >>>> Graduate Student
> >>
>
Re: Cs 4.1.0 plus Xen 6.1 No Instances or VR's starting up
Posted by Hongtu Zang <za...@gmail.com>.
I have met such issue when I reuse an old xenserver which has been added to
another cs before.
It is because that xenserver can not ssh into vrouter by authorizedkeys.
try to copy the systemvm.iso and id_rsa.pub from management server and
replace which in xenserver now.
then you can start your vrouter.
If it doesn't work, you can try to re-install xenserver, i think.
2013/8/25 Kent Johnson <ke...@utah.edu>
> So I found something that could have caused my Virtual Router to not be
> able to start up.
>
> The permissions on my vhd-util may have been set wrong. They were set to
> 0777 and it looks like some things were looking for the permissions to be
> 0755. It doesn't make sense to me why that would be the problem but after I
> changed the permissions on the file in
> /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver/vhd-util and
> then copied that file to
> /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver/xenserver60
> then my virtual router started up properly. Why would the difference in
> permissions be the problem? Was it that I had not copied the file to the
> .../xenserver60 directory? Nothing in the docs told me about copying to
> .../xenserver60 and nothing mentioned particular permissions to be set.
>
> Whether or not that was the cause of my problem I do not know but I do
> know my Virtual Router is up and running now.
>
> Also, my instances used to go to an error state but now they don't.
> However, they stay in a perpetual starting state. I started a small
> instance and after 40 minutes it is still not running. It was successfully
> created and is now in the "Starting" state. How long should I give my
> instances before I start looking for issues?
>
> Kent Johnson
> University of Utah
> Graduate Student
> MSIS Program
>
> -----Original Message-----
> From: Kent Johnson [mailto:kent.johnson@utah.edu]
> Sent: Saturday, August 24, 2013 5:20 PM
> To: users@cloudstack.apache.org
> Subject: RE: Cs 4.1.0 plus Xen 6.1 No Instances or VR's starting up
>
> Which host are you referring to? My XenServer host mounts the secondary
> just fine. My secondary storage is a directory on my NAS using NFS. By the
> way, my NAS is on a separate subnet than all the rest of my devices. Would
> that cause any problems? My management server and my XenServer host mount
> it just fine and I haven't had any problems with it to date.
>
> Kent Johnson
> University of Utah
> Graduate Student
> MSIS Program
>
> -----Original Message-----
> From: Carlos Reátegui [mailto:creategui@gmail.com]
> Sent: Saturday, August 24, 2013 5:05 PM
> To: users@cloudstack.apache.org
> Subject: Re: Cs 4.1.0 plus Xen 6.1 No Instances or VR's starting up
>
> Is your host able to mount secondary storage?
>
> On Aug 24, 2013, at 3:34 PM, Kent Johnson <ke...@utah.edu> wrote:
>
> > Thank you Kirk.
> > As a preface to my response a virtual router is automatically created
> every time I create my first virtual machine. I have not yet successfully
> been able to add an instance. The virtual router created when I make an
> instance is the VR I am having problems with.
> >
> > In answering your questions,
> > 1. The virtual router goes into the starting state and never makes it to
> running. It errors out and then goes from starting state to stopping. I did
> see some suspicious lines on the log file:
> > 2013-08-23 18:47:58,361 DEBUG [xen.resource.CitrixResourceBase]
> > (DirectAgent-9:null) Trying to connect to 169.254.3.124
> > 2013-08-23 18:47:59,178 DEBUG [xen.resource.CitrixResourceBase]
> > (DirectAgent-9:null) Ping command port succeeded for vm r-11-VM
> > 2013-08-23 18:47:59,561 DEBUG [agent.manager.DirectAgentAttache]
> (DirectAgent-9:null) Seq 1-585236521: Cancelling because one of the answers
> is false and it is stop on error.
> >
> > Do you think the problem with my router not starting up is related to
> the "stop on error" part of those log lines? A few lines later in the log
> it tells me the guru did not like the answers so is stopping the router:
> >
> > 2013-08-23 18:47:59,623 INFO [cloud.vm.VirtualMachineManagerImpl]
> > (Job-Executor-11:job-31) The guru did not like the answers so stopping
> > VM[DomainRouter|r-11-VM]
> > 2013-08-23 18:47:59,626 DEBUG [agent.transport.Request]
> > (Job-Executor-11:job-31) Seq 1-585236524: Sending { Cmd , MgmtId:
> > 166316981724, via: 1, Ver: v1, Flags: 100111,
> > [{"StopCommand":{"isProxy":false,"vmName":"r-11-VM","wait":0}}] }
> > 2013-08-23 18:47:59,627 DEBUG [agent.transport.Request]
> > (Job-Executor-11:job-31) Seq 1-585236524: Executing: { Cmd , MgmtId:
> > 166316981724, via: 1, Ver: v1, Flags: 100111,
> > [{"StopCommand":{"isProxy":false,"vmName":"r-11-VM","wait":0}}] }
> > 2013-08-23 18:47:59,627 DEBUG [agent.manager.DirectAgentAttache]
> > (DirectAgent-22:null) Seq 1-585236524: Executing request
> > 2013-08-23 18:47:59,774 DEBUG [xen.resource.CitrixResourceBase]
> > (DirectAgent-22:null) 9. The VM r-11-VM is in Stopping state
> > 2013-08-23 18:48:00,101 INFO [xen.resource.CitrixResourceBase]
> > (DirectAgent-22:null) Removed network rules for vm r-11-VM
> >
> > Later it says it is in stopped state and that stopping succeeded:
> > 2013-08-23 18:48:08,312 DEBUG [xen.resource.CitrixResourceBase]
> > (DirectAgent-22:null) 10. The VM r-11-VM is in Stopped state
> > 2013-08-23 18:48:08,313 DEBUG [agent.manager.AgentManagerImpl]
> > (Job-Executor-11:job-31) Details from executing class
> > com.cloud.agent.api.StopCommand: Stop VM r-11-VM Succeed
> >
> > Then it tells me the error was in finalizeStart:
> > 2013-08-23 18:48:08,313 ERROR [cloud.vm.VirtualMachineManagerImpl]
> > (Job-Executor-11:job-31) Failed to start instance
> > VM[DomainRouter|r-11-VM]
> > com.cloud.utils.exception.ExecutionException: Unable to start
> > VM[DomainRouter|r-11-VM] due to error in finalizeStart, not retrying
> >
> > Where can I see the finalizeStart method and the part of it that is
> failing?
> >
> > 2. I checked the /var/log/SMlog on the XenServer host and did not find
> any lines that contained "getDomRVersion" How would I know if any of the
> entries are related to the getDomRVersion script?
> > I did notice that when I tried to start the VR then the SMlog showed a
> failure after running router_proxy.sh and get_template_version.sh (shown in
> the log below):
> > [31327] 2013-08-24 16:14:20.346551 #### VMOPS enter routerProxy ####
> > [31327] 2013-08-24 16:14:20.346675 ['/bin/bash',
> '/opt/xensource/bin/router_proxy.sh', 'get_template_version.sh',
> '169.254.3.168']
> > [31327] 2013-08-24 16:14:20.464558 FAILED in util.pread: (rc 255)
> stdout: '', stderr: ''
> > [31327] 2013-08-24 16:14:20.464766 routerProxy command
> get_template_version.sh 169.254.3.168 failed
> > [31327] 2013-08-24 16:14:20.464872 #### VMOPS exit routerProxy ####
> >
> > 3. I checked for the /root/.ssh/id_rsa.cloud file on the XenServer host
> and it did exist.
> >
> > 4. I tried forcing reconnect through the UI and that did not solve the
> problem.
> > 5. I tried unmanaging and re-managing the cluster (I had to put my
> > host in maintenance mode first) 6. I tried clearing the host tags
> >
> > None of this solved my problem. I did see some entries in the SMlog that
> made me wonder if vhd-util needs to be in my primary or secondary storage
> directory. To your knowledge, does it need to be? I thought vhd-util only
> needed to be in /usr/bin and /opt/xensource/bin. Is there somewhere else it
> needs to be?
> >
> >
> > I am starting to wonder if I should reinstall everything and use KVM
> with Libvirt. I really want to use Xen and CloudStack. This problem with my
> instances and the virtual router really is an impasse for my project.
> >
> > Thank you for your help, Kirk.
> >
> > Best,
> >
> > Kent Johnson
> > University of Utah
> > Graduate Student
> > MSIS Program
> > -----Original Message-----
> > From: Kirk Kosinski [mailto:kirkkosinski@gmail.com]
> > Sent: Friday, August 23, 2013 8:35 PM
> > To: users@cloudstack.apache.org
> > Cc: Ahmad Emneina
> > Subject: Re: Cs 4.1.0 plus Xen 6.1 No Instances or VR's starting up
> >
> > Does the virtual router start on XenServer and stop after a few seconds
> or minutes? Or does it never start at all?
> >
> > Check /var/log/SMlog on the XenServer host for any entries or
> > (especially) errors related to the getDomRVersion script. If there are
> no useful errors, find the script arguments in SMlog and try running it
> manually with bash -x to see where it is failing.
> >
> > One potential cause is a missing /root/.ssh/id_rsa.cloud on the
> XenServer host, so confirm it exists. Besides that, some general steps
> that might clear up the problem include:
> > 1. Force reconnect the host (CS API or UI).
> > 2. Unmanage and re-manage the cluster (CS API or UI).
> > 3. Unmanage cluster / clear host tags (xe host-param-clear
> > uuid=host_id
> > param-name=tags) / re-manage cluster
> >
> > Best regards,
> > Kirk
> >
> >
> > On 08/23/2013 04:24 PM, Ahmad Emneina wrote:
> >> looks like your management server reaches out to the router but barfs
> >> out
> >> here:
> >> 2013-08-23 17:04:53,938 DEBUG [agent.transport.Request]
> >> (Job-Executor-1:job-18) Seq 1-1363673129: Received: { Ans: , MgmtId:
> >> 166316981724, via: 1, Ver: v1, Flags: 110, { StartAnswer,
> >> CheckSshAnswer, GetDomRVersionAnswer } }
> >> 2013-08-23 17:04:53,983 WARN
> >> [network.router.VirtualNetworkApplianceManagerImpl]
> >> (Job-Executor-1:job-18) Unable to get the template/scripts version of
> >> router r-6-VM due to: getDomRVersionCmd failed
> >> 2013-08-23 17:04:53,984 INFO [cloud.vm.VirtualMachineManagerImpl]
> >> (Job-Executor-1:job-18) The guru did not like the answers so stopping
> >> VM[DomainRouter|r-6-VM]
> >> 2013-08-23 17:04:53,988 DEBUG [agent.transport.Request]
> >> (Job-Executor-1:job-18) Seq 1-1363673130: Sending { Cmd , MgmtId:
> >> 166316981724, via: 1, Ver: v1, Flags: 100111,
> >> [{"StopCommand":{"isProxy":false,"vmName":"r-6-VM","wait":0}}] }
> >>
> >> so the good news is your network config looks good, i just wonder if
> >> youre using the right template or if the system vm isnt patched
> properly.
> >>
> >>
> >> On Fri, Aug 23, 2013 at 4:16 PM, Kent Johnson <ke...@utah.edu>
> wrote:
> >>
> >>> Thank you Marty and Ahmad.
> >>>
> >>> Here is my management-server.log log: http://pastebin.com/TnRpsB8j
> >>>
> >>> Here is my catalina.out log: http://pastebin.com/index
> >>>
> >>> Here is my xensource.log from my xenserver host:
> >>> http://pastebin.com/RC6LhXQ4
> >>>
> >>> Best,
> >>>
> >>> Kent Johnson
> >>>
> >>> -----Original Message-----
> >>> From: Ahmad Emneina [mailto:aemneina@gmail.com]
> >>> Sent: Friday, August 23, 2013 5:00 PM
> >>> To: Cloudstack users mailing list
> >>> Subject: Re: Cs 4.1.0 plus Xen 6.1 No Instances or VR's starting up
> >>>
> >>> probably best to post your logs to pastebin and have us sift over them.
> >>> see if we can spot anything.
> >>>
> >>>
> >>> On Fri, Aug 23, 2013 at 3:53 PM, Kent Johnson
> >>> <ke...@utah.edu>
> >>> wrote:
> >>>
> >>>> Can anyone point me in the right direction on solving my instance
> >>>> startup problem? They won't start up, nor will my virtual router.
> >>>>
> >>>> I am using CloudStack 4.1.0 on CentOS for my Management Server and
> >>>> XenServer 6.1 for my VM host.
> >>>>
> >>>> I installed CS as per the documentation and have successfully added
> >>>> a zone, pod, cluster, host, primary, and secondary storage.
> >>>> My System VM's start up and run correctly. My systemVM template and
> >>>> the default CentOS templates download correctly.
> >>>> I can create instances but they always try to start up and then
> >>>> fail and end up in "Error" state. My Virtual Router tries to start
> >>>> up But it always fails.
> >>>>
> >>>> Are there any gotchas or quick suggestions anyone could give me to
> >>>> help me understand how to get my instances to run and possibly my
> >>> virtual router?
> >>>> I can provide more details from the logs if needed.
> >>>>
> >>>> Kent Johnson
> >>>> University of Utah
> >>>> Graduate Student
> >>
>
RE: Cs 4.1.0 plus Xen 6.1 No Instances or VR's starting up
Posted by Kent Johnson <ke...@utah.edu>.
So I found something that could have caused my Virtual Router to not be able to start up.
The permissions on my vhd-util may have been set wrong. They were set to 0777 and it looks like some things were looking for the permissions to be 0755. It doesn't make sense to me why that would be the problem but after I changed the permissions on the file in /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver/vhd-util and then copied that file to /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver/xenserver60 then my virtual router started up properly. Why would the difference in permissions be the problem? Was it that I had not copied the file to the .../xenserver60 directory? Nothing in the docs told me about copying to .../xenserver60 and nothing mentioned particular permissions to be set.
Whether or not that was the cause of my problem I do not know but I do know my Virtual Router is up and running now.
Also, my instances used to go to an error state but now they don't. However, they stay in a perpetual starting state. I started a small instance and after 40 minutes it is still not running. It was successfully created and is now in the "Starting" state. How long should I give my instances before I start looking for issues?
Kent Johnson
University of Utah
Graduate Student
MSIS Program
-----Original Message-----
From: Kent Johnson [mailto:kent.johnson@utah.edu]
Sent: Saturday, August 24, 2013 5:20 PM
To: users@cloudstack.apache.org
Subject: RE: Cs 4.1.0 plus Xen 6.1 No Instances or VR's starting up
Which host are you referring to? My XenServer host mounts the secondary just fine. My secondary storage is a directory on my NAS using NFS. By the way, my NAS is on a separate subnet than all the rest of my devices. Would that cause any problems? My management server and my XenServer host mount it just fine and I haven't had any problems with it to date.
Kent Johnson
University of Utah
Graduate Student
MSIS Program
-----Original Message-----
From: Carlos Reátegui [mailto:creategui@gmail.com]
Sent: Saturday, August 24, 2013 5:05 PM
To: users@cloudstack.apache.org
Subject: Re: Cs 4.1.0 plus Xen 6.1 No Instances or VR's starting up
Is your host able to mount secondary storage?
On Aug 24, 2013, at 3:34 PM, Kent Johnson <ke...@utah.edu> wrote:
> Thank you Kirk.
> As a preface to my response a virtual router is automatically created every time I create my first virtual machine. I have not yet successfully been able to add an instance. The virtual router created when I make an instance is the VR I am having problems with.
>
> In answering your questions,
> 1. The virtual router goes into the starting state and never makes it to running. It errors out and then goes from starting state to stopping. I did see some suspicious lines on the log file:
> 2013-08-23 18:47:58,361 DEBUG [xen.resource.CitrixResourceBase]
> (DirectAgent-9:null) Trying to connect to 169.254.3.124
> 2013-08-23 18:47:59,178 DEBUG [xen.resource.CitrixResourceBase]
> (DirectAgent-9:null) Ping command port succeeded for vm r-11-VM
> 2013-08-23 18:47:59,561 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-9:null) Seq 1-585236521: Cancelling because one of the answers is false and it is stop on error.
>
> Do you think the problem with my router not starting up is related to the "stop on error" part of those log lines? A few lines later in the log it tells me the guru did not like the answers so is stopping the router:
>
> 2013-08-23 18:47:59,623 INFO [cloud.vm.VirtualMachineManagerImpl]
> (Job-Executor-11:job-31) The guru did not like the answers so stopping
> VM[DomainRouter|r-11-VM]
> 2013-08-23 18:47:59,626 DEBUG [agent.transport.Request]
> (Job-Executor-11:job-31) Seq 1-585236524: Sending { Cmd , MgmtId:
> 166316981724, via: 1, Ver: v1, Flags: 100111,
> [{"StopCommand":{"isProxy":false,"vmName":"r-11-VM","wait":0}}] }
> 2013-08-23 18:47:59,627 DEBUG [agent.transport.Request]
> (Job-Executor-11:job-31) Seq 1-585236524: Executing: { Cmd , MgmtId:
> 166316981724, via: 1, Ver: v1, Flags: 100111,
> [{"StopCommand":{"isProxy":false,"vmName":"r-11-VM","wait":0}}] }
> 2013-08-23 18:47:59,627 DEBUG [agent.manager.DirectAgentAttache]
> (DirectAgent-22:null) Seq 1-585236524: Executing request
> 2013-08-23 18:47:59,774 DEBUG [xen.resource.CitrixResourceBase]
> (DirectAgent-22:null) 9. The VM r-11-VM is in Stopping state
> 2013-08-23 18:48:00,101 INFO [xen.resource.CitrixResourceBase]
> (DirectAgent-22:null) Removed network rules for vm r-11-VM
>
> Later it says it is in stopped state and that stopping succeeded:
> 2013-08-23 18:48:08,312 DEBUG [xen.resource.CitrixResourceBase]
> (DirectAgent-22:null) 10. The VM r-11-VM is in Stopped state
> 2013-08-23 18:48:08,313 DEBUG [agent.manager.AgentManagerImpl]
> (Job-Executor-11:job-31) Details from executing class
> com.cloud.agent.api.StopCommand: Stop VM r-11-VM Succeed
>
> Then it tells me the error was in finalizeStart:
> 2013-08-23 18:48:08,313 ERROR [cloud.vm.VirtualMachineManagerImpl]
> (Job-Executor-11:job-31) Failed to start instance
> VM[DomainRouter|r-11-VM]
> com.cloud.utils.exception.ExecutionException: Unable to start
> VM[DomainRouter|r-11-VM] due to error in finalizeStart, not retrying
>
> Where can I see the finalizeStart method and the part of it that is failing?
>
> 2. I checked the /var/log/SMlog on the XenServer host and did not find any lines that contained "getDomRVersion" How would I know if any of the entries are related to the getDomRVersion script?
> I did notice that when I tried to start the VR then the SMlog showed a failure after running router_proxy.sh and get_template_version.sh (shown in the log below):
> [31327] 2013-08-24 16:14:20.346551 #### VMOPS enter routerProxy ####
> [31327] 2013-08-24 16:14:20.346675 ['/bin/bash', '/opt/xensource/bin/router_proxy.sh', 'get_template_version.sh', '169.254.3.168']
> [31327] 2013-08-24 16:14:20.464558 FAILED in util.pread: (rc 255) stdout: '', stderr: ''
> [31327] 2013-08-24 16:14:20.464766 routerProxy command get_template_version.sh 169.254.3.168 failed
> [31327] 2013-08-24 16:14:20.464872 #### VMOPS exit routerProxy ####
>
> 3. I checked for the /root/.ssh/id_rsa.cloud file on the XenServer host and it did exist.
>
> 4. I tried forcing reconnect through the UI and that did not solve the problem.
> 5. I tried unmanaging and re-managing the cluster (I had to put my
> host in maintenance mode first) 6. I tried clearing the host tags
>
> None of this solved my problem. I did see some entries in the SMlog that made me wonder if vhd-util needs to be in my primary or secondary storage directory. To your knowledge, does it need to be? I thought vhd-util only needed to be in /usr/bin and /opt/xensource/bin. Is there somewhere else it needs to be?
>
>
> I am starting to wonder if I should reinstall everything and use KVM with Libvirt. I really want to use Xen and CloudStack. This problem with my instances and the virtual router really is an impasse for my project.
>
> Thank you for your help, Kirk.
>
> Best,
>
> Kent Johnson
> University of Utah
> Graduate Student
> MSIS Program
> -----Original Message-----
> From: Kirk Kosinski [mailto:kirkkosinski@gmail.com]
> Sent: Friday, August 23, 2013 8:35 PM
> To: users@cloudstack.apache.org
> Cc: Ahmad Emneina
> Subject: Re: Cs 4.1.0 plus Xen 6.1 No Instances or VR's starting up
>
> Does the virtual router start on XenServer and stop after a few seconds or minutes? Or does it never start at all?
>
> Check /var/log/SMlog on the XenServer host for any entries or
> (especially) errors related to the getDomRVersion script. If there are no useful errors, find the script arguments in SMlog and try running it manually with bash -x to see where it is failing.
>
> One potential cause is a missing /root/.ssh/id_rsa.cloud on the XenServer host, so confirm it exists. Besides that, some general steps that might clear up the problem include:
> 1. Force reconnect the host (CS API or UI).
> 2. Unmanage and re-manage the cluster (CS API or UI).
> 3. Unmanage cluster / clear host tags (xe host-param-clear
> uuid=host_id
> param-name=tags) / re-manage cluster
>
> Best regards,
> Kirk
>
>
> On 08/23/2013 04:24 PM, Ahmad Emneina wrote:
>> looks like your management server reaches out to the router but barfs
>> out
>> here:
>> 2013-08-23 17:04:53,938 DEBUG [agent.transport.Request]
>> (Job-Executor-1:job-18) Seq 1-1363673129: Received: { Ans: , MgmtId:
>> 166316981724, via: 1, Ver: v1, Flags: 110, { StartAnswer,
>> CheckSshAnswer, GetDomRVersionAnswer } }
>> 2013-08-23 17:04:53,983 WARN
>> [network.router.VirtualNetworkApplianceManagerImpl]
>> (Job-Executor-1:job-18) Unable to get the template/scripts version of
>> router r-6-VM due to: getDomRVersionCmd failed
>> 2013-08-23 17:04:53,984 INFO [cloud.vm.VirtualMachineManagerImpl]
>> (Job-Executor-1:job-18) The guru did not like the answers so stopping
>> VM[DomainRouter|r-6-VM]
>> 2013-08-23 17:04:53,988 DEBUG [agent.transport.Request]
>> (Job-Executor-1:job-18) Seq 1-1363673130: Sending { Cmd , MgmtId:
>> 166316981724, via: 1, Ver: v1, Flags: 100111,
>> [{"StopCommand":{"isProxy":false,"vmName":"r-6-VM","wait":0}}] }
>>
>> so the good news is your network config looks good, i just wonder if
>> youre using the right template or if the system vm isnt patched properly.
>>
>>
>> On Fri, Aug 23, 2013 at 4:16 PM, Kent Johnson <ke...@utah.edu> wrote:
>>
>>> Thank you Marty and Ahmad.
>>>
>>> Here is my management-server.log log: http://pastebin.com/TnRpsB8j
>>>
>>> Here is my catalina.out log: http://pastebin.com/index
>>>
>>> Here is my xensource.log from my xenserver host:
>>> http://pastebin.com/RC6LhXQ4
>>>
>>> Best,
>>>
>>> Kent Johnson
>>>
>>> -----Original Message-----
>>> From: Ahmad Emneina [mailto:aemneina@gmail.com]
>>> Sent: Friday, August 23, 2013 5:00 PM
>>> To: Cloudstack users mailing list
>>> Subject: Re: Cs 4.1.0 plus Xen 6.1 No Instances or VR's starting up
>>>
>>> probably best to post your logs to pastebin and have us sift over them.
>>> see if we can spot anything.
>>>
>>>
>>> On Fri, Aug 23, 2013 at 3:53 PM, Kent Johnson
>>> <ke...@utah.edu>
>>> wrote:
>>>
>>>> Can anyone point me in the right direction on solving my instance
>>>> startup problem? They won't start up, nor will my virtual router.
>>>>
>>>> I am using CloudStack 4.1.0 on CentOS for my Management Server and
>>>> XenServer 6.1 for my VM host.
>>>>
>>>> I installed CS as per the documentation and have successfully added
>>>> a zone, pod, cluster, host, primary, and secondary storage.
>>>> My System VM's start up and run correctly. My systemVM template and
>>>> the default CentOS templates download correctly.
>>>> I can create instances but they always try to start up and then
>>>> fail and end up in "Error" state. My Virtual Router tries to start
>>>> up But it always fails.
>>>>
>>>> Are there any gotchas or quick suggestions anyone could give me to
>>>> help me understand how to get my instances to run and possibly my
>>> virtual router?
>>>> I can provide more details from the logs if needed.
>>>>
>>>> Kent Johnson
>>>> University of Utah
>>>> Graduate Student
>>
RE: Cs 4.1.0 plus Xen 6.1 No Instances or VR's starting up
Posted by Kent Johnson <ke...@utah.edu>.
Which host are you referring to? My XenServer host mounts the secondary just fine. My secondary storage is a directory on my NAS using NFS. By the way, my NAS is on a separate subnet than all the rest of my devices. Would that cause any problems? My management server and my XenServer host mount it just fine and I haven't had any problems with it to date.
Kent Johnson
University of Utah
Graduate Student
MSIS Program
-----Original Message-----
From: Carlos Reátegui [mailto:creategui@gmail.com]
Sent: Saturday, August 24, 2013 5:05 PM
To: users@cloudstack.apache.org
Subject: Re: Cs 4.1.0 plus Xen 6.1 No Instances or VR's starting up
Is your host able to mount secondary storage?
On Aug 24, 2013, at 3:34 PM, Kent Johnson <ke...@utah.edu> wrote:
> Thank you Kirk.
> As a preface to my response a virtual router is automatically created every time I create my first virtual machine. I have not yet successfully been able to add an instance. The virtual router created when I make an instance is the VR I am having problems with.
>
> In answering your questions,
> 1. The virtual router goes into the starting state and never makes it to running. It errors out and then goes from starting state to stopping. I did see some suspicious lines on the log file:
> 2013-08-23 18:47:58,361 DEBUG [xen.resource.CitrixResourceBase]
> (DirectAgent-9:null) Trying to connect to 169.254.3.124
> 2013-08-23 18:47:59,178 DEBUG [xen.resource.CitrixResourceBase]
> (DirectAgent-9:null) Ping command port succeeded for vm r-11-VM
> 2013-08-23 18:47:59,561 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-9:null) Seq 1-585236521: Cancelling because one of the answers is false and it is stop on error.
>
> Do you think the problem with my router not starting up is related to the "stop on error" part of those log lines? A few lines later in the log it tells me the guru did not like the answers so is stopping the router:
>
> 2013-08-23 18:47:59,623 INFO [cloud.vm.VirtualMachineManagerImpl]
> (Job-Executor-11:job-31) The guru did not like the answers so stopping
> VM[DomainRouter|r-11-VM]
> 2013-08-23 18:47:59,626 DEBUG [agent.transport.Request]
> (Job-Executor-11:job-31) Seq 1-585236524: Sending { Cmd , MgmtId:
> 166316981724, via: 1, Ver: v1, Flags: 100111,
> [{"StopCommand":{"isProxy":false,"vmName":"r-11-VM","wait":0}}] }
> 2013-08-23 18:47:59,627 DEBUG [agent.transport.Request]
> (Job-Executor-11:job-31) Seq 1-585236524: Executing: { Cmd , MgmtId:
> 166316981724, via: 1, Ver: v1, Flags: 100111,
> [{"StopCommand":{"isProxy":false,"vmName":"r-11-VM","wait":0}}] }
> 2013-08-23 18:47:59,627 DEBUG [agent.manager.DirectAgentAttache]
> (DirectAgent-22:null) Seq 1-585236524: Executing request
> 2013-08-23 18:47:59,774 DEBUG [xen.resource.CitrixResourceBase]
> (DirectAgent-22:null) 9. The VM r-11-VM is in Stopping state
> 2013-08-23 18:48:00,101 INFO [xen.resource.CitrixResourceBase]
> (DirectAgent-22:null) Removed network rules for vm r-11-VM
>
> Later it says it is in stopped state and that stopping succeeded:
> 2013-08-23 18:48:08,312 DEBUG [xen.resource.CitrixResourceBase]
> (DirectAgent-22:null) 10. The VM r-11-VM is in Stopped state
> 2013-08-23 18:48:08,313 DEBUG [agent.manager.AgentManagerImpl]
> (Job-Executor-11:job-31) Details from executing class
> com.cloud.agent.api.StopCommand: Stop VM r-11-VM Succeed
>
> Then it tells me the error was in finalizeStart:
> 2013-08-23 18:48:08,313 ERROR [cloud.vm.VirtualMachineManagerImpl]
> (Job-Executor-11:job-31) Failed to start instance
> VM[DomainRouter|r-11-VM]
> com.cloud.utils.exception.ExecutionException: Unable to start
> VM[DomainRouter|r-11-VM] due to error in finalizeStart, not retrying
>
> Where can I see the finalizeStart method and the part of it that is failing?
>
> 2. I checked the /var/log/SMlog on the XenServer host and did not find any lines that contained "getDomRVersion" How would I know if any of the entries are related to the getDomRVersion script?
> I did notice that when I tried to start the VR then the SMlog showed a failure after running router_proxy.sh and get_template_version.sh (shown in the log below):
> [31327] 2013-08-24 16:14:20.346551 #### VMOPS enter routerProxy ####
> [31327] 2013-08-24 16:14:20.346675 ['/bin/bash', '/opt/xensource/bin/router_proxy.sh', 'get_template_version.sh', '169.254.3.168']
> [31327] 2013-08-24 16:14:20.464558 FAILED in util.pread: (rc 255) stdout: '', stderr: ''
> [31327] 2013-08-24 16:14:20.464766 routerProxy command get_template_version.sh 169.254.3.168 failed
> [31327] 2013-08-24 16:14:20.464872 #### VMOPS exit routerProxy ####
>
> 3. I checked for the /root/.ssh/id_rsa.cloud file on the XenServer host and it did exist.
>
> 4. I tried forcing reconnect through the UI and that did not solve the problem.
> 5. I tried unmanaging and re-managing the cluster (I had to put my
> host in maintenance mode first) 6. I tried clearing the host tags
>
> None of this solved my problem. I did see some entries in the SMlog that made me wonder if vhd-util needs to be in my primary or secondary storage directory. To your knowledge, does it need to be? I thought vhd-util only needed to be in /usr/bin and /opt/xensource/bin. Is there somewhere else it needs to be?
>
>
> I am starting to wonder if I should reinstall everything and use KVM with Libvirt. I really want to use Xen and CloudStack. This problem with my instances and the virtual router really is an impasse for my project.
>
> Thank you for your help, Kirk.
>
> Best,
>
> Kent Johnson
> University of Utah
> Graduate Student
> MSIS Program
> -----Original Message-----
> From: Kirk Kosinski [mailto:kirkkosinski@gmail.com]
> Sent: Friday, August 23, 2013 8:35 PM
> To: users@cloudstack.apache.org
> Cc: Ahmad Emneina
> Subject: Re: Cs 4.1.0 plus Xen 6.1 No Instances or VR's starting up
>
> Does the virtual router start on XenServer and stop after a few seconds or minutes? Or does it never start at all?
>
> Check /var/log/SMlog on the XenServer host for any entries or
> (especially) errors related to the getDomRVersion script. If there are no useful errors, find the script arguments in SMlog and try running it manually with bash -x to see where it is failing.
>
> One potential cause is a missing /root/.ssh/id_rsa.cloud on the XenServer host, so confirm it exists. Besides that, some general steps that might clear up the problem include:
> 1. Force reconnect the host (CS API or UI).
> 2. Unmanage and re-manage the cluster (CS API or UI).
> 3. Unmanage cluster / clear host tags (xe host-param-clear
> uuid=host_id
> param-name=tags) / re-manage cluster
>
> Best regards,
> Kirk
>
>
> On 08/23/2013 04:24 PM, Ahmad Emneina wrote:
>> looks like your management server reaches out to the router but barfs
>> out
>> here:
>> 2013-08-23 17:04:53,938 DEBUG [agent.transport.Request]
>> (Job-Executor-1:job-18) Seq 1-1363673129: Received: { Ans: , MgmtId:
>> 166316981724, via: 1, Ver: v1, Flags: 110, { StartAnswer,
>> CheckSshAnswer, GetDomRVersionAnswer } }
>> 2013-08-23 17:04:53,983 WARN
>> [network.router.VirtualNetworkApplianceManagerImpl]
>> (Job-Executor-1:job-18) Unable to get the template/scripts version of
>> router r-6-VM due to: getDomRVersionCmd failed
>> 2013-08-23 17:04:53,984 INFO [cloud.vm.VirtualMachineManagerImpl]
>> (Job-Executor-1:job-18) The guru did not like the answers so stopping
>> VM[DomainRouter|r-6-VM]
>> 2013-08-23 17:04:53,988 DEBUG [agent.transport.Request]
>> (Job-Executor-1:job-18) Seq 1-1363673130: Sending { Cmd , MgmtId:
>> 166316981724, via: 1, Ver: v1, Flags: 100111,
>> [{"StopCommand":{"isProxy":false,"vmName":"r-6-VM","wait":0}}] }
>>
>> so the good news is your network config looks good, i just wonder if
>> youre using the right template or if the system vm isnt patched properly.
>>
>>
>> On Fri, Aug 23, 2013 at 4:16 PM, Kent Johnson <ke...@utah.edu> wrote:
>>
>>> Thank you Marty and Ahmad.
>>>
>>> Here is my management-server.log log: http://pastebin.com/TnRpsB8j
>>>
>>> Here is my catalina.out log: http://pastebin.com/index
>>>
>>> Here is my xensource.log from my xenserver host:
>>> http://pastebin.com/RC6LhXQ4
>>>
>>> Best,
>>>
>>> Kent Johnson
>>>
>>> -----Original Message-----
>>> From: Ahmad Emneina [mailto:aemneina@gmail.com]
>>> Sent: Friday, August 23, 2013 5:00 PM
>>> To: Cloudstack users mailing list
>>> Subject: Re: Cs 4.1.0 plus Xen 6.1 No Instances or VR's starting up
>>>
>>> probably best to post your logs to pastebin and have us sift over them.
>>> see if we can spot anything.
>>>
>>>
>>> On Fri, Aug 23, 2013 at 3:53 PM, Kent Johnson
>>> <ke...@utah.edu>
>>> wrote:
>>>
>>>> Can anyone point me in the right direction on solving my instance
>>>> startup problem? They won't start up, nor will my virtual router.
>>>>
>>>> I am using CloudStack 4.1.0 on CentOS for my Management Server and
>>>> XenServer 6.1 for my VM host.
>>>>
>>>> I installed CS as per the documentation and have successfully added
>>>> a zone, pod, cluster, host, primary, and secondary storage.
>>>> My System VM's start up and run correctly. My systemVM template and
>>>> the default CentOS templates download correctly.
>>>> I can create instances but they always try to start up and then
>>>> fail and end up in "Error" state. My Virtual Router tries to start
>>>> up But it always fails.
>>>>
>>>> Are there any gotchas or quick suggestions anyone could give me to
>>>> help me understand how to get my instances to run and possibly my
>>> virtual router?
>>>> I can provide more details from the logs if needed.
>>>>
>>>> Kent Johnson
>>>> University of Utah
>>>> Graduate Student
>>
Re: Cs 4.1.0 plus Xen 6.1 No Instances or VR's starting up
Posted by Carlos Reátegui <cr...@gmail.com>.
Is your host able to mount secondary storage?
On Aug 24, 2013, at 3:34 PM, Kent Johnson <ke...@utah.edu> wrote:
> Thank you Kirk.
> As a preface to my response a virtual router is automatically created every time I create my first virtual machine. I have not yet successfully been able to add an instance. The virtual router created when I make an instance is the VR I am having problems with.
>
> In answering your questions,
> 1. The virtual router goes into the starting state and never makes it to running. It errors out and then goes from starting state to stopping. I did see some suspicious lines on the log file:
> 2013-08-23 18:47:58,361 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-9:null) Trying to connect to 169.254.3.124
> 2013-08-23 18:47:59,178 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-9:null) Ping command port succeeded for vm r-11-VM
> 2013-08-23 18:47:59,561 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-9:null) Seq 1-585236521: Cancelling because one of the answers is false and it is stop on error.
>
> Do you think the problem with my router not starting up is related to the "stop on error" part of those log lines? A few lines later in the log it tells me the guru did not like the answers so is stopping the router:
>
> 2013-08-23 18:47:59,623 INFO [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-11:job-31) The guru did not like the answers so stopping VM[DomainRouter|r-11-VM]
> 2013-08-23 18:47:59,626 DEBUG [agent.transport.Request] (Job-Executor-11:job-31) Seq 1-585236524: Sending { Cmd , MgmtId: 166316981724, via: 1, Ver: v1, Flags: 100111, [{"StopCommand":{"isProxy":false,"vmName":"r-11-VM","wait":0}}] }
> 2013-08-23 18:47:59,627 DEBUG [agent.transport.Request] (Job-Executor-11:job-31) Seq 1-585236524: Executing: { Cmd , MgmtId: 166316981724, via: 1, Ver: v1, Flags: 100111, [{"StopCommand":{"isProxy":false,"vmName":"r-11-VM","wait":0}}] }
> 2013-08-23 18:47:59,627 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-22:null) Seq 1-585236524: Executing request
> 2013-08-23 18:47:59,774 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-22:null) 9. The VM r-11-VM is in Stopping state
> 2013-08-23 18:48:00,101 INFO [xen.resource.CitrixResourceBase] (DirectAgent-22:null) Removed network rules for vm r-11-VM
>
> Later it says it is in stopped state and that stopping succeeded:
> 2013-08-23 18:48:08,312 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-22:null) 10. The VM r-11-VM is in Stopped state
> 2013-08-23 18:48:08,313 DEBUG [agent.manager.AgentManagerImpl] (Job-Executor-11:job-31) Details from executing class com.cloud.agent.api.StopCommand: Stop VM r-11-VM Succeed
>
> Then it tells me the error was in finalizeStart:
> 2013-08-23 18:48:08,313 ERROR [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-11:job-31) Failed to start instance VM[DomainRouter|r-11-VM]
> com.cloud.utils.exception.ExecutionException: Unable to start VM[DomainRouter|r-11-VM] due to error in finalizeStart, not retrying
>
> Where can I see the finalizeStart method and the part of it that is failing?
>
> 2. I checked the /var/log/SMlog on the XenServer host and did not find any lines that contained "getDomRVersion" How would I know if any of the entries are related to the getDomRVersion script?
> I did notice that when I tried to start the VR then the SMlog showed a failure after running router_proxy.sh and get_template_version.sh (shown in the log below):
> [31327] 2013-08-24 16:14:20.346551 #### VMOPS enter routerProxy ####
> [31327] 2013-08-24 16:14:20.346675 ['/bin/bash', '/opt/xensource/bin/router_proxy.sh', 'get_template_version.sh', '169.254.3.168']
> [31327] 2013-08-24 16:14:20.464558 FAILED in util.pread: (rc 255) stdout: '', stderr: ''
> [31327] 2013-08-24 16:14:20.464766 routerProxy command get_template_version.sh 169.254.3.168 failed
> [31327] 2013-08-24 16:14:20.464872 #### VMOPS exit routerProxy ####
>
> 3. I checked for the /root/.ssh/id_rsa.cloud file on the XenServer host and it did exist.
>
> 4. I tried forcing reconnect through the UI and that did not solve the problem.
> 5. I tried unmanaging and re-managing the cluster (I had to put my host in maintenance mode first)
> 6. I tried clearing the host tags
>
> None of this solved my problem. I did see some entries in the SMlog that made me wonder if vhd-util needs to be in my primary or secondary storage directory. To your knowledge, does it need to be? I thought vhd-util only needed to be in /usr/bin and /opt/xensource/bin. Is there somewhere else it needs to be?
>
>
> I am starting to wonder if I should reinstall everything and use KVM with Libvirt. I really want to use Xen and CloudStack. This problem with my instances and the virtual router really is an impasse for my project.
>
> Thank you for your help, Kirk.
>
> Best,
>
> Kent Johnson
> University of Utah
> Graduate Student
> MSIS Program
> -----Original Message-----
> From: Kirk Kosinski [mailto:kirkkosinski@gmail.com]
> Sent: Friday, August 23, 2013 8:35 PM
> To: users@cloudstack.apache.org
> Cc: Ahmad Emneina
> Subject: Re: Cs 4.1.0 plus Xen 6.1 No Instances or VR's starting up
>
> Does the virtual router start on XenServer and stop after a few seconds or minutes? Or does it never start at all?
>
> Check /var/log/SMlog on the XenServer host for any entries or
> (especially) errors related to the getDomRVersion script. If there are no useful errors, find the script arguments in SMlog and try running it manually with bash -x to see where it is failing.
>
> One potential cause is a missing /root/.ssh/id_rsa.cloud on the XenServer host, so confirm it exists. Besides that, some general steps that might clear up the problem include:
> 1. Force reconnect the host (CS API or UI).
> 2. Unmanage and re-manage the cluster (CS API or UI).
> 3. Unmanage cluster / clear host tags (xe host-param-clear uuid=host_id
> param-name=tags) / re-manage cluster
>
> Best regards,
> Kirk
>
>
> On 08/23/2013 04:24 PM, Ahmad Emneina wrote:
>> looks like your management server reaches out to the router but barfs
>> out
>> here:
>> 2013-08-23 17:04:53,938 DEBUG [agent.transport.Request]
>> (Job-Executor-1:job-18) Seq 1-1363673129: Received: { Ans: , MgmtId:
>> 166316981724, via: 1, Ver: v1, Flags: 110, { StartAnswer,
>> CheckSshAnswer, GetDomRVersionAnswer } }
>> 2013-08-23 17:04:53,983 WARN
>> [network.router.VirtualNetworkApplianceManagerImpl]
>> (Job-Executor-1:job-18) Unable to get the template/scripts version of
>> router r-6-VM due to: getDomRVersionCmd failed
>> 2013-08-23 17:04:53,984 INFO [cloud.vm.VirtualMachineManagerImpl]
>> (Job-Executor-1:job-18) The guru did not like the answers so stopping
>> VM[DomainRouter|r-6-VM]
>> 2013-08-23 17:04:53,988 DEBUG [agent.transport.Request]
>> (Job-Executor-1:job-18) Seq 1-1363673130: Sending { Cmd , MgmtId:
>> 166316981724, via: 1, Ver: v1, Flags: 100111,
>> [{"StopCommand":{"isProxy":false,"vmName":"r-6-VM","wait":0}}] }
>>
>> so the good news is your network config looks good, i just wonder if
>> youre using the right template or if the system vm isnt patched properly.
>>
>>
>> On Fri, Aug 23, 2013 at 4:16 PM, Kent Johnson <ke...@utah.edu> wrote:
>>
>>> Thank you Marty and Ahmad.
>>>
>>> Here is my management-server.log log: http://pastebin.com/TnRpsB8j
>>>
>>> Here is my catalina.out log: http://pastebin.com/index
>>>
>>> Here is my xensource.log from my xenserver host:
>>> http://pastebin.com/RC6LhXQ4
>>>
>>> Best,
>>>
>>> Kent Johnson
>>>
>>> -----Original Message-----
>>> From: Ahmad Emneina [mailto:aemneina@gmail.com]
>>> Sent: Friday, August 23, 2013 5:00 PM
>>> To: Cloudstack users mailing list
>>> Subject: Re: Cs 4.1.0 plus Xen 6.1 No Instances or VR's starting up
>>>
>>> probably best to post your logs to pastebin and have us sift over them.
>>> see if we can spot anything.
>>>
>>>
>>> On Fri, Aug 23, 2013 at 3:53 PM, Kent Johnson <ke...@utah.edu>
>>> wrote:
>>>
>>>> Can anyone point me in the right direction on solving my instance
>>>> startup problem? They won't start up, nor will my virtual router.
>>>>
>>>> I am using CloudStack 4.1.0 on CentOS for my Management Server and
>>>> XenServer 6.1 for my VM host.
>>>>
>>>> I installed CS as per the documentation and have successfully added
>>>> a zone, pod, cluster, host, primary, and secondary storage.
>>>> My System VM's start up and run correctly. My systemVM template and
>>>> the default CentOS templates download correctly.
>>>> I can create instances but they always try to start up and then fail
>>>> and end up in "Error" state. My Virtual Router tries to start up But
>>>> it always fails.
>>>>
>>>> Are there any gotchas or quick suggestions anyone could give me to
>>>> help me understand how to get my instances to run and possibly my
>>> virtual router?
>>>> I can provide more details from the logs if needed.
>>>>
>>>> Kent Johnson
>>>> University of Utah
>>>> Graduate Student
>>
RE: Cs 4.1.0 plus Xen 6.1 No Instances or VR's starting up
Posted by Kent Johnson <ke...@utah.edu>.
Thank you Kirk.
As a preface to my response a virtual router is automatically created every time I create my first virtual machine. I have not yet successfully been able to add an instance. The virtual router created when I make an instance is the VR I am having problems with.
In answering your questions,
1. The virtual router goes into the starting state and never makes it to running. It errors out and then goes from starting state to stopping. I did see some suspicious lines on the log file:
2013-08-23 18:47:58,361 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-9:null) Trying to connect to 169.254.3.124
2013-08-23 18:47:59,178 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-9:null) Ping command port succeeded for vm r-11-VM
2013-08-23 18:47:59,561 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-9:null) Seq 1-585236521: Cancelling because one of the answers is false and it is stop on error.
Do you think the problem with my router not starting up is related to the "stop on error" part of those log lines? A few lines later in the log it tells me the guru did not like the answers so is stopping the router:
2013-08-23 18:47:59,623 INFO [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-11:job-31) The guru did not like the answers so stopping VM[DomainRouter|r-11-VM]
2013-08-23 18:47:59,626 DEBUG [agent.transport.Request] (Job-Executor-11:job-31) Seq 1-585236524: Sending { Cmd , MgmtId: 166316981724, via: 1, Ver: v1, Flags: 100111, [{"StopCommand":{"isProxy":false,"vmName":"r-11-VM","wait":0}}] }
2013-08-23 18:47:59,627 DEBUG [agent.transport.Request] (Job-Executor-11:job-31) Seq 1-585236524: Executing: { Cmd , MgmtId: 166316981724, via: 1, Ver: v1, Flags: 100111, [{"StopCommand":{"isProxy":false,"vmName":"r-11-VM","wait":0}}] }
2013-08-23 18:47:59,627 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-22:null) Seq 1-585236524: Executing request
2013-08-23 18:47:59,774 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-22:null) 9. The VM r-11-VM is in Stopping state
2013-08-23 18:48:00,101 INFO [xen.resource.CitrixResourceBase] (DirectAgent-22:null) Removed network rules for vm r-11-VM
Later it says it is in stopped state and that stopping succeeded:
2013-08-23 18:48:08,312 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-22:null) 10. The VM r-11-VM is in Stopped state
2013-08-23 18:48:08,313 DEBUG [agent.manager.AgentManagerImpl] (Job-Executor-11:job-31) Details from executing class com.cloud.agent.api.StopCommand: Stop VM r-11-VM Succeed
Then it tells me the error was in finalizeStart:
2013-08-23 18:48:08,313 ERROR [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-11:job-31) Failed to start instance VM[DomainRouter|r-11-VM]
com.cloud.utils.exception.ExecutionException: Unable to start VM[DomainRouter|r-11-VM] due to error in finalizeStart, not retrying
Where can I see the finalizeStart method and the part of it that is failing?
2. I checked the /var/log/SMlog on the XenServer host and did not find any lines that contained "getDomRVersion" How would I know if any of the entries are related to the getDomRVersion script?
I did notice that when I tried to start the VR then the SMlog showed a failure after running router_proxy.sh and get_template_version.sh (shown in the log below):
[31327] 2013-08-24 16:14:20.346551 #### VMOPS enter routerProxy ####
[31327] 2013-08-24 16:14:20.346675 ['/bin/bash', '/opt/xensource/bin/router_proxy.sh', 'get_template_version.sh', '169.254.3.168']
[31327] 2013-08-24 16:14:20.464558 FAILED in util.pread: (rc 255) stdout: '', stderr: ''
[31327] 2013-08-24 16:14:20.464766 routerProxy command get_template_version.sh 169.254.3.168 failed
[31327] 2013-08-24 16:14:20.464872 #### VMOPS exit routerProxy ####
3. I checked for the /root/.ssh/id_rsa.cloud file on the XenServer host and it did exist.
4. I tried forcing reconnect through the UI and that did not solve the problem.
5. I tried unmanaging and re-managing the cluster (I had to put my host in maintenance mode first)
6. I tried clearing the host tags
None of this solved my problem. I did see some entries in the SMlog that made me wonder if vhd-util needs to be in my primary or secondary storage directory. To your knowledge, does it need to be? I thought vhd-util only needed to be in /usr/bin and /opt/xensource/bin. Is there somewhere else it needs to be?
I am starting to wonder if I should reinstall everything and use KVM with Libvirt. I really want to use Xen and CloudStack. This problem with my instances and the virtual router really is an impasse for my project.
Thank you for your help, Kirk.
Best,
Kent Johnson
University of Utah
Graduate Student
MSIS Program
-----Original Message-----
From: Kirk Kosinski [mailto:kirkkosinski@gmail.com]
Sent: Friday, August 23, 2013 8:35 PM
To: users@cloudstack.apache.org
Cc: Ahmad Emneina
Subject: Re: Cs 4.1.0 plus Xen 6.1 No Instances or VR's starting up
Does the virtual router start on XenServer and stop after a few seconds or minutes? Or does it never start at all?
Check /var/log/SMlog on the XenServer host for any entries or
(especially) errors related to the getDomRVersion script. If there are no useful errors, find the script arguments in SMlog and try running it manually with bash -x to see where it is failing.
One potential cause is a missing /root/.ssh/id_rsa.cloud on the XenServer host, so confirm it exists. Besides that, some general steps that might clear up the problem include:
1. Force reconnect the host (CS API or UI).
2. Unmanage and re-manage the cluster (CS API or UI).
3. Unmanage cluster / clear host tags (xe host-param-clear uuid=host_id
param-name=tags) / re-manage cluster
Best regards,
Kirk
On 08/23/2013 04:24 PM, Ahmad Emneina wrote:
> looks like your management server reaches out to the router but barfs
> out
> here:
> 2013-08-23 17:04:53,938 DEBUG [agent.transport.Request]
> (Job-Executor-1:job-18) Seq 1-1363673129: Received: { Ans: , MgmtId:
> 166316981724, via: 1, Ver: v1, Flags: 110, { StartAnswer,
> CheckSshAnswer, GetDomRVersionAnswer } }
> 2013-08-23 17:04:53,983 WARN
> [network.router.VirtualNetworkApplianceManagerImpl]
> (Job-Executor-1:job-18) Unable to get the template/scripts version of
> router r-6-VM due to: getDomRVersionCmd failed
> 2013-08-23 17:04:53,984 INFO [cloud.vm.VirtualMachineManagerImpl]
> (Job-Executor-1:job-18) The guru did not like the answers so stopping
> VM[DomainRouter|r-6-VM]
> 2013-08-23 17:04:53,988 DEBUG [agent.transport.Request]
> (Job-Executor-1:job-18) Seq 1-1363673130: Sending { Cmd , MgmtId:
> 166316981724, via: 1, Ver: v1, Flags: 100111,
> [{"StopCommand":{"isProxy":false,"vmName":"r-6-VM","wait":0}}] }
>
> so the good news is your network config looks good, i just wonder if
> youre using the right template or if the system vm isnt patched properly.
>
>
> On Fri, Aug 23, 2013 at 4:16 PM, Kent Johnson <ke...@utah.edu> wrote:
>
>> Thank you Marty and Ahmad.
>>
>> Here is my management-server.log log: http://pastebin.com/TnRpsB8j
>>
>> Here is my catalina.out log: http://pastebin.com/index
>>
>> Here is my xensource.log from my xenserver host:
>> http://pastebin.com/RC6LhXQ4
>>
>> Best,
>>
>> Kent Johnson
>>
>> -----Original Message-----
>> From: Ahmad Emneina [mailto:aemneina@gmail.com]
>> Sent: Friday, August 23, 2013 5:00 PM
>> To: Cloudstack users mailing list
>> Subject: Re: Cs 4.1.0 plus Xen 6.1 No Instances or VR's starting up
>>
>> probably best to post your logs to pastebin and have us sift over them.
>> see if we can spot anything.
>>
>>
>> On Fri, Aug 23, 2013 at 3:53 PM, Kent Johnson <ke...@utah.edu>
>> wrote:
>>
>>> Can anyone point me in the right direction on solving my instance
>>> startup problem? They won't start up, nor will my virtual router.
>>>
>>> I am using CloudStack 4.1.0 on CentOS for my Management Server and
>>> XenServer 6.1 for my VM host.
>>>
>>> I installed CS as per the documentation and have successfully added
>>> a zone, pod, cluster, host, primary, and secondary storage.
>>> My System VM's start up and run correctly. My systemVM template and
>>> the default CentOS templates download correctly.
>>> I can create instances but they always try to start up and then fail
>>> and end up in "Error" state. My Virtual Router tries to start up But
>>> it always fails.
>>>
>>> Are there any gotchas or quick suggestions anyone could give me to
>>> help me understand how to get my instances to run and possibly my
>> virtual router?
>>> I can provide more details from the logs if needed.
>>>
>>> Kent Johnson
>>> University of Utah
>>> Graduate Student
>>>
>>>
>>
>
Re: Cs 4.1.0 plus Xen 6.1 No Instances or VR's starting up
Posted by Kirk Kosinski <ki...@gmail.com>.
Does the virtual router start on XenServer and stop after a few seconds
or minutes? Or does it never start at all?
Check /var/log/SMlog on the XenServer host for any entries or
(especially) errors related to the getDomRVersion script. If there are
no useful errors, find the script arguments in SMlog and try running it
manually with bash -x to see where it is failing.
One potential cause is a missing /root/.ssh/id_rsa.cloud on the
XenServer host, so confirm it exists. Besides that, some general steps
that might clear up the problem include:
1. Force reconnect the host (CS API or UI).
2. Unmanage and re-manage the cluster (CS API or UI).
3. Unmanage cluster / clear host tags (xe host-param-clear uuid=host_id
param-name=tags) / re-manage cluster
Best regards,
Kirk
On 08/23/2013 04:24 PM, Ahmad Emneina wrote:
> looks like your management server reaches out to the router but barfs out
> here:
> 2013-08-23 17:04:53,938 DEBUG [agent.transport.Request]
> (Job-Executor-1:job-18) Seq 1-1363673129: Received: { Ans: , MgmtId:
> 166316981724, via: 1, Ver: v1, Flags: 110, { StartAnswer, CheckSshAnswer,
> GetDomRVersionAnswer } }
> 2013-08-23 17:04:53,983 WARN
> [network.router.VirtualNetworkApplianceManagerImpl]
> (Job-Executor-1:job-18) Unable to get the template/scripts version of
> router r-6-VM due to: getDomRVersionCmd failed
> 2013-08-23 17:04:53,984 INFO [cloud.vm.VirtualMachineManagerImpl]
> (Job-Executor-1:job-18) The guru did not like the answers so stopping
> VM[DomainRouter|r-6-VM]
> 2013-08-23 17:04:53,988 DEBUG [agent.transport.Request]
> (Job-Executor-1:job-18) Seq 1-1363673130: Sending { Cmd , MgmtId:
> 166316981724, via: 1, Ver: v1, Flags: 100111,
> [{"StopCommand":{"isProxy":false,"vmName":"r-6-VM","wait":0}}] }
>
> so the good news is your network config looks good, i just wonder if youre
> using the right template or if the system vm isnt patched properly.
>
>
> On Fri, Aug 23, 2013 at 4:16 PM, Kent Johnson <ke...@utah.edu> wrote:
>
>> Thank you Marty and Ahmad.
>>
>> Here is my management-server.log log: http://pastebin.com/TnRpsB8j
>>
>> Here is my catalina.out log: http://pastebin.com/index
>>
>> Here is my xensource.log from my xenserver host:
>> http://pastebin.com/RC6LhXQ4
>>
>> Best,
>>
>> Kent Johnson
>>
>> -----Original Message-----
>> From: Ahmad Emneina [mailto:aemneina@gmail.com]
>> Sent: Friday, August 23, 2013 5:00 PM
>> To: Cloudstack users mailing list
>> Subject: Re: Cs 4.1.0 plus Xen 6.1 No Instances or VR's starting up
>>
>> probably best to post your logs to pastebin and have us sift over them.
>> see if we can spot anything.
>>
>>
>> On Fri, Aug 23, 2013 at 3:53 PM, Kent Johnson <ke...@utah.edu>
>> wrote:
>>
>>> Can anyone point me in the right direction on solving my instance
>>> startup problem? They won't start up, nor will my virtual router.
>>>
>>> I am using CloudStack 4.1.0 on CentOS for my Management Server and
>>> XenServer 6.1 for my VM host.
>>>
>>> I installed CS as per the documentation and have successfully added a
>>> zone, pod, cluster, host, primary, and secondary storage.
>>> My System VM's start up and run correctly. My systemVM template and
>>> the default CentOS templates download correctly.
>>> I can create instances but they always try to start up and then fail
>>> and end up in "Error" state. My Virtual Router tries to start up But
>>> it always fails.
>>>
>>> Are there any gotchas or quick suggestions anyone could give me to
>>> help me understand how to get my instances to run and possibly my
>> virtual router?
>>> I can provide more details from the logs if needed.
>>>
>>> Kent Johnson
>>> University of Utah
>>> Graduate Student
>>>
>>>
>>
>
RE: Cs 4.1.0 plus Xen 6.1 No Instances or VR's starting up
Posted by Kent Johnson <ke...@utah.edu>.
Iptables is disabled, selinux is disabled also.
In the catalina.out it says "VM does not exist on XenServer322ab8aa-0302-49b2-9f32-f068f7f2f8f8
INFO [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-6:job-23) Unable to contact resource.
Does that mean the management server is looking for the virtual router on the xenserver host and is not finding it there?
Kent
-----Original Message-----
From: creategui@gmail.com [mailto:creategui@gmail.com] On Behalf Of Carlos Reategui
Sent: Friday, August 23, 2013 6:07 PM
To: users@cloudstack.apache.org
Subject: Re: Cs 4.1.0 plus Xen 6.1 No Instances or VR's starting up
try disabling iptables on your hosts (service iptables stop) and see if that makes a difference.
On Fri, Aug 23, 2013 at 4:31 PM, Kent Johnson <ke...@utah.edu> wrote:
> Ok, I don't remember patching the system VM. Do I need to download a
> newer version or something like that?
>
> Kent Johnson
>
> -----Original Message-----
> From: Ahmad Emneina [mailto:aemneina@gmail.com]
> Sent: Friday, August 23, 2013 5:24 PM
> To: Cloudstack users mailing list
> Subject: Re: Cs 4.1.0 plus Xen 6.1 No Instances or VR's starting up
>
> looks like your management server reaches out to the router but barfs
> out
> here:
> 2013-08-23 17:04:53,938 DEBUG [agent.transport.Request]
> (Job-Executor-1:job-18) Seq 1-1363673129: Received: { Ans: , MgmtId:
> 166316981724, via: 1, Ver: v1, Flags: 110, { StartAnswer,
> CheckSshAnswer, GetDomRVersionAnswer } }
> 2013-08-23 17:04:53,983 WARN
> [network.router.VirtualNetworkApplianceManagerImpl]
> (Job-Executor-1:job-18) Unable to get the template/scripts version of
> router r-6-VM due to: getDomRVersionCmd failed
> 2013-08-23 17:04:53,984 INFO [cloud.vm.VirtualMachineManagerImpl]
> (Job-Executor-1:job-18) The guru did not like the answers so stopping
> VM[DomainRouter|r-6-VM]
> 2013-08-23 17:04:53,988 DEBUG [agent.transport.Request]
> (Job-Executor-1:job-18) Seq 1-1363673130: Sending { Cmd , MgmtId:
> 166316981724, via: 1, Ver: v1, Flags: 100111,
> [{"StopCommand":{"isProxy":false,"vmName":"r-6-VM","wait":0}}] }
>
> so the good news is your network config looks good, i just wonder if
> youre using the right template or if the system vm isnt patched properly.
>
>
> On Fri, Aug 23, 2013 at 4:16 PM, Kent Johnson <ke...@utah.edu>
> wrote:
>
> > Thank you Marty and Ahmad.
> >
> > Here is my management-server.log log: http://pastebin.com/TnRpsB8j
> >
> > Here is my catalina.out log: http://pastebin.com/index
> >
> > Here is my xensource.log from my xenserver host:
> > http://pastebin.com/RC6LhXQ4
> >
> > Best,
> >
> > Kent Johnson
> >
> > -----Original Message-----
> > From: Ahmad Emneina [mailto:aemneina@gmail.com]
> > Sent: Friday, August 23, 2013 5:00 PM
> > To: Cloudstack users mailing list
> > Subject: Re: Cs 4.1.0 plus Xen 6.1 No Instances or VR's starting up
> >
> > probably best to post your logs to pastebin and have us sift over them.
> > see if we can spot anything.
> >
> >
> > On Fri, Aug 23, 2013 at 3:53 PM, Kent Johnson
> > <ke...@utah.edu>
> > wrote:
> >
> > > Can anyone point me in the right direction on solving my instance
> > > startup problem? They won't start up, nor will my virtual router.
> > >
> > > I am using CloudStack 4.1.0 on CentOS for my Management Server and
> > > XenServer 6.1 for my VM host.
> > >
> > > I installed CS as per the documentation and have successfully
> > > added a zone, pod, cluster, host, primary, and secondary storage.
> > > My System VM's start up and run correctly. My systemVM template
> > > and the default CentOS templates download correctly.
> > > I can create instances but they always try to start up and then
> > > fail and end up in "Error" state. My Virtual Router tries to start
> > > up But it always fails.
> > >
> > > Are there any gotchas or quick suggestions anyone could give me to
> > > help me understand how to get my instances to run and possibly my
> > virtual router?
> > > I can provide more details from the logs if needed.
> > >
> > > Kent Johnson
> > > University of Utah
> > > Graduate Student
> > >
> > >
> >
>
Re: Cs 4.1.0 plus Xen 6.1 No Instances or VR's starting up
Posted by Carlos Reategui <ca...@reategui.com>.
try disabling iptables on your hosts (service iptables stop) and see if
that makes a difference.
On Fri, Aug 23, 2013 at 4:31 PM, Kent Johnson <ke...@utah.edu> wrote:
> Ok, I don't remember patching the system VM. Do I need to download a newer
> version or something like that?
>
> Kent Johnson
>
> -----Original Message-----
> From: Ahmad Emneina [mailto:aemneina@gmail.com]
> Sent: Friday, August 23, 2013 5:24 PM
> To: Cloudstack users mailing list
> Subject: Re: Cs 4.1.0 plus Xen 6.1 No Instances or VR's starting up
>
> looks like your management server reaches out to the router but barfs out
> here:
> 2013-08-23 17:04:53,938 DEBUG [agent.transport.Request]
> (Job-Executor-1:job-18) Seq 1-1363673129: Received: { Ans: , MgmtId:
> 166316981724, via: 1, Ver: v1, Flags: 110, { StartAnswer, CheckSshAnswer,
> GetDomRVersionAnswer } }
> 2013-08-23 17:04:53,983 WARN
> [network.router.VirtualNetworkApplianceManagerImpl]
> (Job-Executor-1:job-18) Unable to get the template/scripts version of
> router r-6-VM due to: getDomRVersionCmd failed
> 2013-08-23 17:04:53,984 INFO [cloud.vm.VirtualMachineManagerImpl]
> (Job-Executor-1:job-18) The guru did not like the answers so stopping
> VM[DomainRouter|r-6-VM]
> 2013-08-23 17:04:53,988 DEBUG [agent.transport.Request]
> (Job-Executor-1:job-18) Seq 1-1363673130: Sending { Cmd , MgmtId:
> 166316981724, via: 1, Ver: v1, Flags: 100111,
> [{"StopCommand":{"isProxy":false,"vmName":"r-6-VM","wait":0}}] }
>
> so the good news is your network config looks good, i just wonder if youre
> using the right template or if the system vm isnt patched properly.
>
>
> On Fri, Aug 23, 2013 at 4:16 PM, Kent Johnson <ke...@utah.edu>
> wrote:
>
> > Thank you Marty and Ahmad.
> >
> > Here is my management-server.log log: http://pastebin.com/TnRpsB8j
> >
> > Here is my catalina.out log: http://pastebin.com/index
> >
> > Here is my xensource.log from my xenserver host:
> > http://pastebin.com/RC6LhXQ4
> >
> > Best,
> >
> > Kent Johnson
> >
> > -----Original Message-----
> > From: Ahmad Emneina [mailto:aemneina@gmail.com]
> > Sent: Friday, August 23, 2013 5:00 PM
> > To: Cloudstack users mailing list
> > Subject: Re: Cs 4.1.0 plus Xen 6.1 No Instances or VR's starting up
> >
> > probably best to post your logs to pastebin and have us sift over them.
> > see if we can spot anything.
> >
> >
> > On Fri, Aug 23, 2013 at 3:53 PM, Kent Johnson <ke...@utah.edu>
> > wrote:
> >
> > > Can anyone point me in the right direction on solving my instance
> > > startup problem? They won't start up, nor will my virtual router.
> > >
> > > I am using CloudStack 4.1.0 on CentOS for my Management Server and
> > > XenServer 6.1 for my VM host.
> > >
> > > I installed CS as per the documentation and have successfully added
> > > a zone, pod, cluster, host, primary, and secondary storage.
> > > My System VM's start up and run correctly. My systemVM template and
> > > the default CentOS templates download correctly.
> > > I can create instances but they always try to start up and then fail
> > > and end up in "Error" state. My Virtual Router tries to start up But
> > > it always fails.
> > >
> > > Are there any gotchas or quick suggestions anyone could give me to
> > > help me understand how to get my instances to run and possibly my
> > virtual router?
> > > I can provide more details from the logs if needed.
> > >
> > > Kent Johnson
> > > University of Utah
> > > Graduate Student
> > >
> > >
> >
>
RE: Cs 4.1.0 plus Xen 6.1 No Instances or VR's starting up
Posted by Kent Johnson <ke...@utah.edu>.
Ok, I don't remember patching the system VM. Do I need to download a newer version or something like that?
Kent Johnson
-----Original Message-----
From: Ahmad Emneina [mailto:aemneina@gmail.com]
Sent: Friday, August 23, 2013 5:24 PM
To: Cloudstack users mailing list
Subject: Re: Cs 4.1.0 plus Xen 6.1 No Instances or VR's starting up
looks like your management server reaches out to the router but barfs out
here:
2013-08-23 17:04:53,938 DEBUG [agent.transport.Request]
(Job-Executor-1:job-18) Seq 1-1363673129: Received: { Ans: , MgmtId:
166316981724, via: 1, Ver: v1, Flags: 110, { StartAnswer, CheckSshAnswer, GetDomRVersionAnswer } }
2013-08-23 17:04:53,983 WARN
[network.router.VirtualNetworkApplianceManagerImpl]
(Job-Executor-1:job-18) Unable to get the template/scripts version of router r-6-VM due to: getDomRVersionCmd failed
2013-08-23 17:04:53,984 INFO [cloud.vm.VirtualMachineManagerImpl]
(Job-Executor-1:job-18) The guru did not like the answers so stopping VM[DomainRouter|r-6-VM]
2013-08-23 17:04:53,988 DEBUG [agent.transport.Request]
(Job-Executor-1:job-18) Seq 1-1363673130: Sending { Cmd , MgmtId:
166316981724, via: 1, Ver: v1, Flags: 100111, [{"StopCommand":{"isProxy":false,"vmName":"r-6-VM","wait":0}}] }
so the good news is your network config looks good, i just wonder if youre using the right template or if the system vm isnt patched properly.
On Fri, Aug 23, 2013 at 4:16 PM, Kent Johnson <ke...@utah.edu> wrote:
> Thank you Marty and Ahmad.
>
> Here is my management-server.log log: http://pastebin.com/TnRpsB8j
>
> Here is my catalina.out log: http://pastebin.com/index
>
> Here is my xensource.log from my xenserver host:
> http://pastebin.com/RC6LhXQ4
>
> Best,
>
> Kent Johnson
>
> -----Original Message-----
> From: Ahmad Emneina [mailto:aemneina@gmail.com]
> Sent: Friday, August 23, 2013 5:00 PM
> To: Cloudstack users mailing list
> Subject: Re: Cs 4.1.0 plus Xen 6.1 No Instances or VR's starting up
>
> probably best to post your logs to pastebin and have us sift over them.
> see if we can spot anything.
>
>
> On Fri, Aug 23, 2013 at 3:53 PM, Kent Johnson <ke...@utah.edu>
> wrote:
>
> > Can anyone point me in the right direction on solving my instance
> > startup problem? They won't start up, nor will my virtual router.
> >
> > I am using CloudStack 4.1.0 on CentOS for my Management Server and
> > XenServer 6.1 for my VM host.
> >
> > I installed CS as per the documentation and have successfully added
> > a zone, pod, cluster, host, primary, and secondary storage.
> > My System VM's start up and run correctly. My systemVM template and
> > the default CentOS templates download correctly.
> > I can create instances but they always try to start up and then fail
> > and end up in "Error" state. My Virtual Router tries to start up But
> > it always fails.
> >
> > Are there any gotchas or quick suggestions anyone could give me to
> > help me understand how to get my instances to run and possibly my
> virtual router?
> > I can provide more details from the logs if needed.
> >
> > Kent Johnson
> > University of Utah
> > Graduate Student
> >
> >
>
Re: Cs 4.1.0 plus Xen 6.1 No Instances or VR's starting up
Posted by Ahmad Emneina <ae...@gmail.com>.
looks like your management server reaches out to the router but barfs out
here:
2013-08-23 17:04:53,938 DEBUG [agent.transport.Request]
(Job-Executor-1:job-18) Seq 1-1363673129: Received: { Ans: , MgmtId:
166316981724, via: 1, Ver: v1, Flags: 110, { StartAnswer, CheckSshAnswer,
GetDomRVersionAnswer } }
2013-08-23 17:04:53,983 WARN
[network.router.VirtualNetworkApplianceManagerImpl]
(Job-Executor-1:job-18) Unable to get the template/scripts version of
router r-6-VM due to: getDomRVersionCmd failed
2013-08-23 17:04:53,984 INFO [cloud.vm.VirtualMachineManagerImpl]
(Job-Executor-1:job-18) The guru did not like the answers so stopping
VM[DomainRouter|r-6-VM]
2013-08-23 17:04:53,988 DEBUG [agent.transport.Request]
(Job-Executor-1:job-18) Seq 1-1363673130: Sending { Cmd , MgmtId:
166316981724, via: 1, Ver: v1, Flags: 100111,
[{"StopCommand":{"isProxy":false,"vmName":"r-6-VM","wait":0}}] }
so the good news is your network config looks good, i just wonder if youre
using the right template or if the system vm isnt patched properly.
On Fri, Aug 23, 2013 at 4:16 PM, Kent Johnson <ke...@utah.edu> wrote:
> Thank you Marty and Ahmad.
>
> Here is my management-server.log log: http://pastebin.com/TnRpsB8j
>
> Here is my catalina.out log: http://pastebin.com/index
>
> Here is my xensource.log from my xenserver host:
> http://pastebin.com/RC6LhXQ4
>
> Best,
>
> Kent Johnson
>
> -----Original Message-----
> From: Ahmad Emneina [mailto:aemneina@gmail.com]
> Sent: Friday, August 23, 2013 5:00 PM
> To: Cloudstack users mailing list
> Subject: Re: Cs 4.1.0 plus Xen 6.1 No Instances or VR's starting up
>
> probably best to post your logs to pastebin and have us sift over them.
> see if we can spot anything.
>
>
> On Fri, Aug 23, 2013 at 3:53 PM, Kent Johnson <ke...@utah.edu>
> wrote:
>
> > Can anyone point me in the right direction on solving my instance
> > startup problem? They won't start up, nor will my virtual router.
> >
> > I am using CloudStack 4.1.0 on CentOS for my Management Server and
> > XenServer 6.1 for my VM host.
> >
> > I installed CS as per the documentation and have successfully added a
> > zone, pod, cluster, host, primary, and secondary storage.
> > My System VM's start up and run correctly. My systemVM template and
> > the default CentOS templates download correctly.
> > I can create instances but they always try to start up and then fail
> > and end up in "Error" state. My Virtual Router tries to start up But
> > it always fails.
> >
> > Are there any gotchas or quick suggestions anyone could give me to
> > help me understand how to get my instances to run and possibly my
> virtual router?
> > I can provide more details from the logs if needed.
> >
> > Kent Johnson
> > University of Utah
> > Graduate Student
> >
> >
>
RE: Cs 4.1.0 plus Xen 6.1 No Instances or VR's starting up
Posted by Kent Johnson <ke...@utah.edu>.
Thank you Marty and Ahmad.
Here is my management-server.log log: http://pastebin.com/TnRpsB8j
Here is my catalina.out log: http://pastebin.com/index
Here is my xensource.log from my xenserver host: http://pastebin.com/RC6LhXQ4
Best,
Kent Johnson
-----Original Message-----
From: Ahmad Emneina [mailto:aemneina@gmail.com]
Sent: Friday, August 23, 2013 5:00 PM
To: Cloudstack users mailing list
Subject: Re: Cs 4.1.0 plus Xen 6.1 No Instances or VR's starting up
probably best to post your logs to pastebin and have us sift over them. see if we can spot anything.
On Fri, Aug 23, 2013 at 3:53 PM, Kent Johnson <ke...@utah.edu> wrote:
> Can anyone point me in the right direction on solving my instance
> startup problem? They won't start up, nor will my virtual router.
>
> I am using CloudStack 4.1.0 on CentOS for my Management Server and
> XenServer 6.1 for my VM host.
>
> I installed CS as per the documentation and have successfully added a
> zone, pod, cluster, host, primary, and secondary storage.
> My System VM's start up and run correctly. My systemVM template and
> the default CentOS templates download correctly.
> I can create instances but they always try to start up and then fail
> and end up in "Error" state. My Virtual Router tries to start up But
> it always fails.
>
> Are there any gotchas or quick suggestions anyone could give me to
> help me understand how to get my instances to run and possibly my virtual router?
> I can provide more details from the logs if needed.
>
> Kent Johnson
> University of Utah
> Graduate Student
>
>
Re: Cs 4.1.0 plus Xen 6.1 No Instances or VR's starting up
Posted by Ahmad Emneina <ae...@gmail.com>.
probably best to post your logs to pastebin and have us sift over them. see
if we can spot anything.
On Fri, Aug 23, 2013 at 3:53 PM, Kent Johnson <ke...@utah.edu> wrote:
> Can anyone point me in the right direction on solving my instance startup
> problem? They won't start up, nor will my virtual router.
>
> I am using CloudStack 4.1.0 on CentOS for my Management Server and
> XenServer 6.1 for my VM host.
>
> I installed CS as per the documentation and have successfully added a
> zone, pod, cluster, host, primary, and secondary storage.
> My System VM's start up and run correctly. My systemVM template and the
> default CentOS templates download correctly.
> I can create instances but they always try to start up and then fail and
> end up in "Error" state. My Virtual Router tries to start up
> But it always fails.
>
> Are there any gotchas or quick suggestions anyone could give me to help me
> understand how to get my instances to run and possibly my virtual router?
> I can provide more details from the logs if needed.
>
> Kent Johnson
> University of Utah
> Graduate Student
>
>
Re: Cs 4.1.0 plus Xen 6.1 No Instances or VR's starting up
Posted by Marty Sweet <ms...@gmail.com>.
Could you provide the latest management server log, it usually provides a
lot of information useful for debugging these kinds of issues.
Thanks,
Marty
On Fri, Aug 23, 2013 at 11:53 PM, Kent Johnson <ke...@utah.edu>wrote:
> Can anyone point me in the right direction on solving my instance startup
> problem? They won't start up, nor will my virtual router.
>
> I am using CloudStack 4.1.0 on CentOS for my Management Server and
> XenServer 6.1 for my VM host.
>
> I installed CS as per the documentation and have successfully added a
> zone, pod, cluster, host, primary, and secondary storage.
> My System VM's start up and run correctly. My systemVM template and the
> default CentOS templates download correctly.
> I can create instances but they always try to start up and then fail and
> end up in "Error" state. My Virtual Router tries to start up
> But it always fails.
>
> Are there any gotchas or quick suggestions anyone could give me to help me
> understand how to get my instances to run and possibly my virtual router?
> I can provide more details from the logs if needed.
>
> Kent Johnson
> University of Utah
> Graduate Student
>
>