You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Christoffer Pedersen <vr...@vrod.dk> on 2018/06/12 10:30:48 UTC
SSVM's not starting, timeout for libvirt python script in agent.log
Hi all,
I have an issue regarding the system VMs. After deploying an advanced zone,
the system VMs are trying to be created but gets stuck in a "Starting"
state, however the Agent state is "Up". I have these logs in the agent.log
(sorry for the formatting)
2018-06-12 12:22:06,354 WARN [kvm.resource.LibvirtComputingResource]
(Script-8:null) (logid:) Interrupting script.
2018-06-12 12:22:06,355 WARN [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-4:null) (logid:ea9cb55a) Timed out:
/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.py -n
v-1-VM -p
%template=domP%type=consoleproxy%host=1.1.1.1%port=8250%name=v-1-VM%zone=1%pod=1%guid=Proxy.1%proxy_vm=1%disable_rp_filter=true%eth2ip=9.9.9.9%eth2mask=255.255.255.0%gateway=9.9.9.1%eth0ip=169.254.3.159%eth0mask=255.255.0.0%eth1ip=6.6.6.6%eth1mask=255.255.255.0%mgmtcidr=
9.9.9.0/24%localgw=1.2.3.4%internaldns1=1.2.3.4%dns1=8.8.8.8%dns2=8.8.4.4
. Output is:
2018-06-12 12:22:06,355 ERROR [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-4:null) (logid:ea9cb55a) passcmd failed:timeout
2018-06-12 12:22:08,914 WARN [kvm.resource.LibvirtComputingResource]
(Script-4:null) (logid:) Interrupting script.
2018-06-12 12:22:08,915 WARN [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-5:null) (logid:8e44093e) Timed out:
/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.py -n
s-2-VM -p
%template=domP%type=secstorage%host=1.1.1.1%port=8250%name=s-2-VM%zone=1%pod=1%guid=s-2-VM%workers=5%resource=org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource%instance=SecStorage%sslcopy=false%role=templateProcessor%mtu=1500%eth2ip=7.7.7.7%eth2mask=255.255.255.0%gateway=9.9.9.1%public.network.device=eth2%eth0ip=169.254.2.193%eth0mask=255.255.0.0%eth1ip=10.120.0.61%eth1mask=255.255.255.0%mgmtcidr=
9.9.9.0/24%localgw=1.2.3.4%private.network.device=eth1%internaldns1=1.2.3.4%dns1=8.8.8.8%dns2=8.8.4.4%nfsVersion=null
. Output is:
2018-06-12 12:22:08,915 ERROR [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-5:null) (logid:8e44093e) passcmd failed:timeout
I have seen this error around but did not really find a solution to it. I
am not exactly sure whats "timing" out? I can ping both SSVM's on their
private and public interface.
I hope someone can help me out here. :)
--
Thanks,
Chris pedersen
Re: SSVM's not starting, timeout for libvirt python
script in agent.log
Posted by Dag Sonstebo <Da...@shapeblue.com>.
Hi Chris,
- Network menu > select view pulldown box > VPN users
=>>> You mean under "Select view:" (which defaults to Guest Network),
right? There I am only able to see "Guest Network", "VPC" and "VPN Customer
Gateway"
=>>> yes this is where it should be if you have enabled VPN on the source NAT. As above – which CloudStack version is this? I’ve tested this on 4.11.1RC2 and seems fine for both isolated networks and VPC networks.
- I have just tested creating a new project as a normal user, change this
on the pulldown project menu and clicked on the “network” menu item, no
error in RC2 – is this where you found it?
=>>> I just tried again with a user i created, it's an admin though. I
still get the message.
=>>> Can you put a screen dump somewhere? (attachments don’t work on the mailing list)
- Minutes yes, hours no. Going by your previous posts I would guess you
still have some tuning up to do and possibly run on low spec hardware
though?
=>>> The 2 hosts I have run with 2x E5-2680's and 64GB of ram. The
management server itself is a VMware hosted VM with 16gb of memory and
4vCPU's (E5-2650). It could be the tuning I guess, I am currently at a loss
here. The storage is a FreeNAS based NFS share but it's pretty fast.
=>>> This should be sufficient, and I would not expect it to take hours for a VPC to come up. Do keep in mind the first rollout of any template to a previously un-used primary storage pool will take longer as the template has to be copied from secondary storage to primary. Subsequent rollouts should be faster though. What to do is spin up a new VPC, ssh onto this once it’s up and tail the /var/log/cloud.log to determine how long it does processing, then watch how long it take for the VR to come online in ACS. Also check that your “host” global setting is correct.
Regards,
Dag Sonstebo
Cloud Architect
ShapeBlue
On 13/06/2018, 18:12, "Christoffer Pedersen" <vr...@vrod.dk> wrote:
Hi Dag,
- Network menu > select view pulldown box > VPN users
=>>> You mean under "Select view:" (which defaults to Guest Network),
right? There I am only able to see "Guest Network", "VPC" and "VPN Customer
Gateway"
- I have just tested creating a new project as a normal user, change this
on the pulldown project menu and clicked on the “network” menu item, no
error in RC2 – is this where you found it?
=>>> I just tried again with a user i created, it's an admin though. I
still get the message.
- This is by design, for VPCs you have to add additional IPs to configure
services. The only service available on the primary VPC IP is VPN.
=>>> Ok thanks, good to know!
- Minutes yes, hours no. Going by your previous posts I would guess you
still have some tuning up to do and possibly run on low spec hardware
though?
=>>> The 2 hosts I have run with 2x E5-2680's and 64GB of ram. The
management server itself is a VMware hosted VM with 16gb of memory and
4vCPU's (E5-2650). It could be the tuning I guess, I am currently at a loss
here. The storage is a FreeNAS based NFS share but it's pretty fast.
On Wed, Jun 13, 2018 at 5:54 PM, Dag Sonstebo <Da...@shapeblue.com>
wrote:
> Hi Chris,
>
> - I have enabled remote VPN in a VPC but I cannot seem to add users
> anywhere. On the VPN tab where the PSK is displayed it says: "Note: VPN
> users are now accessed by changing views at the networks tab". However, I
> cannot find out where such view would be? The documentation doesn't state
> anything about this.
>
> =>>> Network menu > select view pulldown box > VPN users
>
> - When I create a new project, choose the view from that project and go to
> "Network", I always get this message: " Account and projectId can't be
> specified together". What is it exactly and how can I get rid of it?
>
> =>>> I have just tested creating a new project as a normal user, change
> this on the pulldown project menu and clicked on the “network” menu item,
> no error in RC2 – is this where you found it?
>
> - I don't seem to be able to do port-forwarding with the first configured
> IP for the VPC. However when adding another IP, I can do it on that
> specific IP. Isn't it possible to do forwarding on the first assigned IP?
>
> =>>> This is by design, for VPCs you have to add additional IPs to
> configure services. The only service available on the primary VPC IP is VPN.
>
> - Is it normal that it takes minutes, sometimes hours to create the VPC? It
> really takes a while for the system to get going once I have created a VPC.
>
> =>>> Minutes yes, hours no. Going by your previous posts I would guess you
> still have some tuning up to do and possibly run on low spec hardware
> though?
>
>
> Regards,
> Dag Sonstebo
> Cloud Architect
> ShapeBlue
>
> On 13/06/2018, 16:25, "Christoffer Pedersen" <vr...@vrod.dk> wrote:
>
> The routers managed to start after a while. Looks like most things are
> OK
> now, however I hope some of you could answer some questions for me.
>
> - I have enabled remote VPN in a VPC but I cannot seem to add users
> anywhere. On the VPN tab where the PSK is displayed it says: "Note: VPN
> users are now accessed by changing views at the networks tab".
> However, I
> cannot find out where such view would be? The documentation doesn't
> state
> anything about this.
> - When I create a new project, choose the view from that project and
> go to
> "Network", I always get this message: " Account and projectId can't be
> specified together". What is it exactly and how can I get rid of it?
> - I don't seem to be able to do port-forwarding with the first
> configured
> IP for the VPC. However when adding another IP, I can do it on that
> specific IP. Isn't it possible to do forwarding on the first assigned
> IP?
> - Is it normal that it takes minutes, sometimes hours to create the
> VPC? It
> really takes a while for the system to get going once I have created a
> VPC.
>
> Perhaps this is for another email-thread, please let me know if so.
>
> Apart from that it seems like the cloudstack instance is finally
> working,
> first VM booted up!
>
> Thank you!
> Chris
>
>
> Dag.Sonstebo@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London WC2N 4HSUK
> @shapeblue
>
>
>
Dag.Sonstebo@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
> On Wed, Jun 13, 2018 at 4:15 PM, Nicolas Bouige <n....@dimsi.fr> wrote:
>
> > Hello All,
> >
> > Maybe is not related to your case but in case of....as we don't use
> the
> > same OS
> >
> > I was facing the exact same error during a new creation of virtual
> router.
> > The only change i made was to update my qemu version on our KVM
> (centOS7 +
> > CS 4.11) from 1.5 to 2.3
> > Before that VR creation was working fine.
> >
> > Apperently some users got the same issue but with an old version of
> CS
> > (4.9)
> > http://dev.cloudstack.apache.narkive.com/qsKmnsa4/
> > patchviasocket-seems-to-be-broken-with-qemu-2-3
> >
> >
> >
> > -----Message d'origine-----
> > De : Christoffer Pedersen [mailto:vrod@vrod.dk]
> > Envoyé : mardi 12 juin 2018 19:17
> > À : users@cloudstack.apache.org
> > Objet : Re: SSVM's not starting, timeout for libvirt python script in
> > agent.log
> >
> > Hi James,
> >
> > I looked at the value and it's already on 50GB. I looked a little
> more and
> > it seems like after restarting the cloudstack-management service I
> could no
> > longer reach the console or storage system vm.... looking at the
> host, now
> > the cloud0 interface is gone. I am using OVS, but according to a
> Shapeblue
> > article, no extra config is needed for the cloud0 interface.
> >
> >
> >
> > On Tue, Jun 12, 2018 at 6:17 PM, McClune, James <
> > mcclunej@norwalktruckers.net> wrote:
> >
> > > Hi Christoffer,
> > >
> > > Did you try setting the max.template.iso.size to a higher value
> (e.g.
> > > 50GB)?
> > >
> > > I think the default is set pretty low.
> > >
> > > Best,
> > > James
> > >
> > > On Tue, Jun 12, 2018 at 11:10 AM, Christoffer Pedersen <
> vrod@vrod.dk>
> > > wrote:
> > >
> > > > Just want to update that I still get the ISO error:
> > > >
> > > > 2018-06-12 17:09:42,194 INFO [c.c.t.HypervisorTemplateAdapter]
> > > > (qtp1096283470-12:ctx-962ec49c ctx-2c82c0e2) (logid:0c7df1d7)
> Image
> > > > store doesn't have enough capacity. Skip downloading template to
> > > > this image
> > > store
> > > > 1
> > > >
> > > > Thanks!
> > > >
> > > > On Tue, Jun 12, 2018 at 5:07 PM, Christoffer Pedersen <
> vrod@vrod.dk>
> > > > wrote:
> > > >
> > > >> Hi Dag,
> > > >>
> > > >> Yes, I altered the IP addresses as I do not fancy throwing them
> out
> > > >> on the public net. If you think they have value to the
> > > >> troubleshooting, I
> > > can
> > > >> send you the original logdata directly. I configured the
> networking
> > > >> with OVS and as follows:
> > > >>
> > > >> cloudbr0 - MGMT0 (management interface, VLAN1000), this also
> hosts
> > > >> the system VMs I guess. In the Zone setup, I labelled the
> > > >> Management
> > > network as
> > > >> MGMT0, I guess that's OK? It's a Internal network where other
> > > >> servers
> > > are
> > > >> also connected. I can ping the VMs here, also over a S2S
> connection
> > > >> cloudbr1 - Public (VLAN 4000) and Guest network (VLAN 500-999).
> I
> > > >> use real public IP-addresses from a scope and I am able to ping
> the
> > > addresses
> > > >> of both the CPVM and SSVM from my home.
> > > >>
> > > >> I have to update the situation though...
> > > >>
> > > >> Somehow at 13:02, I had the last error from the agent. Now, I do
> > > >> not
> > > have
> > > >> any other errors and both VMs are now showing as "running" as
> well
> > > >> as
> > > agent
> > > >> being "up" in the UI... had these errors since yesterday and
> did
> > > >> not change anything after 9:00 this morning. Maybe I'm impatient
> > > >> but 4 hours seems a bit long for them to get to work. :)
> > > >>
> > > >> Console VM works though at least:
> > > >>
> > > >>
> > > >>
> > > >> My impression is that cloudstack needs a while to get hold of
> > things...
> > > >> or am I just experiencing unusual things? I have a question
> though:
> > > >> When adding an ISO this morning, I had an error about that there
> > > >> was no space left (though the storage is 20TB). Was this because
> > > >> the SSVM was not running at the time?
> > > >>
> > > >> Thank you!
> > > >> Chris
> > > >>
> > > >> On Tue, Jun 12, 2018 at 4:39 PM, Dag Sonstebo <
> > > Dag.Sonstebo@shapeblue.com
> > > >> > wrote:
> > > >>
> > > >>> Chris,
> > > >>>
> > > >>> Going off in a slightly different direction to previous
> answers. I
> > > >>> suspect your problem is with networking - how have you
> configured
> > this?
> > > >>> When you say you can ping the SSVM on the private interface
> which
> > > >>> IP address do you use and where do you successfully ping from?
> > > >>>
> > > >>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/
> > > patchviasocket.py
> > > >>> -n
> > > >>> v-1-VM -p
> > > >>> %template=domP%type=consoleproxy%host=1.1.1.1%port=8250%name
> > > >>> =v-1-VM%zone=1%pod=1%guid=Proxy.1%proxy_vm=1%disable_rp_filt
> > > >>> er=true%eth2ip=9.9.9.9%eth2mask=255.255.255.0%gateway=9.9.9.
> > > >>> 1%eth0ip=169.254.3.159%eth0mask=255.255.0.0%eth1ip=6.
> > > >>> 6.6.6%eth1mask=255.255.255.0%mgmtcidr=
> > > >>> 9.9.9.0/24%localgw=1.2.3.4%internaldns1=1.2.3.4%dns1=8.8.8.8
> > > >>> %dns2=8.8.4.4
> > > >>>
> > > >>> It could be you have edited the above IP addresses to mask your
> > > >>> real addresses – if so ignore this.
> > > >>>
> > > >>> If not then the above points to:
> > > >>> - Management host is on 1.1.1.1
> > > >>> - Eth2 which for a console proxy is public traffic is on
> > > >>> 9.9.9.9/24
> > > >>> - Eth0 which is the link local management interface is on
> > > >>> 169.254.3.159/16 (system generated)
> > > >>> - Eth1 is the main management interface on 6.6.6.6/24
> > > >>> - You have a gateway address of 1.2.3.4
> > > >>>
> > > >>> So in this case – the CPVM can not check in to the management
> host
> > > >>> on
> > > >>> 1.1.1.1 - It’s got no interface on that subnet and it also
> has a
> > > gateway
> > > >>> it’s not able to reach.
> > > >>>
> > > >>> Regards,
> > > >>> Dag Sonstebo
> > > >>> Cloud Architect
> > > >>> ShapeBlue
> > > >>>
> > > >>> On 12/06/2018, 13:12, "Nicolas Bouige" <n....@dimsi.fr>
> wrote:
> > > >>>
> > > >>> Hi Ivan,
> > > >>>
> > > >>>
> > > >>> Are you talking about this global parameters :
> > > >>>
> > > >>> router.aggregation.command.each.timeout
> > > >>>
> > > >>>
> > > >>>
> > > >>> Best regards,
> > > >>>
> > > >>> Nicolas Bouige
> > > >>> DIMSI
> > > >>> cloud.dimsi.fr<http://www.cloud.dimsi.fr>
> > > >>> 4, avenue Laurent Cely
> > > >>> <https://maps.google.com/?q=4,+avenue+Laurent+Cely&entry=
> > > gmail&source=g>
> > > >>> Tour d’Asnière – 92600 Asnière sur Seine
> > > >>> T/ +33 (0)6 28 98 53 40
> > > >>>
> > > >>>
> > > >>> ________________________________
> > > >>> De : Ivan Kudryavtsev <ku...@bw-sw.com>
> > > >>> Envoyé : mardi 12 juin 2018 13:59:39
> > > >>> À : users
> > > >>> Objet : Re: SSVM's not starting, timeout for libvirt python
> > > >>> script in agent.log
> > > >>>
> > > >>> Increasing command timeouts in global parameters can work
> > > >>> here. At least I
> > > >>> met similar behaviour with VR.
> > > >>>
> > > >>> вт, 12 июн. 2018 г., 14:39 Christoffer Pedersen <
> vrod@vrod.dk>:
> > > >>>
> > > >>> > Hi Nicolas,
> > > >>> >
> > > >>> > I did a apt show qemu and it gave me this version:
> > > >>> >
> > > >>> > Version: 1:2.5+dfsg-5ubuntu10.29
> > > >>> >
> > > >>> > So I guess tha would be version 2.5?
> > > >>> >
> > > >>>
> > > >>> Dag.Sonstebo@shapeblue.com
> > > >>> www.shapeblue.com
> > > >>> 53 Chandos Place, Covent Garden, London
> > > >>> <https://maps.google.com/?q=53+Chandos+Place,+Covent+
> > > Garden,+London&entry=gmail&source=g>
> > > >>> WC2N 4HSUK
> > > >>> @shapeblue
> > > >>>
> > > >>>
> > > >>>
> > > >>> > On Tue, Jun 12, 2018 at 1:04 PM, Nicolas Bouige
> > > >>> > <n....@dimsi.fr>
> > > >>> wrote:
> > > >>> >
> > > >>> > > Hello Christoffer,
> > > >>> > >
> > > >>> > >
> > > >>> > > Could you tell us wich qemu version are you using ?
> > > >>> > >
> > > >>> > > Nicolas Bouige
> > > >>> > > DIMSI
> > > >>> > > cloud.dimsi.fr<http://www.cloud.dimsi.fr>
> > > >>> > > 4, avenue Laurent Cely
> > > >>> <https://maps.google.com/?q=4,+avenue+Laurent+Cely&entry=
> > > gmail&source=g>
> > > >>> > > Tour d’Asnière – 92600 Asnière sur Seine
> > > >>> > > T/ +33 (0)6 28 98 53 40
> > > >>> > >
> > > >>> > >
> > > >>> > > ________________________________
> > > >>> > > De : Christoffer Pedersen <vr...@vrod.dk>
> > > >>> > > Envoyé : mardi 12 juin 2018 12:30:48
> > > >>> > > À : users@cloudstack.apache.org
> > > >>> > > Objet : SSVM's not starting, timeout for libvirt python
> > > >>> script
> > > in
> > > >>> > agent.log
> > > >>> > >
> > > >>> > > Hi all,
> > > >>> > >
> > > >>> > > I have an issue regarding the system VMs. After
> deploying
> > > >>> an advanced
> > > >>> > zone,
> > > >>> > > the system VMs are trying to be created but gets stuck
> in
> > > >>> a "Starting"
> > > >>> > > state, however the Agent state is "Up". I have these
> logs
> > > >>> in
> > > the
> > > >>> > agent.log
> > > >>> > > (sorry for the formatting)
> > > >>> > >
> > > >>> > > 2018-06-12 12:22:06,354 WARN
> > > >>> [kvm.resource.LibvirtComputing Resource]
> > > >>> > > (Script-8:null) (logid:) Interrupting script.
> > > >>> > > 2018-06-12 12:22:06,355 WARN
> > > >>> [kvm.resource.LibvirtComputing Resource]
> > > >>> > > (agentRequest-Handler-4:null) (logid:ea9cb55a) Timed
> out:
> > > >>> > >
> > > >>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patch
> > > >>> viasocket.py
> > > >>> > > -n
> > > >>> > > v-1-VM -p
> > > >>> > > %template=domP%type=consoleproxy%host=1.1.1.1%
> > > >>> > > port=8250%name=v-1-VM%zone=1%
> pod=1%guid=Proxy.1%proxy_vm=1%
> > > >>> > > disable_rp_filter=true%eth2ip=
> 9.9.9.9%eth2mask=255.255.255.
> > > >>> > > 0%gateway=9.9.9.1%eth0ip=169.
> 254.3.159%eth0mask=255.255.0.
> > > >>> > > 0%eth1ip=6.6.6.6%eth1mask=255.255.255.0%mgmtcidr=
> > > >>> > >
> > > >>> > 9.9.9.0/24%localgw=1.2.3.4%
> internaldns1=1.2.3.4%dns1=8.8.8.8
> > > >>> %dns2=8.8.4.4
> > > >>> > > . Output is:
> > > >>> > > 2018-06-12 12:22:06,355 ERROR
> > > >>> [kvm.resource.LibvirtComputing Resource]
> > > >>> > > (agentRequest-Handler-4:null) (logid:ea9cb55a) passcmd
> > > >>> failed:timeout
> > > >>> > > 2018-06-12 12:22:08,914 WARN
> > > >>> [kvm.resource.LibvirtComputing Resource]
> > > >>> > > (Script-4:null) (logid:) Interrupting script.
> > > >>> > > 2018-06-12 12:22:08,915 WARN
> > > >>> [kvm.resource.LibvirtComputing Resource]
> > > >>> > > (agentRequest-Handler-5:null) (logid:8e44093e) Timed
> out:
> > > >>> > >
> > > >>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patch
> > > >>> viasocket.py
> > > >>> > > -n
> > > >>> > > s-2-VM -p
> > > >>> > > %template=domP%type=secstorage%host=1.1.1.1%port=
> > > >>> > >
> > > >>> 8250%name=s-2-VM%zone=1%pod=1%guid=s-2-VM%workers=5%resource
> > > >>> =org.apache.
> > > >>> > > cloudstack.storage.resource.
> NfsSecondaryStorageResource%
> > > >>> > > instance=SecStorage%sslcopy=
> false%role=templateProcessor%
> > > >>> > >
> > > >>> mtu=1500%eth2ip=7.7.7.7%eth2mask=255.255.255.0%gateway=9.9.9
> > > >>> .1%public.
> > > >>> > > network.device=eth2%eth0ip=169.254.2.193%eth0mask=255.
> > > >>> > > 255.0.0%eth1ip=10.120.0.61%eth1mask=255.255.255.0%
> mgmtcidr=
> > > >>> > > 9.9.9.0/24%localgw=1.2.3.4%
> private.network.device=eth1%
> > > >>> > > internaldns1=1.2.3.4%dns1=8.8.
> 8.8%dns2=8.8.4.4%nfsVersion=
> > null
> > > >>> > > . Output is:
> > > >>> > > 2018-06-12 12:22:08,915 ERROR
> > > >>> [kvm.resource.LibvirtComputing Resource]
> > > >>> > > (agentRequest-Handler-5:null) (logid:8e44093e) passcmd
> > > >>> failed:timeout
> > > >>> > >
> > > >>> > > I have seen this error around but did not really find a
> > > solution
> > > >>> to it. I
> > > >>> > > am not exactly sure whats "timing" out? I can ping both
> > > >>> SSVM's on their
> > > >>> > > private and public interface.
> > > >>> > >
> > > >>> > > I hope someone can help me out here. :)
> > > >>> > >
> > > >>> > > --
> > > >>> > > Thanks,
> > > >>> > > Chris pedersen
> > > >>> > >
> > > >>> >
> > > >>> >
> > > >>> >
> > > >>> > --
> > > >>> > Thanks,
> > > >>> > Chris pedersen
> > > >>> >
> > > >>>
> > > >>>
> > > >>>
> > > >>
> > > >>
> > > >> --
> > > >> Thanks,
> > > >> Chris pedersen
> > > >>
> > > >
> > > >
> > > >
> > > > --
> > > > Thanks,
> > > > Chris pedersen
> > > >
> > >
> >
> >
> >
> > --
> > Thanks,
> > Chris pedersen
> >
>
>
>
> --
> Thanks,
> Chris pedersen
>
>
>
--
Thanks,
Chris pedersen
Re: SSVM's not starting, timeout for libvirt python script in agent.log
Posted by Christoffer Pedersen <vr...@vrod.dk>.
Hi Dag,
- Network menu > select view pulldown box > VPN users
=>>> You mean under "Select view:" (which defaults to Guest Network),
right? There I am only able to see "Guest Network", "VPC" and "VPN Customer
Gateway"
- I have just tested creating a new project as a normal user, change this
on the pulldown project menu and clicked on the “network” menu item, no
error in RC2 – is this where you found it?
=>>> I just tried again with a user i created, it's an admin though. I
still get the message.
- This is by design, for VPCs you have to add additional IPs to configure
services. The only service available on the primary VPC IP is VPN.
=>>> Ok thanks, good to know!
- Minutes yes, hours no. Going by your previous posts I would guess you
still have some tuning up to do and possibly run on low spec hardware
though?
=>>> The 2 hosts I have run with 2x E5-2680's and 64GB of ram. The
management server itself is a VMware hosted VM with 16gb of memory and
4vCPU's (E5-2650). It could be the tuning I guess, I am currently at a loss
here. The storage is a FreeNAS based NFS share but it's pretty fast.
On Wed, Jun 13, 2018 at 5:54 PM, Dag Sonstebo <Da...@shapeblue.com>
wrote:
> Hi Chris,
>
> - I have enabled remote VPN in a VPC but I cannot seem to add users
> anywhere. On the VPN tab where the PSK is displayed it says: "Note: VPN
> users are now accessed by changing views at the networks tab". However, I
> cannot find out where such view would be? The documentation doesn't state
> anything about this.
>
> =>>> Network menu > select view pulldown box > VPN users
>
> - When I create a new project, choose the view from that project and go to
> "Network", I always get this message: " Account and projectId can't be
> specified together". What is it exactly and how can I get rid of it?
>
> =>>> I have just tested creating a new project as a normal user, change
> this on the pulldown project menu and clicked on the “network” menu item,
> no error in RC2 – is this where you found it?
>
> - I don't seem to be able to do port-forwarding with the first configured
> IP for the VPC. However when adding another IP, I can do it on that
> specific IP. Isn't it possible to do forwarding on the first assigned IP?
>
> =>>> This is by design, for VPCs you have to add additional IPs to
> configure services. The only service available on the primary VPC IP is VPN.
>
> - Is it normal that it takes minutes, sometimes hours to create the VPC? It
> really takes a while for the system to get going once I have created a VPC.
>
> =>>> Minutes yes, hours no. Going by your previous posts I would guess you
> still have some tuning up to do and possibly run on low spec hardware
> though?
>
>
> Regards,
> Dag Sonstebo
> Cloud Architect
> ShapeBlue
>
> On 13/06/2018, 16:25, "Christoffer Pedersen" <vr...@vrod.dk> wrote:
>
> The routers managed to start after a while. Looks like most things are
> OK
> now, however I hope some of you could answer some questions for me.
>
> - I have enabled remote VPN in a VPC but I cannot seem to add users
> anywhere. On the VPN tab where the PSK is displayed it says: "Note: VPN
> users are now accessed by changing views at the networks tab".
> However, I
> cannot find out where such view would be? The documentation doesn't
> state
> anything about this.
> - When I create a new project, choose the view from that project and
> go to
> "Network", I always get this message: " Account and projectId can't be
> specified together". What is it exactly and how can I get rid of it?
> - I don't seem to be able to do port-forwarding with the first
> configured
> IP for the VPC. However when adding another IP, I can do it on that
> specific IP. Isn't it possible to do forwarding on the first assigned
> IP?
> - Is it normal that it takes minutes, sometimes hours to create the
> VPC? It
> really takes a while for the system to get going once I have created a
> VPC.
>
> Perhaps this is for another email-thread, please let me know if so.
>
> Apart from that it seems like the cloudstack instance is finally
> working,
> first VM booted up!
>
> Thank you!
> Chris
>
>
> Dag.Sonstebo@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London WC2N 4HSUK
> @shapeblue
>
>
>
> On Wed, Jun 13, 2018 at 4:15 PM, Nicolas Bouige <n....@dimsi.fr> wrote:
>
> > Hello All,
> >
> > Maybe is not related to your case but in case of....as we don't use
> the
> > same OS
> >
> > I was facing the exact same error during a new creation of virtual
> router.
> > The only change i made was to update my qemu version on our KVM
> (centOS7 +
> > CS 4.11) from 1.5 to 2.3
> > Before that VR creation was working fine.
> >
> > Apperently some users got the same issue but with an old version of
> CS
> > (4.9)
> > http://dev.cloudstack.apache.narkive.com/qsKmnsa4/
> > patchviasocket-seems-to-be-broken-with-qemu-2-3
> >
> >
> >
> > -----Message d'origine-----
> > De : Christoffer Pedersen [mailto:vrod@vrod.dk]
> > Envoyé : mardi 12 juin 2018 19:17
> > À : users@cloudstack.apache.org
> > Objet : Re: SSVM's not starting, timeout for libvirt python script in
> > agent.log
> >
> > Hi James,
> >
> > I looked at the value and it's already on 50GB. I looked a little
> more and
> > it seems like after restarting the cloudstack-management service I
> could no
> > longer reach the console or storage system vm.... looking at the
> host, now
> > the cloud0 interface is gone. I am using OVS, but according to a
> Shapeblue
> > article, no extra config is needed for the cloud0 interface.
> >
> >
> >
> > On Tue, Jun 12, 2018 at 6:17 PM, McClune, James <
> > mcclunej@norwalktruckers.net> wrote:
> >
> > > Hi Christoffer,
> > >
> > > Did you try setting the max.template.iso.size to a higher value
> (e.g.
> > > 50GB)?
> > >
> > > I think the default is set pretty low.
> > >
> > > Best,
> > > James
> > >
> > > On Tue, Jun 12, 2018 at 11:10 AM, Christoffer Pedersen <
> vrod@vrod.dk>
> > > wrote:
> > >
> > > > Just want to update that I still get the ISO error:
> > > >
> > > > 2018-06-12 17:09:42,194 INFO [c.c.t.HypervisorTemplateAdapter]
> > > > (qtp1096283470-12:ctx-962ec49c ctx-2c82c0e2) (logid:0c7df1d7)
> Image
> > > > store doesn't have enough capacity. Skip downloading template to
> > > > this image
> > > store
> > > > 1
> > > >
> > > > Thanks!
> > > >
> > > > On Tue, Jun 12, 2018 at 5:07 PM, Christoffer Pedersen <
> vrod@vrod.dk>
> > > > wrote:
> > > >
> > > >> Hi Dag,
> > > >>
> > > >> Yes, I altered the IP addresses as I do not fancy throwing them
> out
> > > >> on the public net. If you think they have value to the
> > > >> troubleshooting, I
> > > can
> > > >> send you the original logdata directly. I configured the
> networking
> > > >> with OVS and as follows:
> > > >>
> > > >> cloudbr0 - MGMT0 (management interface, VLAN1000), this also
> hosts
> > > >> the system VMs I guess. In the Zone setup, I labelled the
> > > >> Management
> > > network as
> > > >> MGMT0, I guess that's OK? It's a Internal network where other
> > > >> servers
> > > are
> > > >> also connected. I can ping the VMs here, also over a S2S
> connection
> > > >> cloudbr1 - Public (VLAN 4000) and Guest network (VLAN 500-999).
> I
> > > >> use real public IP-addresses from a scope and I am able to ping
> the
> > > addresses
> > > >> of both the CPVM and SSVM from my home.
> > > >>
> > > >> I have to update the situation though...
> > > >>
> > > >> Somehow at 13:02, I had the last error from the agent. Now, I do
> > > >> not
> > > have
> > > >> any other errors and both VMs are now showing as "running" as
> well
> > > >> as
> > > agent
> > > >> being "up" in the UI... had these errors since yesterday and
> did
> > > >> not change anything after 9:00 this morning. Maybe I'm impatient
> > > >> but 4 hours seems a bit long for them to get to work. :)
> > > >>
> > > >> Console VM works though at least:
> > > >>
> > > >>
> > > >>
> > > >> My impression is that cloudstack needs a while to get hold of
> > things...
> > > >> or am I just experiencing unusual things? I have a question
> though:
> > > >> When adding an ISO this morning, I had an error about that there
> > > >> was no space left (though the storage is 20TB). Was this because
> > > >> the SSVM was not running at the time?
> > > >>
> > > >> Thank you!
> > > >> Chris
> > > >>
> > > >> On Tue, Jun 12, 2018 at 4:39 PM, Dag Sonstebo <
> > > Dag.Sonstebo@shapeblue.com
> > > >> > wrote:
> > > >>
> > > >>> Chris,
> > > >>>
> > > >>> Going off in a slightly different direction to previous
> answers. I
> > > >>> suspect your problem is with networking - how have you
> configured
> > this?
> > > >>> When you say you can ping the SSVM on the private interface
> which
> > > >>> IP address do you use and where do you successfully ping from?
> > > >>>
> > > >>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/
> > > patchviasocket.py
> > > >>> -n
> > > >>> v-1-VM -p
> > > >>> %template=domP%type=consoleproxy%host=1.1.1.1%port=8250%name
> > > >>> =v-1-VM%zone=1%pod=1%guid=Proxy.1%proxy_vm=1%disable_rp_filt
> > > >>> er=true%eth2ip=9.9.9.9%eth2mask=255.255.255.0%gateway=9.9.9.
> > > >>> 1%eth0ip=169.254.3.159%eth0mask=255.255.0.0%eth1ip=6.
> > > >>> 6.6.6%eth1mask=255.255.255.0%mgmtcidr=
> > > >>> 9.9.9.0/24%localgw=1.2.3.4%internaldns1=1.2.3.4%dns1=8.8.8.8
> > > >>> %dns2=8.8.4.4
> > > >>>
> > > >>> It could be you have edited the above IP addresses to mask your
> > > >>> real addresses – if so ignore this.
> > > >>>
> > > >>> If not then the above points to:
> > > >>> - Management host is on 1.1.1.1
> > > >>> - Eth2 which for a console proxy is public traffic is on
> > > >>> 9.9.9.9/24
> > > >>> - Eth0 which is the link local management interface is on
> > > >>> 169.254.3.159/16 (system generated)
> > > >>> - Eth1 is the main management interface on 6.6.6.6/24
> > > >>> - You have a gateway address of 1.2.3.4
> > > >>>
> > > >>> So in this case – the CPVM can not check in to the management
> host
> > > >>> on
> > > >>> 1.1.1.1 - It’s got no interface on that subnet and it also
> has a
> > > gateway
> > > >>> it’s not able to reach.
> > > >>>
> > > >>> Regards,
> > > >>> Dag Sonstebo
> > > >>> Cloud Architect
> > > >>> ShapeBlue
> > > >>>
> > > >>> On 12/06/2018, 13:12, "Nicolas Bouige" <n....@dimsi.fr>
> wrote:
> > > >>>
> > > >>> Hi Ivan,
> > > >>>
> > > >>>
> > > >>> Are you talking about this global parameters :
> > > >>>
> > > >>> router.aggregation.command.each.timeout
> > > >>>
> > > >>>
> > > >>>
> > > >>> Best regards,
> > > >>>
> > > >>> Nicolas Bouige
> > > >>> DIMSI
> > > >>> cloud.dimsi.fr<http://www.cloud.dimsi.fr>
> > > >>> 4, avenue Laurent Cely
> > > >>> <https://maps.google.com/?q=4,+avenue+Laurent+Cely&entry=
> > > gmail&source=g>
> > > >>> Tour d’Asnière – 92600 Asnière sur Seine
> > > >>> T/ +33 (0)6 28 98 53 40
> > > >>>
> > > >>>
> > > >>> ________________________________
> > > >>> De : Ivan Kudryavtsev <ku...@bw-sw.com>
> > > >>> Envoyé : mardi 12 juin 2018 13:59:39
> > > >>> À : users
> > > >>> Objet : Re: SSVM's not starting, timeout for libvirt python
> > > >>> script in agent.log
> > > >>>
> > > >>> Increasing command timeouts in global parameters can work
> > > >>> here. At least I
> > > >>> met similar behaviour with VR.
> > > >>>
> > > >>> вт, 12 июн. 2018 г., 14:39 Christoffer Pedersen <
> vrod@vrod.dk>:
> > > >>>
> > > >>> > Hi Nicolas,
> > > >>> >
> > > >>> > I did a apt show qemu and it gave me this version:
> > > >>> >
> > > >>> > Version: 1:2.5+dfsg-5ubuntu10.29
> > > >>> >
> > > >>> > So I guess tha would be version 2.5?
> > > >>> >
> > > >>>
> > > >>> Dag.Sonstebo@shapeblue.com
> > > >>> www.shapeblue.com
> > > >>> 53 Chandos Place, Covent Garden, London
> > > >>> <https://maps.google.com/?q=53+Chandos+Place,+Covent+
> > > Garden,+London&entry=gmail&source=g>
> > > >>> WC2N 4HSUK
> > > >>> @shapeblue
> > > >>>
> > > >>>
> > > >>>
> > > >>> > On Tue, Jun 12, 2018 at 1:04 PM, Nicolas Bouige
> > > >>> > <n....@dimsi.fr>
> > > >>> wrote:
> > > >>> >
> > > >>> > > Hello Christoffer,
> > > >>> > >
> > > >>> > >
> > > >>> > > Could you tell us wich qemu version are you using ?
> > > >>> > >
> > > >>> > > Nicolas Bouige
> > > >>> > > DIMSI
> > > >>> > > cloud.dimsi.fr<http://www.cloud.dimsi.fr>
> > > >>> > > 4, avenue Laurent Cely
> > > >>> <https://maps.google.com/?q=4,+avenue+Laurent+Cely&entry=
> > > gmail&source=g>
> > > >>> > > Tour d’Asnière – 92600 Asnière sur Seine
> > > >>> > > T/ +33 (0)6 28 98 53 40
> > > >>> > >
> > > >>> > >
> > > >>> > > ________________________________
> > > >>> > > De : Christoffer Pedersen <vr...@vrod.dk>
> > > >>> > > Envoyé : mardi 12 juin 2018 12:30:48
> > > >>> > > À : users@cloudstack.apache.org
> > > >>> > > Objet : SSVM's not starting, timeout for libvirt python
> > > >>> script
> > > in
> > > >>> > agent.log
> > > >>> > >
> > > >>> > > Hi all,
> > > >>> > >
> > > >>> > > I have an issue regarding the system VMs. After
> deploying
> > > >>> an advanced
> > > >>> > zone,
> > > >>> > > the system VMs are trying to be created but gets stuck
> in
> > > >>> a "Starting"
> > > >>> > > state, however the Agent state is "Up". I have these
> logs
> > > >>> in
> > > the
> > > >>> > agent.log
> > > >>> > > (sorry for the formatting)
> > > >>> > >
> > > >>> > > 2018-06-12 12:22:06,354 WARN
> > > >>> [kvm.resource.LibvirtComputing Resource]
> > > >>> > > (Script-8:null) (logid:) Interrupting script.
> > > >>> > > 2018-06-12 12:22:06,355 WARN
> > > >>> [kvm.resource.LibvirtComputing Resource]
> > > >>> > > (agentRequest-Handler-4:null) (logid:ea9cb55a) Timed
> out:
> > > >>> > >
> > > >>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patch
> > > >>> viasocket.py
> > > >>> > > -n
> > > >>> > > v-1-VM -p
> > > >>> > > %template=domP%type=consoleproxy%host=1.1.1.1%
> > > >>> > > port=8250%name=v-1-VM%zone=1%
> pod=1%guid=Proxy.1%proxy_vm=1%
> > > >>> > > disable_rp_filter=true%eth2ip=
> 9.9.9.9%eth2mask=255.255.255.
> > > >>> > > 0%gateway=9.9.9.1%eth0ip=169.
> 254.3.159%eth0mask=255.255.0.
> > > >>> > > 0%eth1ip=6.6.6.6%eth1mask=255.255.255.0%mgmtcidr=
> > > >>> > >
> > > >>> > 9.9.9.0/24%localgw=1.2.3.4%
> internaldns1=1.2.3.4%dns1=8.8.8.8
> > > >>> %dns2=8.8.4.4
> > > >>> > > . Output is:
> > > >>> > > 2018-06-12 12:22:06,355 ERROR
> > > >>> [kvm.resource.LibvirtComputing Resource]
> > > >>> > > (agentRequest-Handler-4:null) (logid:ea9cb55a) passcmd
> > > >>> failed:timeout
> > > >>> > > 2018-06-12 12:22:08,914 WARN
> > > >>> [kvm.resource.LibvirtComputing Resource]
> > > >>> > > (Script-4:null) (logid:) Interrupting script.
> > > >>> > > 2018-06-12 12:22:08,915 WARN
> > > >>> [kvm.resource.LibvirtComputing Resource]
> > > >>> > > (agentRequest-Handler-5:null) (logid:8e44093e) Timed
> out:
> > > >>> > >
> > > >>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patch
> > > >>> viasocket.py
> > > >>> > > -n
> > > >>> > > s-2-VM -p
> > > >>> > > %template=domP%type=secstorage%host=1.1.1.1%port=
> > > >>> > >
> > > >>> 8250%name=s-2-VM%zone=1%pod=1%guid=s-2-VM%workers=5%resource
> > > >>> =org.apache.
> > > >>> > > cloudstack.storage.resource.
> NfsSecondaryStorageResource%
> > > >>> > > instance=SecStorage%sslcopy=
> false%role=templateProcessor%
> > > >>> > >
> > > >>> mtu=1500%eth2ip=7.7.7.7%eth2mask=255.255.255.0%gateway=9.9.9
> > > >>> .1%public.
> > > >>> > > network.device=eth2%eth0ip=169.254.2.193%eth0mask=255.
> > > >>> > > 255.0.0%eth1ip=10.120.0.61%eth1mask=255.255.255.0%
> mgmtcidr=
> > > >>> > > 9.9.9.0/24%localgw=1.2.3.4%
> private.network.device=eth1%
> > > >>> > > internaldns1=1.2.3.4%dns1=8.8.
> 8.8%dns2=8.8.4.4%nfsVersion=
> > null
> > > >>> > > . Output is:
> > > >>> > > 2018-06-12 12:22:08,915 ERROR
> > > >>> [kvm.resource.LibvirtComputing Resource]
> > > >>> > > (agentRequest-Handler-5:null) (logid:8e44093e) passcmd
> > > >>> failed:timeout
> > > >>> > >
> > > >>> > > I have seen this error around but did not really find a
> > > solution
> > > >>> to it. I
> > > >>> > > am not exactly sure whats "timing" out? I can ping both
> > > >>> SSVM's on their
> > > >>> > > private and public interface.
> > > >>> > >
> > > >>> > > I hope someone can help me out here. :)
> > > >>> > >
> > > >>> > > --
> > > >>> > > Thanks,
> > > >>> > > Chris pedersen
> > > >>> > >
> > > >>> >
> > > >>> >
> > > >>> >
> > > >>> > --
> > > >>> > Thanks,
> > > >>> > Chris pedersen
> > > >>> >
> > > >>>
> > > >>>
> > > >>>
> > > >>
> > > >>
> > > >> --
> > > >> Thanks,
> > > >> Chris pedersen
> > > >>
> > > >
> > > >
> > > >
> > > > --
> > > > Thanks,
> > > > Chris pedersen
> > > >
> > >
> >
> >
> >
> > --
> > Thanks,
> > Chris pedersen
> >
>
>
>
> --
> Thanks,
> Chris pedersen
>
>
>
--
Thanks,
Chris pedersen
Re: SSVM's not starting, timeout for libvirt python
script in agent.log
Posted by Dag Sonstebo <Da...@shapeblue.com>.
Hi Chris,
- I have enabled remote VPN in a VPC but I cannot seem to add users
anywhere. On the VPN tab where the PSK is displayed it says: "Note: VPN
users are now accessed by changing views at the networks tab". However, I
cannot find out where such view would be? The documentation doesn't state
anything about this.
=>>> Network menu > select view pulldown box > VPN users
- When I create a new project, choose the view from that project and go to
"Network", I always get this message: " Account and projectId can't be
specified together". What is it exactly and how can I get rid of it?
=>>> I have just tested creating a new project as a normal user, change this on the pulldown project menu and clicked on the “network” menu item, no error in RC2 – is this where you found it?
- I don't seem to be able to do port-forwarding with the first configured
IP for the VPC. However when adding another IP, I can do it on that
specific IP. Isn't it possible to do forwarding on the first assigned IP?
=>>> This is by design, for VPCs you have to add additional IPs to configure services. The only service available on the primary VPC IP is VPN.
- Is it normal that it takes minutes, sometimes hours to create the VPC? It
really takes a while for the system to get going once I have created a VPC.
=>>> Minutes yes, hours no. Going by your previous posts I would guess you still have some tuning up to do and possibly run on low spec hardware though?
Regards,
Dag Sonstebo
Cloud Architect
ShapeBlue
On 13/06/2018, 16:25, "Christoffer Pedersen" <vr...@vrod.dk> wrote:
The routers managed to start after a while. Looks like most things are OK
now, however I hope some of you could answer some questions for me.
- I have enabled remote VPN in a VPC but I cannot seem to add users
anywhere. On the VPN tab where the PSK is displayed it says: "Note: VPN
users are now accessed by changing views at the networks tab". However, I
cannot find out where such view would be? The documentation doesn't state
anything about this.
- When I create a new project, choose the view from that project and go to
"Network", I always get this message: " Account and projectId can't be
specified together". What is it exactly and how can I get rid of it?
- I don't seem to be able to do port-forwarding with the first configured
IP for the VPC. However when adding another IP, I can do it on that
specific IP. Isn't it possible to do forwarding on the first assigned IP?
- Is it normal that it takes minutes, sometimes hours to create the VPC? It
really takes a while for the system to get going once I have created a VPC.
Perhaps this is for another email-thread, please let me know if so.
Apart from that it seems like the cloudstack instance is finally working,
first VM booted up!
Thank you!
Chris
Dag.Sonstebo@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
On Wed, Jun 13, 2018 at 4:15 PM, Nicolas Bouige <n....@dimsi.fr> wrote:
> Hello All,
>
> Maybe is not related to your case but in case of....as we don't use the
> same OS
>
> I was facing the exact same error during a new creation of virtual router.
> The only change i made was to update my qemu version on our KVM (centOS7 +
> CS 4.11) from 1.5 to 2.3
> Before that VR creation was working fine.
>
> Apperently some users got the same issue but with an old version of CS
> (4.9)
> http://dev.cloudstack.apache.narkive.com/qsKmnsa4/
> patchviasocket-seems-to-be-broken-with-qemu-2-3
>
>
>
> -----Message d'origine-----
> De : Christoffer Pedersen [mailto:vrod@vrod.dk]
> Envoyé : mardi 12 juin 2018 19:17
> À : users@cloudstack.apache.org
> Objet : Re: SSVM's not starting, timeout for libvirt python script in
> agent.log
>
> Hi James,
>
> I looked at the value and it's already on 50GB. I looked a little more and
> it seems like after restarting the cloudstack-management service I could no
> longer reach the console or storage system vm.... looking at the host, now
> the cloud0 interface is gone. I am using OVS, but according to a Shapeblue
> article, no extra config is needed for the cloud0 interface.
>
>
>
> On Tue, Jun 12, 2018 at 6:17 PM, McClune, James <
> mcclunej@norwalktruckers.net> wrote:
>
> > Hi Christoffer,
> >
> > Did you try setting the max.template.iso.size to a higher value (e.g.
> > 50GB)?
> >
> > I think the default is set pretty low.
> >
> > Best,
> > James
> >
> > On Tue, Jun 12, 2018 at 11:10 AM, Christoffer Pedersen <vr...@vrod.dk>
> > wrote:
> >
> > > Just want to update that I still get the ISO error:
> > >
> > > 2018-06-12 17:09:42,194 INFO [c.c.t.HypervisorTemplateAdapter]
> > > (qtp1096283470-12:ctx-962ec49c ctx-2c82c0e2) (logid:0c7df1d7) Image
> > > store doesn't have enough capacity. Skip downloading template to
> > > this image
> > store
> > > 1
> > >
> > > Thanks!
> > >
> > > On Tue, Jun 12, 2018 at 5:07 PM, Christoffer Pedersen <vr...@vrod.dk>
> > > wrote:
> > >
> > >> Hi Dag,
> > >>
> > >> Yes, I altered the IP addresses as I do not fancy throwing them out
> > >> on the public net. If you think they have value to the
> > >> troubleshooting, I
> > can
> > >> send you the original logdata directly. I configured the networking
> > >> with OVS and as follows:
> > >>
> > >> cloudbr0 - MGMT0 (management interface, VLAN1000), this also hosts
> > >> the system VMs I guess. In the Zone setup, I labelled the
> > >> Management
> > network as
> > >> MGMT0, I guess that's OK? It's a Internal network where other
> > >> servers
> > are
> > >> also connected. I can ping the VMs here, also over a S2S connection
> > >> cloudbr1 - Public (VLAN 4000) and Guest network (VLAN 500-999). I
> > >> use real public IP-addresses from a scope and I am able to ping the
> > addresses
> > >> of both the CPVM and SSVM from my home.
> > >>
> > >> I have to update the situation though...
> > >>
> > >> Somehow at 13:02, I had the last error from the agent. Now, I do
> > >> not
> > have
> > >> any other errors and both VMs are now showing as "running" as well
> > >> as
> > agent
> > >> being "up" in the UI... had these errors since yesterday and did
> > >> not change anything after 9:00 this morning. Maybe I'm impatient
> > >> but 4 hours seems a bit long for them to get to work. :)
> > >>
> > >> Console VM works though at least:
> > >>
> > >>
> > >>
> > >> My impression is that cloudstack needs a while to get hold of
> things...
> > >> or am I just experiencing unusual things? I have a question though:
> > >> When adding an ISO this morning, I had an error about that there
> > >> was no space left (though the storage is 20TB). Was this because
> > >> the SSVM was not running at the time?
> > >>
> > >> Thank you!
> > >> Chris
> > >>
> > >> On Tue, Jun 12, 2018 at 4:39 PM, Dag Sonstebo <
> > Dag.Sonstebo@shapeblue.com
> > >> > wrote:
> > >>
> > >>> Chris,
> > >>>
> > >>> Going off in a slightly different direction to previous answers. I
> > >>> suspect your problem is with networking - how have you configured
> this?
> > >>> When you say you can ping the SSVM on the private interface which
> > >>> IP address do you use and where do you successfully ping from?
> > >>>
> > >>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/
> > patchviasocket.py
> > >>> -n
> > >>> v-1-VM -p
> > >>> %template=domP%type=consoleproxy%host=1.1.1.1%port=8250%name
> > >>> =v-1-VM%zone=1%pod=1%guid=Proxy.1%proxy_vm=1%disable_rp_filt
> > >>> er=true%eth2ip=9.9.9.9%eth2mask=255.255.255.0%gateway=9.9.9.
> > >>> 1%eth0ip=169.254.3.159%eth0mask=255.255.0.0%eth1ip=6.
> > >>> 6.6.6%eth1mask=255.255.255.0%mgmtcidr=
> > >>> 9.9.9.0/24%localgw=1.2.3.4%internaldns1=1.2.3.4%dns1=8.8.8.8
> > >>> %dns2=8.8.4.4
> > >>>
> > >>> It could be you have edited the above IP addresses to mask your
> > >>> real addresses – if so ignore this.
> > >>>
> > >>> If not then the above points to:
> > >>> - Management host is on 1.1.1.1
> > >>> - Eth2 which for a console proxy is public traffic is on
> > >>> 9.9.9.9/24
> > >>> - Eth0 which is the link local management interface is on
> > >>> 169.254.3.159/16 (system generated)
> > >>> - Eth1 is the main management interface on 6.6.6.6/24
> > >>> - You have a gateway address of 1.2.3.4
> > >>>
> > >>> So in this case – the CPVM can not check in to the management host
> > >>> on
> > >>> 1.1.1.1 - It’s got no interface on that subnet and it also has a
> > gateway
> > >>> it’s not able to reach.
> > >>>
> > >>> Regards,
> > >>> Dag Sonstebo
> > >>> Cloud Architect
> > >>> ShapeBlue
> > >>>
> > >>> On 12/06/2018, 13:12, "Nicolas Bouige" <n....@dimsi.fr> wrote:
> > >>>
> > >>> Hi Ivan,
> > >>>
> > >>>
> > >>> Are you talking about this global parameters :
> > >>>
> > >>> router.aggregation.command.each.timeout
> > >>>
> > >>>
> > >>>
> > >>> Best regards,
> > >>>
> > >>> Nicolas Bouige
> > >>> DIMSI
> > >>> cloud.dimsi.fr<http://www.cloud.dimsi.fr>
> > >>> 4, avenue Laurent Cely
> > >>> <https://maps.google.com/?q=4,+avenue+Laurent+Cely&entry=
> > gmail&source=g>
> > >>> Tour d’Asnière – 92600 Asnière sur Seine
> > >>> T/ +33 (0)6 28 98 53 40
> > >>>
> > >>>
> > >>> ________________________________
> > >>> De : Ivan Kudryavtsev <ku...@bw-sw.com>
> > >>> Envoyé : mardi 12 juin 2018 13:59:39
> > >>> À : users
> > >>> Objet : Re: SSVM's not starting, timeout for libvirt python
> > >>> script in agent.log
> > >>>
> > >>> Increasing command timeouts in global parameters can work
> > >>> here. At least I
> > >>> met similar behaviour with VR.
> > >>>
> > >>> вт, 12 июн. 2018 г., 14:39 Christoffer Pedersen <vr...@vrod.dk>:
> > >>>
> > >>> > Hi Nicolas,
> > >>> >
> > >>> > I did a apt show qemu and it gave me this version:
> > >>> >
> > >>> > Version: 1:2.5+dfsg-5ubuntu10.29
> > >>> >
> > >>> > So I guess tha would be version 2.5?
> > >>> >
> > >>>
> > >>> Dag.Sonstebo@shapeblue.com
> > >>> www.shapeblue.com
> > >>> 53 Chandos Place, Covent Garden, London
> > >>> <https://maps.google.com/?q=53+Chandos+Place,+Covent+
> > Garden,+London&entry=gmail&source=g>
> > >>> WC2N 4HSUK
> > >>> @shapeblue
> > >>>
> > >>>
> > >>>
> > >>> > On Tue, Jun 12, 2018 at 1:04 PM, Nicolas Bouige
> > >>> > <n....@dimsi.fr>
> > >>> wrote:
> > >>> >
> > >>> > > Hello Christoffer,
> > >>> > >
> > >>> > >
> > >>> > > Could you tell us wich qemu version are you using ?
> > >>> > >
> > >>> > > Nicolas Bouige
> > >>> > > DIMSI
> > >>> > > cloud.dimsi.fr<http://www.cloud.dimsi.fr>
> > >>> > > 4, avenue Laurent Cely
> > >>> <https://maps.google.com/?q=4,+avenue+Laurent+Cely&entry=
> > gmail&source=g>
> > >>> > > Tour d’Asnière – 92600 Asnière sur Seine
> > >>> > > T/ +33 (0)6 28 98 53 40
> > >>> > >
> > >>> > >
> > >>> > > ________________________________
> > >>> > > De : Christoffer Pedersen <vr...@vrod.dk>
> > >>> > > Envoyé : mardi 12 juin 2018 12:30:48
> > >>> > > À : users@cloudstack.apache.org
> > >>> > > Objet : SSVM's not starting, timeout for libvirt python
> > >>> script
> > in
> > >>> > agent.log
> > >>> > >
> > >>> > > Hi all,
> > >>> > >
> > >>> > > I have an issue regarding the system VMs. After deploying
> > >>> an advanced
> > >>> > zone,
> > >>> > > the system VMs are trying to be created but gets stuck in
> > >>> a "Starting"
> > >>> > > state, however the Agent state is "Up". I have these logs
> > >>> in
> > the
> > >>> > agent.log
> > >>> > > (sorry for the formatting)
> > >>> > >
> > >>> > > 2018-06-12 12:22:06,354 WARN
> > >>> [kvm.resource.LibvirtComputing Resource]
> > >>> > > (Script-8:null) (logid:) Interrupting script.
> > >>> > > 2018-06-12 12:22:06,355 WARN
> > >>> [kvm.resource.LibvirtComputing Resource]
> > >>> > > (agentRequest-Handler-4:null) (logid:ea9cb55a) Timed out:
> > >>> > >
> > >>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patch
> > >>> viasocket.py
> > >>> > > -n
> > >>> > > v-1-VM -p
> > >>> > > %template=domP%type=consoleproxy%host=1.1.1.1%
> > >>> > > port=8250%name=v-1-VM%zone=1%pod=1%guid=Proxy.1%proxy_vm=1%
> > >>> > > disable_rp_filter=true%eth2ip=9.9.9.9%eth2mask=255.255.255.
> > >>> > > 0%gateway=9.9.9.1%eth0ip=169.254.3.159%eth0mask=255.255.0.
> > >>> > > 0%eth1ip=6.6.6.6%eth1mask=255.255.255.0%mgmtcidr=
> > >>> > >
> > >>> > 9.9.9.0/24%localgw=1.2.3.4%internaldns1=1.2.3.4%dns1=8.8.8.8
> > >>> %dns2=8.8.4.4
> > >>> > > . Output is:
> > >>> > > 2018-06-12 12:22:06,355 ERROR
> > >>> [kvm.resource.LibvirtComputing Resource]
> > >>> > > (agentRequest-Handler-4:null) (logid:ea9cb55a) passcmd
> > >>> failed:timeout
> > >>> > > 2018-06-12 12:22:08,914 WARN
> > >>> [kvm.resource.LibvirtComputing Resource]
> > >>> > > (Script-4:null) (logid:) Interrupting script.
> > >>> > > 2018-06-12 12:22:08,915 WARN
> > >>> [kvm.resource.LibvirtComputing Resource]
> > >>> > > (agentRequest-Handler-5:null) (logid:8e44093e) Timed out:
> > >>> > >
> > >>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patch
> > >>> viasocket.py
> > >>> > > -n
> > >>> > > s-2-VM -p
> > >>> > > %template=domP%type=secstorage%host=1.1.1.1%port=
> > >>> > >
> > >>> 8250%name=s-2-VM%zone=1%pod=1%guid=s-2-VM%workers=5%resource
> > >>> =org.apache.
> > >>> > > cloudstack.storage.resource.NfsSecondaryStorageResource%
> > >>> > > instance=SecStorage%sslcopy=false%role=templateProcessor%
> > >>> > >
> > >>> mtu=1500%eth2ip=7.7.7.7%eth2mask=255.255.255.0%gateway=9.9.9
> > >>> .1%public.
> > >>> > > network.device=eth2%eth0ip=169.254.2.193%eth0mask=255.
> > >>> > > 255.0.0%eth1ip=10.120.0.61%eth1mask=255.255.255.0%mgmtcidr=
> > >>> > > 9.9.9.0/24%localgw=1.2.3.4%private.network.device=eth1%
> > >>> > > internaldns1=1.2.3.4%dns1=8.8.8.8%dns2=8.8.4.4%nfsVersion=
> null
> > >>> > > . Output is:
> > >>> > > 2018-06-12 12:22:08,915 ERROR
> > >>> [kvm.resource.LibvirtComputing Resource]
> > >>> > > (agentRequest-Handler-5:null) (logid:8e44093e) passcmd
> > >>> failed:timeout
> > >>> > >
> > >>> > > I have seen this error around but did not really find a
> > solution
> > >>> to it. I
> > >>> > > am not exactly sure whats "timing" out? I can ping both
> > >>> SSVM's on their
> > >>> > > private and public interface.
> > >>> > >
> > >>> > > I hope someone can help me out here. :)
> > >>> > >
> > >>> > > --
> > >>> > > Thanks,
> > >>> > > Chris pedersen
> > >>> > >
> > >>> >
> > >>> >
> > >>> >
> > >>> > --
> > >>> > Thanks,
> > >>> > Chris pedersen
> > >>> >
> > >>>
> > >>>
> > >>>
> > >>
> > >>
> > >> --
> > >> Thanks,
> > >> Chris pedersen
> > >>
> > >
> > >
> > >
> > > --
> > > Thanks,
> > > Chris pedersen
> > >
> >
>
>
>
> --
> Thanks,
> Chris pedersen
>
--
Thanks,
Chris pedersen
Re: SSVM's not starting, timeout for libvirt python script in agent.log
Posted by Christoffer Pedersen <vr...@vrod.dk>.
The routers managed to start after a while. Looks like most things are OK
now, however I hope some of you could answer some questions for me.
- I have enabled remote VPN in a VPC but I cannot seem to add users
anywhere. On the VPN tab where the PSK is displayed it says: "Note: VPN
users are now accessed by changing views at the networks tab". However, I
cannot find out where such view would be? The documentation doesn't state
anything about this.
- When I create a new project, choose the view from that project and go to
"Network", I always get this message: " Account and projectId can't be
specified together". What is it exactly and how can I get rid of it?
- I don't seem to be able to do port-forwarding with the first configured
IP for the VPC. However when adding another IP, I can do it on that
specific IP. Isn't it possible to do forwarding on the first assigned IP?
- Is it normal that it takes minutes, sometimes hours to create the VPC? It
really takes a while for the system to get going once I have created a VPC.
Perhaps this is for another email-thread, please let me know if so.
Apart from that it seems like the cloudstack instance is finally working,
first VM booted up!
Thank you!
Chris
On Wed, Jun 13, 2018 at 4:15 PM, Nicolas Bouige <n....@dimsi.fr> wrote:
> Hello All,
>
> Maybe is not related to your case but in case of....as we don't use the
> same OS
>
> I was facing the exact same error during a new creation of virtual router.
> The only change i made was to update my qemu version on our KVM (centOS7 +
> CS 4.11) from 1.5 to 2.3
> Before that VR creation was working fine.
>
> Apperently some users got the same issue but with an old version of CS
> (4.9)
> http://dev.cloudstack.apache.narkive.com/qsKmnsa4/
> patchviasocket-seems-to-be-broken-with-qemu-2-3
>
>
>
> -----Message d'origine-----
> De : Christoffer Pedersen [mailto:vrod@vrod.dk]
> Envoyé : mardi 12 juin 2018 19:17
> À : users@cloudstack.apache.org
> Objet : Re: SSVM's not starting, timeout for libvirt python script in
> agent.log
>
> Hi James,
>
> I looked at the value and it's already on 50GB. I looked a little more and
> it seems like after restarting the cloudstack-management service I could no
> longer reach the console or storage system vm.... looking at the host, now
> the cloud0 interface is gone. I am using OVS, but according to a Shapeblue
> article, no extra config is needed for the cloud0 interface.
>
>
>
> On Tue, Jun 12, 2018 at 6:17 PM, McClune, James <
> mcclunej@norwalktruckers.net> wrote:
>
> > Hi Christoffer,
> >
> > Did you try setting the max.template.iso.size to a higher value (e.g.
> > 50GB)?
> >
> > I think the default is set pretty low.
> >
> > Best,
> > James
> >
> > On Tue, Jun 12, 2018 at 11:10 AM, Christoffer Pedersen <vr...@vrod.dk>
> > wrote:
> >
> > > Just want to update that I still get the ISO error:
> > >
> > > 2018-06-12 17:09:42,194 INFO [c.c.t.HypervisorTemplateAdapter]
> > > (qtp1096283470-12:ctx-962ec49c ctx-2c82c0e2) (logid:0c7df1d7) Image
> > > store doesn't have enough capacity. Skip downloading template to
> > > this image
> > store
> > > 1
> > >
> > > Thanks!
> > >
> > > On Tue, Jun 12, 2018 at 5:07 PM, Christoffer Pedersen <vr...@vrod.dk>
> > > wrote:
> > >
> > >> Hi Dag,
> > >>
> > >> Yes, I altered the IP addresses as I do not fancy throwing them out
> > >> on the public net. If you think they have value to the
> > >> troubleshooting, I
> > can
> > >> send you the original logdata directly. I configured the networking
> > >> with OVS and as follows:
> > >>
> > >> cloudbr0 - MGMT0 (management interface, VLAN1000), this also hosts
> > >> the system VMs I guess. In the Zone setup, I labelled the
> > >> Management
> > network as
> > >> MGMT0, I guess that's OK? It's a Internal network where other
> > >> servers
> > are
> > >> also connected. I can ping the VMs here, also over a S2S connection
> > >> cloudbr1 - Public (VLAN 4000) and Guest network (VLAN 500-999). I
> > >> use real public IP-addresses from a scope and I am able to ping the
> > addresses
> > >> of both the CPVM and SSVM from my home.
> > >>
> > >> I have to update the situation though...
> > >>
> > >> Somehow at 13:02, I had the last error from the agent. Now, I do
> > >> not
> > have
> > >> any other errors and both VMs are now showing as "running" as well
> > >> as
> > agent
> > >> being "up" in the UI... had these errors since yesterday and did
> > >> not change anything after 9:00 this morning. Maybe I'm impatient
> > >> but 4 hours seems a bit long for them to get to work. :)
> > >>
> > >> Console VM works though at least:
> > >>
> > >>
> > >>
> > >> My impression is that cloudstack needs a while to get hold of
> things...
> > >> or am I just experiencing unusual things? I have a question though:
> > >> When adding an ISO this morning, I had an error about that there
> > >> was no space left (though the storage is 20TB). Was this because
> > >> the SSVM was not running at the time?
> > >>
> > >> Thank you!
> > >> Chris
> > >>
> > >> On Tue, Jun 12, 2018 at 4:39 PM, Dag Sonstebo <
> > Dag.Sonstebo@shapeblue.com
> > >> > wrote:
> > >>
> > >>> Chris,
> > >>>
> > >>> Going off in a slightly different direction to previous answers. I
> > >>> suspect your problem is with networking - how have you configured
> this?
> > >>> When you say you can ping the SSVM on the private interface which
> > >>> IP address do you use and where do you successfully ping from?
> > >>>
> > >>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/
> > patchviasocket.py
> > >>> -n
> > >>> v-1-VM -p
> > >>> %template=domP%type=consoleproxy%host=1.1.1.1%port=8250%name
> > >>> =v-1-VM%zone=1%pod=1%guid=Proxy.1%proxy_vm=1%disable_rp_filt
> > >>> er=true%eth2ip=9.9.9.9%eth2mask=255.255.255.0%gateway=9.9.9.
> > >>> 1%eth0ip=169.254.3.159%eth0mask=255.255.0.0%eth1ip=6.
> > >>> 6.6.6%eth1mask=255.255.255.0%mgmtcidr=
> > >>> 9.9.9.0/24%localgw=1.2.3.4%internaldns1=1.2.3.4%dns1=8.8.8.8
> > >>> %dns2=8.8.4.4
> > >>>
> > >>> It could be you have edited the above IP addresses to mask your
> > >>> real addresses – if so ignore this.
> > >>>
> > >>> If not then the above points to:
> > >>> - Management host is on 1.1.1.1
> > >>> - Eth2 which for a console proxy is public traffic is on
> > >>> 9.9.9.9/24
> > >>> - Eth0 which is the link local management interface is on
> > >>> 169.254.3.159/16 (system generated)
> > >>> - Eth1 is the main management interface on 6.6.6.6/24
> > >>> - You have a gateway address of 1.2.3.4
> > >>>
> > >>> So in this case – the CPVM can not check in to the management host
> > >>> on
> > >>> 1.1.1.1 - It’s got no interface on that subnet and it also has a
> > gateway
> > >>> it’s not able to reach.
> > >>>
> > >>> Regards,
> > >>> Dag Sonstebo
> > >>> Cloud Architect
> > >>> ShapeBlue
> > >>>
> > >>> On 12/06/2018, 13:12, "Nicolas Bouige" <n....@dimsi.fr> wrote:
> > >>>
> > >>> Hi Ivan,
> > >>>
> > >>>
> > >>> Are you talking about this global parameters :
> > >>>
> > >>> router.aggregation.command.each.timeout
> > >>>
> > >>>
> > >>>
> > >>> Best regards,
> > >>>
> > >>> Nicolas Bouige
> > >>> DIMSI
> > >>> cloud.dimsi.fr<http://www.cloud.dimsi.fr>
> > >>> 4, avenue Laurent Cely
> > >>> <https://maps.google.com/?q=4,+avenue+Laurent+Cely&entry=
> > gmail&source=g>
> > >>> Tour d’Asnière – 92600 Asnière sur Seine
> > >>> T/ +33 (0)6 28 98 53 40
> > >>>
> > >>>
> > >>> ________________________________
> > >>> De : Ivan Kudryavtsev <ku...@bw-sw.com>
> > >>> Envoyé : mardi 12 juin 2018 13:59:39
> > >>> À : users
> > >>> Objet : Re: SSVM's not starting, timeout for libvirt python
> > >>> script in agent.log
> > >>>
> > >>> Increasing command timeouts in global parameters can work
> > >>> here. At least I
> > >>> met similar behaviour with VR.
> > >>>
> > >>> вт, 12 июн. 2018 г., 14:39 Christoffer Pedersen <vr...@vrod.dk>:
> > >>>
> > >>> > Hi Nicolas,
> > >>> >
> > >>> > I did a apt show qemu and it gave me this version:
> > >>> >
> > >>> > Version: 1:2.5+dfsg-5ubuntu10.29
> > >>> >
> > >>> > So I guess tha would be version 2.5?
> > >>> >
> > >>>
> > >>> Dag.Sonstebo@shapeblue.com
> > >>> www.shapeblue.com
> > >>> 53 Chandos Place, Covent Garden, London
> > >>> <https://maps.google.com/?q=53+Chandos+Place,+Covent+
> > Garden,+London&entry=gmail&source=g>
> > >>> WC2N 4HSUK
> > >>> @shapeblue
> > >>>
> > >>>
> > >>>
> > >>> > On Tue, Jun 12, 2018 at 1:04 PM, Nicolas Bouige
> > >>> > <n....@dimsi.fr>
> > >>> wrote:
> > >>> >
> > >>> > > Hello Christoffer,
> > >>> > >
> > >>> > >
> > >>> > > Could you tell us wich qemu version are you using ?
> > >>> > >
> > >>> > > Nicolas Bouige
> > >>> > > DIMSI
> > >>> > > cloud.dimsi.fr<http://www.cloud.dimsi.fr>
> > >>> > > 4, avenue Laurent Cely
> > >>> <https://maps.google.com/?q=4,+avenue+Laurent+Cely&entry=
> > gmail&source=g>
> > >>> > > Tour d’Asnière – 92600 Asnière sur Seine
> > >>> > > T/ +33 (0)6 28 98 53 40
> > >>> > >
> > >>> > >
> > >>> > > ________________________________
> > >>> > > De : Christoffer Pedersen <vr...@vrod.dk>
> > >>> > > Envoyé : mardi 12 juin 2018 12:30:48
> > >>> > > À : users@cloudstack.apache.org
> > >>> > > Objet : SSVM's not starting, timeout for libvirt python
> > >>> script
> > in
> > >>> > agent.log
> > >>> > >
> > >>> > > Hi all,
> > >>> > >
> > >>> > > I have an issue regarding the system VMs. After deploying
> > >>> an advanced
> > >>> > zone,
> > >>> > > the system VMs are trying to be created but gets stuck in
> > >>> a "Starting"
> > >>> > > state, however the Agent state is "Up". I have these logs
> > >>> in
> > the
> > >>> > agent.log
> > >>> > > (sorry for the formatting)
> > >>> > >
> > >>> > > 2018-06-12 12:22:06,354 WARN
> > >>> [kvm.resource.LibvirtComputing Resource]
> > >>> > > (Script-8:null) (logid:) Interrupting script.
> > >>> > > 2018-06-12 12:22:06,355 WARN
> > >>> [kvm.resource.LibvirtComputing Resource]
> > >>> > > (agentRequest-Handler-4:null) (logid:ea9cb55a) Timed out:
> > >>> > >
> > >>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patch
> > >>> viasocket.py
> > >>> > > -n
> > >>> > > v-1-VM -p
> > >>> > > %template=domP%type=consoleproxy%host=1.1.1.1%
> > >>> > > port=8250%name=v-1-VM%zone=1%pod=1%guid=Proxy.1%proxy_vm=1%
> > >>> > > disable_rp_filter=true%eth2ip=9.9.9.9%eth2mask=255.255.255.
> > >>> > > 0%gateway=9.9.9.1%eth0ip=169.254.3.159%eth0mask=255.255.0.
> > >>> > > 0%eth1ip=6.6.6.6%eth1mask=255.255.255.0%mgmtcidr=
> > >>> > >
> > >>> > 9.9.9.0/24%localgw=1.2.3.4%internaldns1=1.2.3.4%dns1=8.8.8.8
> > >>> %dns2=8.8.4.4
> > >>> > > . Output is:
> > >>> > > 2018-06-12 12:22:06,355 ERROR
> > >>> [kvm.resource.LibvirtComputing Resource]
> > >>> > > (agentRequest-Handler-4:null) (logid:ea9cb55a) passcmd
> > >>> failed:timeout
> > >>> > > 2018-06-12 12:22:08,914 WARN
> > >>> [kvm.resource.LibvirtComputing Resource]
> > >>> > > (Script-4:null) (logid:) Interrupting script.
> > >>> > > 2018-06-12 12:22:08,915 WARN
> > >>> [kvm.resource.LibvirtComputing Resource]
> > >>> > > (agentRequest-Handler-5:null) (logid:8e44093e) Timed out:
> > >>> > >
> > >>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patch
> > >>> viasocket.py
> > >>> > > -n
> > >>> > > s-2-VM -p
> > >>> > > %template=domP%type=secstorage%host=1.1.1.1%port=
> > >>> > >
> > >>> 8250%name=s-2-VM%zone=1%pod=1%guid=s-2-VM%workers=5%resource
> > >>> =org.apache.
> > >>> > > cloudstack.storage.resource.NfsSecondaryStorageResource%
> > >>> > > instance=SecStorage%sslcopy=false%role=templateProcessor%
> > >>> > >
> > >>> mtu=1500%eth2ip=7.7.7.7%eth2mask=255.255.255.0%gateway=9.9.9
> > >>> .1%public.
> > >>> > > network.device=eth2%eth0ip=169.254.2.193%eth0mask=255.
> > >>> > > 255.0.0%eth1ip=10.120.0.61%eth1mask=255.255.255.0%mgmtcidr=
> > >>> > > 9.9.9.0/24%localgw=1.2.3.4%private.network.device=eth1%
> > >>> > > internaldns1=1.2.3.4%dns1=8.8.8.8%dns2=8.8.4.4%nfsVersion=
> null
> > >>> > > . Output is:
> > >>> > > 2018-06-12 12:22:08,915 ERROR
> > >>> [kvm.resource.LibvirtComputing Resource]
> > >>> > > (agentRequest-Handler-5:null) (logid:8e44093e) passcmd
> > >>> failed:timeout
> > >>> > >
> > >>> > > I have seen this error around but did not really find a
> > solution
> > >>> to it. I
> > >>> > > am not exactly sure whats "timing" out? I can ping both
> > >>> SSVM's on their
> > >>> > > private and public interface.
> > >>> > >
> > >>> > > I hope someone can help me out here. :)
> > >>> > >
> > >>> > > --
> > >>> > > Thanks,
> > >>> > > Chris pedersen
> > >>> > >
> > >>> >
> > >>> >
> > >>> >
> > >>> > --
> > >>> > Thanks,
> > >>> > Chris pedersen
> > >>> >
> > >>>
> > >>>
> > >>>
> > >>
> > >>
> > >> --
> > >> Thanks,
> > >> Chris pedersen
> > >>
> > >
> > >
> > >
> > > --
> > > Thanks,
> > > Chris pedersen
> > >
> >
>
>
>
> --
> Thanks,
> Chris pedersen
>
--
Thanks,
Chris pedersen
RE: SSVM's not starting, timeout for libvirt python script in
agent.log
Posted by Nicolas Bouige <n....@dimsi.fr>.
Hello All,
Maybe is not related to your case but in case of....as we don't use the same OS
I was facing the exact same error during a new creation of virtual router.
The only change i made was to update my qemu version on our KVM (centOS7 + CS 4.11) from 1.5 to 2.3
Before that VR creation was working fine.
Apperently some users got the same issue but with an old version of CS (4.9)
http://dev.cloudstack.apache.narkive.com/qsKmnsa4/patchviasocket-seems-to-be-broken-with-qemu-2-3
-----Message d'origine-----
De : Christoffer Pedersen [mailto:vrod@vrod.dk]
Envoyé : mardi 12 juin 2018 19:17
À : users@cloudstack.apache.org
Objet : Re: SSVM's not starting, timeout for libvirt python script in agent.log
Hi James,
I looked at the value and it's already on 50GB. I looked a little more and it seems like after restarting the cloudstack-management service I could no longer reach the console or storage system vm.... looking at the host, now the cloud0 interface is gone. I am using OVS, but according to a Shapeblue article, no extra config is needed for the cloud0 interface.
On Tue, Jun 12, 2018 at 6:17 PM, McClune, James < mcclunej@norwalktruckers.net> wrote:
> Hi Christoffer,
>
> Did you try setting the max.template.iso.size to a higher value (e.g.
> 50GB)?
>
> I think the default is set pretty low.
>
> Best,
> James
>
> On Tue, Jun 12, 2018 at 11:10 AM, Christoffer Pedersen <vr...@vrod.dk>
> wrote:
>
> > Just want to update that I still get the ISO error:
> >
> > 2018-06-12 17:09:42,194 INFO [c.c.t.HypervisorTemplateAdapter]
> > (qtp1096283470-12:ctx-962ec49c ctx-2c82c0e2) (logid:0c7df1d7) Image
> > store doesn't have enough capacity. Skip downloading template to
> > this image
> store
> > 1
> >
> > Thanks!
> >
> > On Tue, Jun 12, 2018 at 5:07 PM, Christoffer Pedersen <vr...@vrod.dk>
> > wrote:
> >
> >> Hi Dag,
> >>
> >> Yes, I altered the IP addresses as I do not fancy throwing them out
> >> on the public net. If you think they have value to the
> >> troubleshooting, I
> can
> >> send you the original logdata directly. I configured the networking
> >> with OVS and as follows:
> >>
> >> cloudbr0 - MGMT0 (management interface, VLAN1000), this also hosts
> >> the system VMs I guess. In the Zone setup, I labelled the
> >> Management
> network as
> >> MGMT0, I guess that's OK? It's a Internal network where other
> >> servers
> are
> >> also connected. I can ping the VMs here, also over a S2S connection
> >> cloudbr1 - Public (VLAN 4000) and Guest network (VLAN 500-999). I
> >> use real public IP-addresses from a scope and I am able to ping the
> addresses
> >> of both the CPVM and SSVM from my home.
> >>
> >> I have to update the situation though...
> >>
> >> Somehow at 13:02, I had the last error from the agent. Now, I do
> >> not
> have
> >> any other errors and both VMs are now showing as "running" as well
> >> as
> agent
> >> being "up" in the UI... had these errors since yesterday and did
> >> not change anything after 9:00 this morning. Maybe I'm impatient
> >> but 4 hours seems a bit long for them to get to work. :)
> >>
> >> Console VM works though at least:
> >>
> >>
> >>
> >> My impression is that cloudstack needs a while to get hold of things...
> >> or am I just experiencing unusual things? I have a question though:
> >> When adding an ISO this morning, I had an error about that there
> >> was no space left (though the storage is 20TB). Was this because
> >> the SSVM was not running at the time?
> >>
> >> Thank you!
> >> Chris
> >>
> >> On Tue, Jun 12, 2018 at 4:39 PM, Dag Sonstebo <
> Dag.Sonstebo@shapeblue.com
> >> > wrote:
> >>
> >>> Chris,
> >>>
> >>> Going off in a slightly different direction to previous answers. I
> >>> suspect your problem is with networking - how have you configured this?
> >>> When you say you can ping the SSVM on the private interface which
> >>> IP address do you use and where do you successfully ping from?
> >>>
> >>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/
> patchviasocket.py
> >>> -n
> >>> v-1-VM -p
> >>> %template=domP%type=consoleproxy%host=1.1.1.1%port=8250%name
> >>> =v-1-VM%zone=1%pod=1%guid=Proxy.1%proxy_vm=1%disable_rp_filt
> >>> er=true%eth2ip=9.9.9.9%eth2mask=255.255.255.0%gateway=9.9.9.
> >>> 1%eth0ip=169.254.3.159%eth0mask=255.255.0.0%eth1ip=6.
> >>> 6.6.6%eth1mask=255.255.255.0%mgmtcidr=
> >>> 9.9.9.0/24%localgw=1.2.3.4%internaldns1=1.2.3.4%dns1=8.8.8.8
> >>> %dns2=8.8.4.4
> >>>
> >>> It could be you have edited the above IP addresses to mask your
> >>> real addresses – if so ignore this.
> >>>
> >>> If not then the above points to:
> >>> - Management host is on 1.1.1.1
> >>> - Eth2 which for a console proxy is public traffic is on
> >>> 9.9.9.9/24
> >>> - Eth0 which is the link local management interface is on
> >>> 169.254.3.159/16 (system generated)
> >>> - Eth1 is the main management interface on 6.6.6.6/24
> >>> - You have a gateway address of 1.2.3.4
> >>>
> >>> So in this case – the CPVM can not check in to the management host
> >>> on
> >>> 1.1.1.1 - It’s got no interface on that subnet and it also has a
> gateway
> >>> it’s not able to reach.
> >>>
> >>> Regards,
> >>> Dag Sonstebo
> >>> Cloud Architect
> >>> ShapeBlue
> >>>
> >>> On 12/06/2018, 13:12, "Nicolas Bouige" <n....@dimsi.fr> wrote:
> >>>
> >>> Hi Ivan,
> >>>
> >>>
> >>> Are you talking about this global parameters :
> >>>
> >>> router.aggregation.command.each.timeout
> >>>
> >>>
> >>>
> >>> Best regards,
> >>>
> >>> Nicolas Bouige
> >>> DIMSI
> >>> cloud.dimsi.fr<http://www.cloud.dimsi.fr>
> >>> 4, avenue Laurent Cely
> >>> <https://maps.google.com/?q=4,+avenue+Laurent+Cely&entry=
> gmail&source=g>
> >>> Tour d’Asnière – 92600 Asnière sur Seine
> >>> T/ +33 (0)6 28 98 53 40
> >>>
> >>>
> >>> ________________________________
> >>> De : Ivan Kudryavtsev <ku...@bw-sw.com>
> >>> Envoyé : mardi 12 juin 2018 13:59:39
> >>> À : users
> >>> Objet : Re: SSVM's not starting, timeout for libvirt python
> >>> script in agent.log
> >>>
> >>> Increasing command timeouts in global parameters can work
> >>> here. At least I
> >>> met similar behaviour with VR.
> >>>
> >>> вт, 12 июн. 2018 г., 14:39 Christoffer Pedersen <vr...@vrod.dk>:
> >>>
> >>> > Hi Nicolas,
> >>> >
> >>> > I did a apt show qemu and it gave me this version:
> >>> >
> >>> > Version: 1:2.5+dfsg-5ubuntu10.29
> >>> >
> >>> > So I guess tha would be version 2.5?
> >>> >
> >>>
> >>> Dag.Sonstebo@shapeblue.com
> >>> www.shapeblue.com
> >>> 53 Chandos Place, Covent Garden, London
> >>> <https://maps.google.com/?q=53+Chandos+Place,+Covent+
> Garden,+London&entry=gmail&source=g>
> >>> WC2N 4HSUK
> >>> @shapeblue
> >>>
> >>>
> >>>
> >>> > On Tue, Jun 12, 2018 at 1:04 PM, Nicolas Bouige
> >>> > <n....@dimsi.fr>
> >>> wrote:
> >>> >
> >>> > > Hello Christoffer,
> >>> > >
> >>> > >
> >>> > > Could you tell us wich qemu version are you using ?
> >>> > >
> >>> > > Nicolas Bouige
> >>> > > DIMSI
> >>> > > cloud.dimsi.fr<http://www.cloud.dimsi.fr>
> >>> > > 4, avenue Laurent Cely
> >>> <https://maps.google.com/?q=4,+avenue+Laurent+Cely&entry=
> gmail&source=g>
> >>> > > Tour d’Asnière – 92600 Asnière sur Seine
> >>> > > T/ +33 (0)6 28 98 53 40
> >>> > >
> >>> > >
> >>> > > ________________________________
> >>> > > De : Christoffer Pedersen <vr...@vrod.dk>
> >>> > > Envoyé : mardi 12 juin 2018 12:30:48
> >>> > > À : users@cloudstack.apache.org
> >>> > > Objet : SSVM's not starting, timeout for libvirt python
> >>> script
> in
> >>> > agent.log
> >>> > >
> >>> > > Hi all,
> >>> > >
> >>> > > I have an issue regarding the system VMs. After deploying
> >>> an advanced
> >>> > zone,
> >>> > > the system VMs are trying to be created but gets stuck in
> >>> a "Starting"
> >>> > > state, however the Agent state is "Up". I have these logs
> >>> in
> the
> >>> > agent.log
> >>> > > (sorry for the formatting)
> >>> > >
> >>> > > 2018-06-12 12:22:06,354 WARN
> >>> [kvm.resource.LibvirtComputing Resource]
> >>> > > (Script-8:null) (logid:) Interrupting script.
> >>> > > 2018-06-12 12:22:06,355 WARN
> >>> [kvm.resource.LibvirtComputing Resource]
> >>> > > (agentRequest-Handler-4:null) (logid:ea9cb55a) Timed out:
> >>> > >
> >>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patch
> >>> viasocket.py
> >>> > > -n
> >>> > > v-1-VM -p
> >>> > > %template=domP%type=consoleproxy%host=1.1.1.1%
> >>> > > port=8250%name=v-1-VM%zone=1%pod=1%guid=Proxy.1%proxy_vm=1%
> >>> > > disable_rp_filter=true%eth2ip=9.9.9.9%eth2mask=255.255.255.
> >>> > > 0%gateway=9.9.9.1%eth0ip=169.254.3.159%eth0mask=255.255.0.
> >>> > > 0%eth1ip=6.6.6.6%eth1mask=255.255.255.0%mgmtcidr=
> >>> > >
> >>> > 9.9.9.0/24%localgw=1.2.3.4%internaldns1=1.2.3.4%dns1=8.8.8.8
> >>> %dns2=8.8.4.4
> >>> > > . Output is:
> >>> > > 2018-06-12 12:22:06,355 ERROR
> >>> [kvm.resource.LibvirtComputing Resource]
> >>> > > (agentRequest-Handler-4:null) (logid:ea9cb55a) passcmd
> >>> failed:timeout
> >>> > > 2018-06-12 12:22:08,914 WARN
> >>> [kvm.resource.LibvirtComputing Resource]
> >>> > > (Script-4:null) (logid:) Interrupting script.
> >>> > > 2018-06-12 12:22:08,915 WARN
> >>> [kvm.resource.LibvirtComputing Resource]
> >>> > > (agentRequest-Handler-5:null) (logid:8e44093e) Timed out:
> >>> > >
> >>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patch
> >>> viasocket.py
> >>> > > -n
> >>> > > s-2-VM -p
> >>> > > %template=domP%type=secstorage%host=1.1.1.1%port=
> >>> > >
> >>> 8250%name=s-2-VM%zone=1%pod=1%guid=s-2-VM%workers=5%resource
> >>> =org.apache.
> >>> > > cloudstack.storage.resource.NfsSecondaryStorageResource%
> >>> > > instance=SecStorage%sslcopy=false%role=templateProcessor%
> >>> > >
> >>> mtu=1500%eth2ip=7.7.7.7%eth2mask=255.255.255.0%gateway=9.9.9
> >>> .1%public.
> >>> > > network.device=eth2%eth0ip=169.254.2.193%eth0mask=255.
> >>> > > 255.0.0%eth1ip=10.120.0.61%eth1mask=255.255.255.0%mgmtcidr=
> >>> > > 9.9.9.0/24%localgw=1.2.3.4%private.network.device=eth1%
> >>> > > internaldns1=1.2.3.4%dns1=8.8.8.8%dns2=8.8.4.4%nfsVersion=null
> >>> > > . Output is:
> >>> > > 2018-06-12 12:22:08,915 ERROR
> >>> [kvm.resource.LibvirtComputing Resource]
> >>> > > (agentRequest-Handler-5:null) (logid:8e44093e) passcmd
> >>> failed:timeout
> >>> > >
> >>> > > I have seen this error around but did not really find a
> solution
> >>> to it. I
> >>> > > am not exactly sure whats "timing" out? I can ping both
> >>> SSVM's on their
> >>> > > private and public interface.
> >>> > >
> >>> > > I hope someone can help me out here. :)
> >>> > >
> >>> > > --
> >>> > > Thanks,
> >>> > > Chris pedersen
> >>> > >
> >>> >
> >>> >
> >>> >
> >>> > --
> >>> > Thanks,
> >>> > Chris pedersen
> >>> >
> >>>
> >>>
> >>>
> >>
> >>
> >> --
> >> Thanks,
> >> Chris pedersen
> >>
> >
> >
> >
> > --
> > Thanks,
> > Chris pedersen
> >
>
--
Thanks,
Chris pedersen
Re: SSVM's not starting, timeout for libvirt python script in agent.log
Posted by Christoffer Pedersen <vr...@vrod.dk>.
Hi James,
I looked at the value and it's already on 50GB. I looked a little more and
it seems like after restarting the cloudstack-management service I could no
longer reach the console or storage system vm.... looking at the host, now
the cloud0 interface is gone. I am using OVS, but according to a Shapeblue
article, no extra config is needed for the cloud0 interface.
On Tue, Jun 12, 2018 at 6:17 PM, McClune, James <
mcclunej@norwalktruckers.net> wrote:
> Hi Christoffer,
>
> Did you try setting the max.template.iso.size to a higher value (e.g.
> 50GB)?
>
> I think the default is set pretty low.
>
> Best,
> James
>
> On Tue, Jun 12, 2018 at 11:10 AM, Christoffer Pedersen <vr...@vrod.dk>
> wrote:
>
> > Just want to update that I still get the ISO error:
> >
> > 2018-06-12 17:09:42,194 INFO [c.c.t.HypervisorTemplateAdapter]
> > (qtp1096283470-12:ctx-962ec49c ctx-2c82c0e2) (logid:0c7df1d7) Image store
> > doesn't have enough capacity. Skip downloading template to this image
> store
> > 1
> >
> > Thanks!
> >
> > On Tue, Jun 12, 2018 at 5:07 PM, Christoffer Pedersen <vr...@vrod.dk>
> > wrote:
> >
> >> Hi Dag,
> >>
> >> Yes, I altered the IP addresses as I do not fancy throwing them out on
> >> the public net. If you think they have value to the troubleshooting, I
> can
> >> send you the original logdata directly. I configured the networking with
> >> OVS and as follows:
> >>
> >> cloudbr0 - MGMT0 (management interface, VLAN1000), this also hosts the
> >> system VMs I guess. In the Zone setup, I labelled the Management
> network as
> >> MGMT0, I guess that's OK? It's a Internal network where other servers
> are
> >> also connected. I can ping the VMs here, also over a S2S connection
> >> cloudbr1 - Public (VLAN 4000) and Guest network (VLAN 500-999). I use
> >> real public IP-addresses from a scope and I am able to ping the
> addresses
> >> of both the CPVM and SSVM from my home.
> >>
> >> I have to update the situation though...
> >>
> >> Somehow at 13:02, I had the last error from the agent. Now, I do not
> have
> >> any other errors and both VMs are now showing as "running" as well as
> agent
> >> being "up" in the UI... had these errors since yesterday and did not
> >> change anything after 9:00 this morning. Maybe I'm impatient but 4 hours
> >> seems a bit long for them to get to work. :)
> >>
> >> Console VM works though at least:
> >>
> >>
> >>
> >> My impression is that cloudstack needs a while to get hold of things...
> >> or am I just experiencing unusual things? I have a question though: When
> >> adding an ISO this morning, I had an error about that there was no space
> >> left (though the storage is 20TB). Was this because the SSVM was not
> >> running at the time?
> >>
> >> Thank you!
> >> Chris
> >>
> >> On Tue, Jun 12, 2018 at 4:39 PM, Dag Sonstebo <
> Dag.Sonstebo@shapeblue.com
> >> > wrote:
> >>
> >>> Chris,
> >>>
> >>> Going off in a slightly different direction to previous answers. I
> >>> suspect your problem is with networking - how have you configured this?
> >>> When you say you can ping the SSVM on the private interface which IP
> >>> address do you use and where do you successfully ping from?
> >>>
> >>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/
> patchviasocket.py
> >>> -n
> >>> v-1-VM -p
> >>> %template=domP%type=consoleproxy%host=1.1.1.1%port=8250%name
> >>> =v-1-VM%zone=1%pod=1%guid=Proxy.1%proxy_vm=1%disable_rp_filt
> >>> er=true%eth2ip=9.9.9.9%eth2mask=255.255.255.0%gateway=9.9.9.
> >>> 1%eth0ip=169.254.3.159%eth0mask=255.255.0.0%eth1ip=6.
> >>> 6.6.6%eth1mask=255.255.255.0%mgmtcidr=
> >>> 9.9.9.0/24%localgw=1.2.3.4%internaldns1=1.2.3.4%dns1=8.8.8.8
> >>> %dns2=8.8.4.4
> >>>
> >>> It could be you have edited the above IP addresses to mask your real
> >>> addresses – if so ignore this.
> >>>
> >>> If not then the above points to:
> >>> - Management host is on 1.1.1.1
> >>> - Eth2 which for a console proxy is public traffic is on 9.9.9.9/24
> >>> - Eth0 which is the link local management interface is on
> >>> 169.254.3.159/16 (system generated)
> >>> - Eth1 is the main management interface on 6.6.6.6/24
> >>> - You have a gateway address of 1.2.3.4
> >>>
> >>> So in this case – the CPVM can not check in to the management host on
> >>> 1.1.1.1 - It’s got no interface on that subnet and it also has a
> gateway
> >>> it’s not able to reach.
> >>>
> >>> Regards,
> >>> Dag Sonstebo
> >>> Cloud Architect
> >>> ShapeBlue
> >>>
> >>> On 12/06/2018, 13:12, "Nicolas Bouige" <n....@dimsi.fr> wrote:
> >>>
> >>> Hi Ivan,
> >>>
> >>>
> >>> Are you talking about this global parameters :
> >>>
> >>> router.aggregation.command.each.timeout
> >>>
> >>>
> >>>
> >>> Best regards,
> >>>
> >>> Nicolas Bouige
> >>> DIMSI
> >>> cloud.dimsi.fr<http://www.cloud.dimsi.fr>
> >>> 4, avenue Laurent Cely
> >>> <https://maps.google.com/?q=4,+avenue+Laurent+Cely&entry=
> gmail&source=g>
> >>> Tour d’Asnière – 92600 Asnière sur Seine
> >>> T/ +33 (0)6 28 98 53 40
> >>>
> >>>
> >>> ________________________________
> >>> De : Ivan Kudryavtsev <ku...@bw-sw.com>
> >>> Envoyé : mardi 12 juin 2018 13:59:39
> >>> À : users
> >>> Objet : Re: SSVM's not starting, timeout for libvirt python script
> >>> in agent.log
> >>>
> >>> Increasing command timeouts in global parameters can work here. At
> >>> least I
> >>> met similar behaviour with VR.
> >>>
> >>> вт, 12 июн. 2018 г., 14:39 Christoffer Pedersen <vr...@vrod.dk>:
> >>>
> >>> > Hi Nicolas,
> >>> >
> >>> > I did a apt show qemu and it gave me this version:
> >>> >
> >>> > Version: 1:2.5+dfsg-5ubuntu10.29
> >>> >
> >>> > So I guess tha would be version 2.5?
> >>> >
> >>>
> >>> Dag.Sonstebo@shapeblue.com
> >>> www.shapeblue.com
> >>> 53 Chandos Place, Covent Garden, London
> >>> <https://maps.google.com/?q=53+Chandos+Place,+Covent+
> Garden,+London&entry=gmail&source=g>
> >>> WC2N 4HSUK
> >>> @shapeblue
> >>>
> >>>
> >>>
> >>> > On Tue, Jun 12, 2018 at 1:04 PM, Nicolas Bouige <n....@dimsi.fr>
> >>> wrote:
> >>> >
> >>> > > Hello Christoffer,
> >>> > >
> >>> > >
> >>> > > Could you tell us wich qemu version are you using ?
> >>> > >
> >>> > > Nicolas Bouige
> >>> > > DIMSI
> >>> > > cloud.dimsi.fr<http://www.cloud.dimsi.fr>
> >>> > > 4, avenue Laurent Cely
> >>> <https://maps.google.com/?q=4,+avenue+Laurent+Cely&entry=
> gmail&source=g>
> >>> > > Tour d’Asnière – 92600 Asnière sur Seine
> >>> > > T/ +33 (0)6 28 98 53 40
> >>> > >
> >>> > >
> >>> > > ________________________________
> >>> > > De : Christoffer Pedersen <vr...@vrod.dk>
> >>> > > Envoyé : mardi 12 juin 2018 12:30:48
> >>> > > À : users@cloudstack.apache.org
> >>> > > Objet : SSVM's not starting, timeout for libvirt python script
> in
> >>> > agent.log
> >>> > >
> >>> > > Hi all,
> >>> > >
> >>> > > I have an issue regarding the system VMs. After deploying an
> >>> advanced
> >>> > zone,
> >>> > > the system VMs are trying to be created but gets stuck in a
> >>> "Starting"
> >>> > > state, however the Agent state is "Up". I have these logs in
> the
> >>> > agent.log
> >>> > > (sorry for the formatting)
> >>> > >
> >>> > > 2018-06-12 12:22:06,354 WARN [kvm.resource.LibvirtComputing
> >>> Resource]
> >>> > > (Script-8:null) (logid:) Interrupting script.
> >>> > > 2018-06-12 12:22:06,355 WARN [kvm.resource.LibvirtComputing
> >>> Resource]
> >>> > > (agentRequest-Handler-4:null) (logid:ea9cb55a) Timed out:
> >>> > > /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patch
> >>> viasocket.py
> >>> > > -n
> >>> > > v-1-VM -p
> >>> > > %template=domP%type=consoleproxy%host=1.1.1.1%
> >>> > > port=8250%name=v-1-VM%zone=1%pod=1%guid=Proxy.1%proxy_vm=1%
> >>> > > disable_rp_filter=true%eth2ip=9.9.9.9%eth2mask=255.255.255.
> >>> > > 0%gateway=9.9.9.1%eth0ip=169.254.3.159%eth0mask=255.255.0.
> >>> > > 0%eth1ip=6.6.6.6%eth1mask=255.255.255.0%mgmtcidr=
> >>> > >
> >>> > 9.9.9.0/24%localgw=1.2.3.4%internaldns1=1.2.3.4%dns1=8.8.8.8
> >>> %dns2=8.8.4.4
> >>> > > . Output is:
> >>> > > 2018-06-12 12:22:06,355 ERROR [kvm.resource.LibvirtComputing
> >>> Resource]
> >>> > > (agentRequest-Handler-4:null) (logid:ea9cb55a) passcmd
> >>> failed:timeout
> >>> > > 2018-06-12 12:22:08,914 WARN [kvm.resource.LibvirtComputing
> >>> Resource]
> >>> > > (Script-4:null) (logid:) Interrupting script.
> >>> > > 2018-06-12 12:22:08,915 WARN [kvm.resource.LibvirtComputing
> >>> Resource]
> >>> > > (agentRequest-Handler-5:null) (logid:8e44093e) Timed out:
> >>> > > /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patch
> >>> viasocket.py
> >>> > > -n
> >>> > > s-2-VM -p
> >>> > > %template=domP%type=secstorage%host=1.1.1.1%port=
> >>> > > 8250%name=s-2-VM%zone=1%pod=1%guid=s-2-VM%workers=5%resource
> >>> =org.apache.
> >>> > > cloudstack.storage.resource.NfsSecondaryStorageResource%
> >>> > > instance=SecStorage%sslcopy=false%role=templateProcessor%
> >>> > > mtu=1500%eth2ip=7.7.7.7%eth2mask=255.255.255.0%gateway=9.9.9
> >>> .1%public.
> >>> > > network.device=eth2%eth0ip=169.254.2.193%eth0mask=255.
> >>> > > 255.0.0%eth1ip=10.120.0.61%eth1mask=255.255.255.0%mgmtcidr=
> >>> > > 9.9.9.0/24%localgw=1.2.3.4%private.network.device=eth1%
> >>> > > internaldns1=1.2.3.4%dns1=8.8.8.8%dns2=8.8.4.4%nfsVersion=null
> >>> > > . Output is:
> >>> > > 2018-06-12 12:22:08,915 ERROR [kvm.resource.LibvirtComputing
> >>> Resource]
> >>> > > (agentRequest-Handler-5:null) (logid:8e44093e) passcmd
> >>> failed:timeout
> >>> > >
> >>> > > I have seen this error around but did not really find a
> solution
> >>> to it. I
> >>> > > am not exactly sure whats "timing" out? I can ping both SSVM's
> >>> on their
> >>> > > private and public interface.
> >>> > >
> >>> > > I hope someone can help me out here. :)
> >>> > >
> >>> > > --
> >>> > > Thanks,
> >>> > > Chris pedersen
> >>> > >
> >>> >
> >>> >
> >>> >
> >>> > --
> >>> > Thanks,
> >>> > Chris pedersen
> >>> >
> >>>
> >>>
> >>>
> >>
> >>
> >> --
> >> Thanks,
> >> Chris pedersen
> >>
> >
> >
> >
> > --
> > Thanks,
> > Chris pedersen
> >
>
--
Thanks,
Chris pedersen
Re: SSVM's not starting, timeout for libvirt python script in agent.log
Posted by "McClune, James" <mc...@norwalktruckers.net>.
Hi Christoffer,
Did you try setting the max.template.iso.size to a higher value (e.g. 50GB)?
I think the default is set pretty low.
Best,
James
On Tue, Jun 12, 2018 at 11:10 AM, Christoffer Pedersen <vr...@vrod.dk> wrote:
> Just want to update that I still get the ISO error:
>
> 2018-06-12 17:09:42,194 INFO [c.c.t.HypervisorTemplateAdapter]
> (qtp1096283470-12:ctx-962ec49c ctx-2c82c0e2) (logid:0c7df1d7) Image store
> doesn't have enough capacity. Skip downloading template to this image store
> 1
>
> Thanks!
>
> On Tue, Jun 12, 2018 at 5:07 PM, Christoffer Pedersen <vr...@vrod.dk>
> wrote:
>
>> Hi Dag,
>>
>> Yes, I altered the IP addresses as I do not fancy throwing them out on
>> the public net. If you think they have value to the troubleshooting, I can
>> send you the original logdata directly. I configured the networking with
>> OVS and as follows:
>>
>> cloudbr0 - MGMT0 (management interface, VLAN1000), this also hosts the
>> system VMs I guess. In the Zone setup, I labelled the Management network as
>> MGMT0, I guess that's OK? It's a Internal network where other servers are
>> also connected. I can ping the VMs here, also over a S2S connection
>> cloudbr1 - Public (VLAN 4000) and Guest network (VLAN 500-999). I use
>> real public IP-addresses from a scope and I am able to ping the addresses
>> of both the CPVM and SSVM from my home.
>>
>> I have to update the situation though...
>>
>> Somehow at 13:02, I had the last error from the agent. Now, I do not have
>> any other errors and both VMs are now showing as "running" as well as agent
>> being "up" in the UI... had these errors since yesterday and did not
>> change anything after 9:00 this morning. Maybe I'm impatient but 4 hours
>> seems a bit long for them to get to work. :)
>>
>> Console VM works though at least:
>>
>>
>>
>> My impression is that cloudstack needs a while to get hold of things...
>> or am I just experiencing unusual things? I have a question though: When
>> adding an ISO this morning, I had an error about that there was no space
>> left (though the storage is 20TB). Was this because the SSVM was not
>> running at the time?
>>
>> Thank you!
>> Chris
>>
>> On Tue, Jun 12, 2018 at 4:39 PM, Dag Sonstebo <Dag.Sonstebo@shapeblue.com
>> > wrote:
>>
>>> Chris,
>>>
>>> Going off in a slightly different direction to previous answers. I
>>> suspect your problem is with networking - how have you configured this?
>>> When you say you can ping the SSVM on the private interface which IP
>>> address do you use and where do you successfully ping from?
>>>
>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.py
>>> -n
>>> v-1-VM -p
>>> %template=domP%type=consoleproxy%host=1.1.1.1%port=8250%name
>>> =v-1-VM%zone=1%pod=1%guid=Proxy.1%proxy_vm=1%disable_rp_filt
>>> er=true%eth2ip=9.9.9.9%eth2mask=255.255.255.0%gateway=9.9.9.
>>> 1%eth0ip=169.254.3.159%eth0mask=255.255.0.0%eth1ip=6.
>>> 6.6.6%eth1mask=255.255.255.0%mgmtcidr=
>>> 9.9.9.0/24%localgw=1.2.3.4%internaldns1=1.2.3.4%dns1=8.8.8.8
>>> %dns2=8.8.4.4
>>>
>>> It could be you have edited the above IP addresses to mask your real
>>> addresses – if so ignore this.
>>>
>>> If not then the above points to:
>>> - Management host is on 1.1.1.1
>>> - Eth2 which for a console proxy is public traffic is on 9.9.9.9/24
>>> - Eth0 which is the link local management interface is on
>>> 169.254.3.159/16 (system generated)
>>> - Eth1 is the main management interface on 6.6.6.6/24
>>> - You have a gateway address of 1.2.3.4
>>>
>>> So in this case – the CPVM can not check in to the management host on
>>> 1.1.1.1 - It’s got no interface on that subnet and it also has a gateway
>>> it’s not able to reach.
>>>
>>> Regards,
>>> Dag Sonstebo
>>> Cloud Architect
>>> ShapeBlue
>>>
>>> On 12/06/2018, 13:12, "Nicolas Bouige" <n....@dimsi.fr> wrote:
>>>
>>> Hi Ivan,
>>>
>>>
>>> Are you talking about this global parameters :
>>>
>>> router.aggregation.command.each.timeout
>>>
>>>
>>>
>>> Best regards,
>>>
>>> Nicolas Bouige
>>> DIMSI
>>> cloud.dimsi.fr<http://www.cloud.dimsi.fr>
>>> 4, avenue Laurent Cely
>>> <https://maps.google.com/?q=4,+avenue+Laurent+Cely&entry=gmail&source=g>
>>> Tour d’Asnière – 92600 Asnière sur Seine
>>> T/ +33 (0)6 28 98 53 40
>>>
>>>
>>> ________________________________
>>> De : Ivan Kudryavtsev <ku...@bw-sw.com>
>>> Envoyé : mardi 12 juin 2018 13:59:39
>>> À : users
>>> Objet : Re: SSVM's not starting, timeout for libvirt python script
>>> in agent.log
>>>
>>> Increasing command timeouts in global parameters can work here. At
>>> least I
>>> met similar behaviour with VR.
>>>
>>> вт, 12 июн. 2018 г., 14:39 Christoffer Pedersen <vr...@vrod.dk>:
>>>
>>> > Hi Nicolas,
>>> >
>>> > I did a apt show qemu and it gave me this version:
>>> >
>>> > Version: 1:2.5+dfsg-5ubuntu10.29
>>> >
>>> > So I guess tha would be version 2.5?
>>> >
>>>
>>> Dag.Sonstebo@shapeblue.com
>>> www.shapeblue.com
>>> 53 Chandos Place, Covent Garden, London
>>> <https://maps.google.com/?q=53+Chandos+Place,+Covent+Garden,+London&entry=gmail&source=g>
>>> WC2N 4HSUK
>>> @shapeblue
>>>
>>>
>>>
>>> > On Tue, Jun 12, 2018 at 1:04 PM, Nicolas Bouige <n....@dimsi.fr>
>>> wrote:
>>> >
>>> > > Hello Christoffer,
>>> > >
>>> > >
>>> > > Could you tell us wich qemu version are you using ?
>>> > >
>>> > > Nicolas Bouige
>>> > > DIMSI
>>> > > cloud.dimsi.fr<http://www.cloud.dimsi.fr>
>>> > > 4, avenue Laurent Cely
>>> <https://maps.google.com/?q=4,+avenue+Laurent+Cely&entry=gmail&source=g>
>>> > > Tour d’Asnière – 92600 Asnière sur Seine
>>> > > T/ +33 (0)6 28 98 53 40
>>> > >
>>> > >
>>> > > ________________________________
>>> > > De : Christoffer Pedersen <vr...@vrod.dk>
>>> > > Envoyé : mardi 12 juin 2018 12:30:48
>>> > > À : users@cloudstack.apache.org
>>> > > Objet : SSVM's not starting, timeout for libvirt python script in
>>> > agent.log
>>> > >
>>> > > Hi all,
>>> > >
>>> > > I have an issue regarding the system VMs. After deploying an
>>> advanced
>>> > zone,
>>> > > the system VMs are trying to be created but gets stuck in a
>>> "Starting"
>>> > > state, however the Agent state is "Up". I have these logs in the
>>> > agent.log
>>> > > (sorry for the formatting)
>>> > >
>>> > > 2018-06-12 12:22:06,354 WARN [kvm.resource.LibvirtComputing
>>> Resource]
>>> > > (Script-8:null) (logid:) Interrupting script.
>>> > > 2018-06-12 12:22:06,355 WARN [kvm.resource.LibvirtComputing
>>> Resource]
>>> > > (agentRequest-Handler-4:null) (logid:ea9cb55a) Timed out:
>>> > > /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patch
>>> viasocket.py
>>> > > -n
>>> > > v-1-VM -p
>>> > > %template=domP%type=consoleproxy%host=1.1.1.1%
>>> > > port=8250%name=v-1-VM%zone=1%pod=1%guid=Proxy.1%proxy_vm=1%
>>> > > disable_rp_filter=true%eth2ip=9.9.9.9%eth2mask=255.255.255.
>>> > > 0%gateway=9.9.9.1%eth0ip=169.254.3.159%eth0mask=255.255.0.
>>> > > 0%eth1ip=6.6.6.6%eth1mask=255.255.255.0%mgmtcidr=
>>> > >
>>> > 9.9.9.0/24%localgw=1.2.3.4%internaldns1=1.2.3.4%dns1=8.8.8.8
>>> %dns2=8.8.4.4
>>> > > . Output is:
>>> > > 2018-06-12 12:22:06,355 ERROR [kvm.resource.LibvirtComputing
>>> Resource]
>>> > > (agentRequest-Handler-4:null) (logid:ea9cb55a) passcmd
>>> failed:timeout
>>> > > 2018-06-12 12:22:08,914 WARN [kvm.resource.LibvirtComputing
>>> Resource]
>>> > > (Script-4:null) (logid:) Interrupting script.
>>> > > 2018-06-12 12:22:08,915 WARN [kvm.resource.LibvirtComputing
>>> Resource]
>>> > > (agentRequest-Handler-5:null) (logid:8e44093e) Timed out:
>>> > > /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patch
>>> viasocket.py
>>> > > -n
>>> > > s-2-VM -p
>>> > > %template=domP%type=secstorage%host=1.1.1.1%port=
>>> > > 8250%name=s-2-VM%zone=1%pod=1%guid=s-2-VM%workers=5%resource
>>> =org.apache.
>>> > > cloudstack.storage.resource.NfsSecondaryStorageResource%
>>> > > instance=SecStorage%sslcopy=false%role=templateProcessor%
>>> > > mtu=1500%eth2ip=7.7.7.7%eth2mask=255.255.255.0%gateway=9.9.9
>>> .1%public.
>>> > > network.device=eth2%eth0ip=169.254.2.193%eth0mask=255.
>>> > > 255.0.0%eth1ip=10.120.0.61%eth1mask=255.255.255.0%mgmtcidr=
>>> > > 9.9.9.0/24%localgw=1.2.3.4%private.network.device=eth1%
>>> > > internaldns1=1.2.3.4%dns1=8.8.8.8%dns2=8.8.4.4%nfsVersion=null
>>> > > . Output is:
>>> > > 2018-06-12 12:22:08,915 ERROR [kvm.resource.LibvirtComputing
>>> Resource]
>>> > > (agentRequest-Handler-5:null) (logid:8e44093e) passcmd
>>> failed:timeout
>>> > >
>>> > > I have seen this error around but did not really find a solution
>>> to it. I
>>> > > am not exactly sure whats "timing" out? I can ping both SSVM's
>>> on their
>>> > > private and public interface.
>>> > >
>>> > > I hope someone can help me out here. :)
>>> > >
>>> > > --
>>> > > Thanks,
>>> > > Chris pedersen
>>> > >
>>> >
>>> >
>>> >
>>> > --
>>> > Thanks,
>>> > Chris pedersen
>>> >
>>>
>>>
>>>
>>
>>
>> --
>> Thanks,
>> Chris pedersen
>>
>
>
>
> --
> Thanks,
> Chris pedersen
>
Re: SSVM's not starting, timeout for libvirt python script in agent.log
Posted by Christoffer Pedersen <vr...@vrod.dk>.
Just want to update that I still get the ISO error:
2018-06-12 17:09:42,194 INFO [c.c.t.HypervisorTemplateAdapter]
(qtp1096283470-12:ctx-962ec49c ctx-2c82c0e2) (logid:0c7df1d7) Image store
doesn't have enough capacity. Skip downloading template to this image store
1
Thanks!
On Tue, Jun 12, 2018 at 5:07 PM, Christoffer Pedersen <vr...@vrod.dk> wrote:
> Hi Dag,
>
> Yes, I altered the IP addresses as I do not fancy throwing them out on the
> public net. If you think they have value to the troubleshooting, I can send
> you the original logdata directly. I configured the networking with OVS and
> as follows:
>
> cloudbr0 - MGMT0 (management interface, VLAN1000), this also hosts the
> system VMs I guess. In the Zone setup, I labelled the Management network as
> MGMT0, I guess that's OK? It's a Internal network where other servers are
> also connected. I can ping the VMs here, also over a S2S connection
> cloudbr1 - Public (VLAN 4000) and Guest network (VLAN 500-999). I use real
> public IP-addresses from a scope and I am able to ping the addresses of
> both the CPVM and SSVM from my home.
>
> I have to update the situation though...
>
> Somehow at 13:02, I had the last error from the agent. Now, I do not have
> any other errors and both VMs are now showing as "running" as well as agent
> being "up" in the UI... had these errors since yesterday and did not
> change anything after 9:00 this morning. Maybe I'm impatient but 4 hours
> seems a bit long for them to get to work. :)
>
> Console VM works though at least:
>
>
>
> My impression is that cloudstack needs a while to get hold of things... or
> am I just experiencing unusual things? I have a question though: When
> adding an ISO this morning, I had an error about that there was no space
> left (though the storage is 20TB). Was this because the SSVM was not
> running at the time?
>
> Thank you!
> Chris
>
> On Tue, Jun 12, 2018 at 4:39 PM, Dag Sonstebo <Da...@shapeblue.com>
> wrote:
>
>> Chris,
>>
>> Going off in a slightly different direction to previous answers. I
>> suspect your problem is with networking - how have you configured this?
>> When you say you can ping the SSVM on the private interface which IP
>> address do you use and where do you successfully ping from?
>>
>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.py
>> -n
>> v-1-VM -p
>> %template=domP%type=consoleproxy%host=1.1.1.1%port=8250%
>> name=v-1-VM%zone=1%pod=1%guid=Proxy.1%proxy_vm=1%disable_rp_
>> filter=true%eth2ip=9.9.9.9%eth2mask=255.255.255.0%
>> gateway=9.9.9.1%eth0ip=169.254.3.159%eth0mask=255.255.0.0%
>> eth1ip=6.6.6.6%eth1mask=255.255.255.0%mgmtcidr=
>> 9.9.9.0/24%localgw=1.2.3.4%internaldns1=1.2.3.4%dns1=8.8.8.8%dns2=8.8.4.4
>>
>> It could be you have edited the above IP addresses to mask your real
>> addresses – if so ignore this.
>>
>> If not then the above points to:
>> - Management host is on 1.1.1.1
>> - Eth2 which for a console proxy is public traffic is on 9.9.9.9/24
>> - Eth0 which is the link local management interface is on
>> 169.254.3.159/16 (system generated)
>> - Eth1 is the main management interface on 6.6.6.6/24
>> - You have a gateway address of 1.2.3.4
>>
>> So in this case – the CPVM can not check in to the management host on
>> 1.1.1.1 - It’s got no interface on that subnet and it also has a gateway
>> it’s not able to reach.
>>
>> Regards,
>> Dag Sonstebo
>> Cloud Architect
>> ShapeBlue
>>
>> On 12/06/2018, 13:12, "Nicolas Bouige" <n....@dimsi.fr> wrote:
>>
>> Hi Ivan,
>>
>>
>> Are you talking about this global parameters :
>>
>> router.aggregation.command.each.timeout
>>
>>
>>
>> Best regards,
>>
>> Nicolas Bouige
>> DIMSI
>> cloud.dimsi.fr<http://www.cloud.dimsi.fr>
>> 4, avenue Laurent Cely
>> Tour d’Asnière – 92600 Asnière sur Seine
>> T/ +33 (0)6 28 98 53 40
>>
>>
>> ________________________________
>> De : Ivan Kudryavtsev <ku...@bw-sw.com>
>> Envoyé : mardi 12 juin 2018 13:59:39
>> À : users
>> Objet : Re: SSVM's not starting, timeout for libvirt python script in
>> agent.log
>>
>> Increasing command timeouts in global parameters can work here. At
>> least I
>> met similar behaviour with VR.
>>
>> вт, 12 июн. 2018 г., 14:39 Christoffer Pedersen <vr...@vrod.dk>:
>>
>> > Hi Nicolas,
>> >
>> > I did a apt show qemu and it gave me this version:
>> >
>> > Version: 1:2.5+dfsg-5ubuntu10.29
>> >
>> > So I guess tha would be version 2.5?
>> >
>>
>> Dag.Sonstebo@shapeblue.com
>> www.shapeblue.com
>> 53 Chandos Place, Covent Garden, London WC2N 4HSUK
>> @shapeblue
>>
>>
>>
>> > On Tue, Jun 12, 2018 at 1:04 PM, Nicolas Bouige <n....@dimsi.fr>
>> wrote:
>> >
>> > > Hello Christoffer,
>> > >
>> > >
>> > > Could you tell us wich qemu version are you using ?
>> > >
>> > > Nicolas Bouige
>> > > DIMSI
>> > > cloud.dimsi.fr<http://www.cloud.dimsi.fr>
>> > > 4, avenue Laurent Cely
>> > > Tour d’Asnière – 92600 Asnière sur Seine
>> > > T/ +33 (0)6 28 98 53 40
>> > >
>> > >
>> > > ________________________________
>> > > De : Christoffer Pedersen <vr...@vrod.dk>
>> > > Envoyé : mardi 12 juin 2018 12:30:48
>> > > À : users@cloudstack.apache.org
>> > > Objet : SSVM's not starting, timeout for libvirt python script in
>> > agent.log
>> > >
>> > > Hi all,
>> > >
>> > > I have an issue regarding the system VMs. After deploying an
>> advanced
>> > zone,
>> > > the system VMs are trying to be created but gets stuck in a
>> "Starting"
>> > > state, however the Agent state is "Up". I have these logs in the
>> > agent.log
>> > > (sorry for the formatting)
>> > >
>> > > 2018-06-12 12:22:06,354 WARN [kvm.resource.LibvirtComputing
>> Resource]
>> > > (Script-8:null) (logid:) Interrupting script.
>> > > 2018-06-12 12:22:06,355 WARN [kvm.resource.LibvirtComputing
>> Resource]
>> > > (agentRequest-Handler-4:null) (logid:ea9cb55a) Timed out:
>> > > /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patch
>> viasocket.py
>> > > -n
>> > > v-1-VM -p
>> > > %template=domP%type=consoleproxy%host=1.1.1.1%
>> > > port=8250%name=v-1-VM%zone=1%pod=1%guid=Proxy.1%proxy_vm=1%
>> > > disable_rp_filter=true%eth2ip=9.9.9.9%eth2mask=255.255.255.
>> > > 0%gateway=9.9.9.1%eth0ip=169.254.3.159%eth0mask=255.255.0.
>> > > 0%eth1ip=6.6.6.6%eth1mask=255.255.255.0%mgmtcidr=
>> > >
>> > 9.9.9.0/24%localgw=1.2.3.4%internaldns1=1.2.3.4%dns1=8.8.8.
>> 8%dns2=8.8.4.4
>> > > . Output is:
>> > > 2018-06-12 12:22:06,355 ERROR [kvm.resource.LibvirtComputing
>> Resource]
>> > > (agentRequest-Handler-4:null) (logid:ea9cb55a) passcmd
>> failed:timeout
>> > > 2018-06-12 12:22:08,914 WARN [kvm.resource.LibvirtComputing
>> Resource]
>> > > (Script-4:null) (logid:) Interrupting script.
>> > > 2018-06-12 12:22:08,915 WARN [kvm.resource.LibvirtComputing
>> Resource]
>> > > (agentRequest-Handler-5:null) (logid:8e44093e) Timed out:
>> > > /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patch
>> viasocket.py
>> > > -n
>> > > s-2-VM -p
>> > > %template=domP%type=secstorage%host=1.1.1.1%port=
>> > > 8250%name=s-2-VM%zone=1%pod=1%guid=s-2-VM%workers=5%resource
>> =org.apache.
>> > > cloudstack.storage.resource.NfsSecondaryStorageResource%
>> > > instance=SecStorage%sslcopy=false%role=templateProcessor%
>> > > mtu=1500%eth2ip=7.7.7.7%eth2mask=255.255.255.0%gateway=9.9.
>> 9.1%public.
>> > > network.device=eth2%eth0ip=169.254.2.193%eth0mask=255.
>> > > 255.0.0%eth1ip=10.120.0.61%eth1mask=255.255.255.0%mgmtcidr=
>> > > 9.9.9.0/24%localgw=1.2.3.4%private.network.device=eth1%
>> > > internaldns1=1.2.3.4%dns1=8.8.8.8%dns2=8.8.4.4%nfsVersion=null
>> > > . Output is:
>> > > 2018-06-12 12:22:08,915 ERROR [kvm.resource.LibvirtComputing
>> Resource]
>> > > (agentRequest-Handler-5:null) (logid:8e44093e) passcmd
>> failed:timeout
>> > >
>> > > I have seen this error around but did not really find a solution
>> to it. I
>> > > am not exactly sure whats "timing" out? I can ping both SSVM's on
>> their
>> > > private and public interface.
>> > >
>> > > I hope someone can help me out here. :)
>> > >
>> > > --
>> > > Thanks,
>> > > Chris pedersen
>> > >
>> >
>> >
>> >
>> > --
>> > Thanks,
>> > Chris pedersen
>> >
>>
>>
>>
>
>
> --
> Thanks,
> Chris pedersen
>
--
Thanks,
Chris pedersen
Re: SSVM's not starting, timeout for libvirt python script in agent.log
Posted by Christoffer Pedersen <vr...@vrod.dk>.
Hi Dag,
Yes, I altered the IP addresses as I do not fancy throwing them out on the
public net. If you think they have value to the troubleshooting, I can send
you the original logdata directly. I configured the networking with OVS and
as follows:
cloudbr0 - MGMT0 (management interface, VLAN1000), this also hosts the
system VMs I guess. In the Zone setup, I labelled the Management network as
MGMT0, I guess that's OK? It's a Internal network where other servers are
also connected. I can ping the VMs here, also over a S2S connection
cloudbr1 - Public (VLAN 4000) and Guest network (VLAN 500-999). I use real
public IP-addresses from a scope and I am able to ping the addresses of
both the CPVM and SSVM from my home.
I have to update the situation though...
Somehow at 13:02, I had the last error from the agent. Now, I do not have
any other errors and both VMs are now showing as "running" as well as agent
being "up" in the UI... had these errors since yesterday and did not
change anything after 9:00 this morning. Maybe I'm impatient but 4 hours
seems a bit long for them to get to work. :)
Console VM works though at least:
My impression is that cloudstack needs a while to get hold of things... or
am I just experiencing unusual things? I have a question though: When
adding an ISO this morning, I had an error about that there was no space
left (though the storage is 20TB). Was this because the SSVM was not
running at the time?
Thank you!
Chris
On Tue, Jun 12, 2018 at 4:39 PM, Dag Sonstebo <Da...@shapeblue.com>
wrote:
> Chris,
>
> Going off in a slightly different direction to previous answers. I suspect
> your problem is with networking - how have you configured this? When you
> say you can ping the SSVM on the private interface which IP address do you
> use and where do you successfully ping from?
>
> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.py
> -n
> v-1-VM -p
> %template=domP%type=consoleproxy%host=1.1.1.1%
> port=8250%name=v-1-VM%zone=1%pod=1%guid=Proxy.1%proxy_vm=1%
> disable_rp_filter=true%eth2ip=9.9.9.9%eth2mask=255.255.255.
> 0%gateway=9.9.9.1%eth0ip=169.254.3.159%eth0mask=255.255.0.
> 0%eth1ip=6.6.6.6%eth1mask=255.255.255.0%mgmtcidr=
> 9.9.9.0/24%localgw=1.2.3.4%internaldns1=1.2.3.4%dns1=8.8.8.8%dns2=8.8.4.4
>
> It could be you have edited the above IP addresses to mask your real
> addresses – if so ignore this.
>
> If not then the above points to:
> - Management host is on 1.1.1.1
> - Eth2 which for a console proxy is public traffic is on 9.9.9.9/24
> - Eth0 which is the link local management interface is on 169.254.3.159/16
> (system generated)
> - Eth1 is the main management interface on 6.6.6.6/24
> - You have a gateway address of 1.2.3.4
>
> So in this case – the CPVM can not check in to the management host on
> 1.1.1.1 - It’s got no interface on that subnet and it also has a gateway
> it’s not able to reach.
>
> Regards,
> Dag Sonstebo
> Cloud Architect
> ShapeBlue
>
> On 12/06/2018, 13:12, "Nicolas Bouige" <n....@dimsi.fr> wrote:
>
> Hi Ivan,
>
>
> Are you talking about this global parameters :
>
> router.aggregation.command.each.timeout
>
>
>
> Best regards,
>
> Nicolas Bouige
> DIMSI
> cloud.dimsi.fr<http://www.cloud.dimsi.fr>
> 4, avenue Laurent Cely
> Tour d’Asnière – 92600 Asnière sur Seine
> T/ +33 (0)6 28 98 53 40
>
>
> ________________________________
> De : Ivan Kudryavtsev <ku...@bw-sw.com>
> Envoyé : mardi 12 juin 2018 13:59:39
> À : users
> Objet : Re: SSVM's not starting, timeout for libvirt python script in
> agent.log
>
> Increasing command timeouts in global parameters can work here. At
> least I
> met similar behaviour with VR.
>
> вт, 12 июн. 2018 г., 14:39 Christoffer Pedersen <vr...@vrod.dk>:
>
> > Hi Nicolas,
> >
> > I did a apt show qemu and it gave me this version:
> >
> > Version: 1:2.5+dfsg-5ubuntu10.29
> >
> > So I guess tha would be version 2.5?
> >
>
> Dag.Sonstebo@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London WC2N 4HSUK
> @shapeblue
>
>
>
> > On Tue, Jun 12, 2018 at 1:04 PM, Nicolas Bouige <n....@dimsi.fr>
> wrote:
> >
> > > Hello Christoffer,
> > >
> > >
> > > Could you tell us wich qemu version are you using ?
> > >
> > > Nicolas Bouige
> > > DIMSI
> > > cloud.dimsi.fr<http://www.cloud.dimsi.fr>
> > > 4, avenue Laurent Cely
> > > Tour d’Asnière – 92600 Asnière sur Seine
> > > T/ +33 (0)6 28 98 53 40
> > >
> > >
> > > ________________________________
> > > De : Christoffer Pedersen <vr...@vrod.dk>
> > > Envoyé : mardi 12 juin 2018 12:30:48
> > > À : users@cloudstack.apache.org
> > > Objet : SSVM's not starting, timeout for libvirt python script in
> > agent.log
> > >
> > > Hi all,
> > >
> > > I have an issue regarding the system VMs. After deploying an
> advanced
> > zone,
> > > the system VMs are trying to be created but gets stuck in a
> "Starting"
> > > state, however the Agent state is "Up". I have these logs in the
> > agent.log
> > > (sorry for the formatting)
> > >
> > > 2018-06-12 12:22:06,354 WARN [kvm.resource.
> LibvirtComputingResource]
> > > (Script-8:null) (logid:) Interrupting script.
> > > 2018-06-12 12:22:06,355 WARN [kvm.resource.
> LibvirtComputingResource]
> > > (agentRequest-Handler-4:null) (logid:ea9cb55a) Timed out:
> > > /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/
> patchviasocket.py
> > > -n
> > > v-1-VM -p
> > > %template=domP%type=consoleproxy%host=1.1.1.1%
> > > port=8250%name=v-1-VM%zone=1%pod=1%guid=Proxy.1%proxy_vm=1%
> > > disable_rp_filter=true%eth2ip=9.9.9.9%eth2mask=255.255.255.
> > > 0%gateway=9.9.9.1%eth0ip=169.254.3.159%eth0mask=255.255.0.
> > > 0%eth1ip=6.6.6.6%eth1mask=255.255.255.0%mgmtcidr=
> > >
> > 9.9.9.0/24%localgw=1.2.3.4%internaldns1=1.2.3.4%dns1=8.8.
> 8.8%dns2=8.8.4.4
> > > . Output is:
> > > 2018-06-12 12:22:06,355 ERROR [kvm.resource.
> LibvirtComputingResource]
> > > (agentRequest-Handler-4:null) (logid:ea9cb55a) passcmd
> failed:timeout
> > > 2018-06-12 12:22:08,914 WARN [kvm.resource.
> LibvirtComputingResource]
> > > (Script-4:null) (logid:) Interrupting script.
> > > 2018-06-12 12:22:08,915 WARN [kvm.resource.
> LibvirtComputingResource]
> > > (agentRequest-Handler-5:null) (logid:8e44093e) Timed out:
> > > /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/
> patchviasocket.py
> > > -n
> > > s-2-VM -p
> > > %template=domP%type=secstorage%host=1.1.1.1%port=
> > > 8250%name=s-2-VM%zone=1%pod=1%guid=s-2-VM%workers=5%
> resource=org.apache.
> > > cloudstack.storage.resource.NfsSecondaryStorageResource%
> > > instance=SecStorage%sslcopy=false%role=templateProcessor%
> > > mtu=1500%eth2ip=7.7.7.7%eth2mask=255.255.255.0%
> gateway=9.9.9.1%public.
> > > network.device=eth2%eth0ip=169.254.2.193%eth0mask=255.
> > > 255.0.0%eth1ip=10.120.0.61%eth1mask=255.255.255.0%mgmtcidr=
> > > 9.9.9.0/24%localgw=1.2.3.4%private.network.device=eth1%
> > > internaldns1=1.2.3.4%dns1=8.8.8.8%dns2=8.8.4.4%nfsVersion=null
> > > . Output is:
> > > 2018-06-12 12:22:08,915 ERROR [kvm.resource.
> LibvirtComputingResource]
> > > (agentRequest-Handler-5:null) (logid:8e44093e) passcmd
> failed:timeout
> > >
> > > I have seen this error around but did not really find a solution
> to it. I
> > > am not exactly sure whats "timing" out? I can ping both SSVM's on
> their
> > > private and public interface.
> > >
> > > I hope someone can help me out here. :)
> > >
> > > --
> > > Thanks,
> > > Chris pedersen
> > >
> >
> >
> >
> > --
> > Thanks,
> > Chris pedersen
> >
>
>
>
--
Thanks,
Chris pedersen
Re: SSVM's not starting, timeout for libvirt python
script in agent.log
Posted by Dag Sonstebo <Da...@shapeblue.com>.
Chris,
Going off in a slightly different direction to previous answers. I suspect your problem is with networking - how have you configured this? When you say you can ping the SSVM on the private interface which IP address do you use and where do you successfully ping from?
/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.py -n
v-1-VM -p
%template=domP%type=consoleproxy%host=1.1.1.1%port=8250%name=v-1-VM%zone=1%pod=1%guid=Proxy.1%proxy_vm=1%disable_rp_filter=true%eth2ip=9.9.9.9%eth2mask=255.255.255.0%gateway=9.9.9.1%eth0ip=169.254.3.159%eth0mask=255.255.0.0%eth1ip=6.6.6.6%eth1mask=255.255.255.0%mgmtcidr=
9.9.9.0/24%localgw=1.2.3.4%internaldns1=1.2.3.4%dns1=8.8.8.8%dns2=8.8.4.4
It could be you have edited the above IP addresses to mask your real addresses – if so ignore this.
If not then the above points to:
- Management host is on 1.1.1.1
- Eth2 which for a console proxy is public traffic is on 9.9.9.9/24
- Eth0 which is the link local management interface is on 169.254.3.159/16 (system generated)
- Eth1 is the main management interface on 6.6.6.6/24
- You have a gateway address of 1.2.3.4
So in this case – the CPVM can not check in to the management host on 1.1.1.1 - It’s got no interface on that subnet and it also has a gateway it’s not able to reach.
Regards,
Dag Sonstebo
Cloud Architect
ShapeBlue
On 12/06/2018, 13:12, "Nicolas Bouige" <n....@dimsi.fr> wrote:
Hi Ivan,
Are you talking about this global parameters :
router.aggregation.command.each.timeout
Best regards,
Nicolas Bouige
DIMSI
cloud.dimsi.fr<http://www.cloud.dimsi.fr>
4, avenue Laurent Cely
Tour d’Asnière – 92600 Asnière sur Seine
T/ +33 (0)6 28 98 53 40
________________________________
De : Ivan Kudryavtsev <ku...@bw-sw.com>
Envoyé : mardi 12 juin 2018 13:59:39
À : users
Objet : Re: SSVM's not starting, timeout for libvirt python script in agent.log
Increasing command timeouts in global parameters can work here. At least I
met similar behaviour with VR.
вт, 12 июн. 2018 г., 14:39 Christoffer Pedersen <vr...@vrod.dk>:
> Hi Nicolas,
>
> I did a apt show qemu and it gave me this version:
>
> Version: 1:2.5+dfsg-5ubuntu10.29
>
> So I guess tha would be version 2.5?
>
Dag.Sonstebo@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
> On Tue, Jun 12, 2018 at 1:04 PM, Nicolas Bouige <n....@dimsi.fr> wrote:
>
> > Hello Christoffer,
> >
> >
> > Could you tell us wich qemu version are you using ?
> >
> > Nicolas Bouige
> > DIMSI
> > cloud.dimsi.fr<http://www.cloud.dimsi.fr>
> > 4, avenue Laurent Cely
> > Tour d’Asnière – 92600 Asnière sur Seine
> > T/ +33 (0)6 28 98 53 40
> >
> >
> > ________________________________
> > De : Christoffer Pedersen <vr...@vrod.dk>
> > Envoyé : mardi 12 juin 2018 12:30:48
> > À : users@cloudstack.apache.org
> > Objet : SSVM's not starting, timeout for libvirt python script in
> agent.log
> >
> > Hi all,
> >
> > I have an issue regarding the system VMs. After deploying an advanced
> zone,
> > the system VMs are trying to be created but gets stuck in a "Starting"
> > state, however the Agent state is "Up". I have these logs in the
> agent.log
> > (sorry for the formatting)
> >
> > 2018-06-12 12:22:06,354 WARN [kvm.resource.LibvirtComputingResource]
> > (Script-8:null) (logid:) Interrupting script.
> > 2018-06-12 12:22:06,355 WARN [kvm.resource.LibvirtComputingResource]
> > (agentRequest-Handler-4:null) (logid:ea9cb55a) Timed out:
> > /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.py
> > -n
> > v-1-VM -p
> > %template=domP%type=consoleproxy%host=1.1.1.1%
> > port=8250%name=v-1-VM%zone=1%pod=1%guid=Proxy.1%proxy_vm=1%
> > disable_rp_filter=true%eth2ip=9.9.9.9%eth2mask=255.255.255.
> > 0%gateway=9.9.9.1%eth0ip=169.254.3.159%eth0mask=255.255.0.
> > 0%eth1ip=6.6.6.6%eth1mask=255.255.255.0%mgmtcidr=
> >
> 9.9.9.0/24%localgw=1.2.3.4%internaldns1=1.2.3.4%dns1=8.8.8.8%dns2=8.8.4.4
> > . Output is:
> > 2018-06-12 12:22:06,355 ERROR [kvm.resource.LibvirtComputingResource]
> > (agentRequest-Handler-4:null) (logid:ea9cb55a) passcmd failed:timeout
> > 2018-06-12 12:22:08,914 WARN [kvm.resource.LibvirtComputingResource]
> > (Script-4:null) (logid:) Interrupting script.
> > 2018-06-12 12:22:08,915 WARN [kvm.resource.LibvirtComputingResource]
> > (agentRequest-Handler-5:null) (logid:8e44093e) Timed out:
> > /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.py
> > -n
> > s-2-VM -p
> > %template=domP%type=secstorage%host=1.1.1.1%port=
> > 8250%name=s-2-VM%zone=1%pod=1%guid=s-2-VM%workers=5%resource=org.apache.
> > cloudstack.storage.resource.NfsSecondaryStorageResource%
> > instance=SecStorage%sslcopy=false%role=templateProcessor%
> > mtu=1500%eth2ip=7.7.7.7%eth2mask=255.255.255.0%gateway=9.9.9.1%public.
> > network.device=eth2%eth0ip=169.254.2.193%eth0mask=255.
> > 255.0.0%eth1ip=10.120.0.61%eth1mask=255.255.255.0%mgmtcidr=
> > 9.9.9.0/24%localgw=1.2.3.4%private.network.device=eth1%
> > internaldns1=1.2.3.4%dns1=8.8.8.8%dns2=8.8.4.4%nfsVersion=null
> > . Output is:
> > 2018-06-12 12:22:08,915 ERROR [kvm.resource.LibvirtComputingResource]
> > (agentRequest-Handler-5:null) (logid:8e44093e) passcmd failed:timeout
> >
> > I have seen this error around but did not really find a solution to it. I
> > am not exactly sure whats "timing" out? I can ping both SSVM's on their
> > private and public interface.
> >
> > I hope someone can help me out here. :)
> >
> > --
> > Thanks,
> > Chris pedersen
> >
>
>
>
> --
> Thanks,
> Chris pedersen
>
RE: SSVM's not starting, timeout for libvirt python script in
agent.log
Posted by Nicolas Bouige <n....@dimsi.fr>.
Hi Ivan,
Are you talking about this global parameters :
router.aggregation.command.each.timeout
Best regards,
Nicolas Bouige
DIMSI
cloud.dimsi.fr<http://www.cloud.dimsi.fr>
4, avenue Laurent Cely
Tour d’Asnière – 92600 Asnière sur Seine
T/ +33 (0)6 28 98 53 40
________________________________
De : Ivan Kudryavtsev <ku...@bw-sw.com>
Envoyé : mardi 12 juin 2018 13:59:39
À : users
Objet : Re: SSVM's not starting, timeout for libvirt python script in agent.log
Increasing command timeouts in global parameters can work here. At least I
met similar behaviour with VR.
вт, 12 июн. 2018 г., 14:39 Christoffer Pedersen <vr...@vrod.dk>:
> Hi Nicolas,
>
> I did a apt show qemu and it gave me this version:
>
> Version: 1:2.5+dfsg-5ubuntu10.29
>
> So I guess tha would be version 2.5?
>
> On Tue, Jun 12, 2018 at 1:04 PM, Nicolas Bouige <n....@dimsi.fr> wrote:
>
> > Hello Christoffer,
> >
> >
> > Could you tell us wich qemu version are you using ?
> >
> > Nicolas Bouige
> > DIMSI
> > cloud.dimsi.fr<http://www.cloud.dimsi.fr>
> > 4, avenue Laurent Cely
> > Tour d’Asnière – 92600 Asnière sur Seine
> > T/ +33 (0)6 28 98 53 40
> >
> >
> > ________________________________
> > De : Christoffer Pedersen <vr...@vrod.dk>
> > Envoyé : mardi 12 juin 2018 12:30:48
> > À : users@cloudstack.apache.org
> > Objet : SSVM's not starting, timeout for libvirt python script in
> agent.log
> >
> > Hi all,
> >
> > I have an issue regarding the system VMs. After deploying an advanced
> zone,
> > the system VMs are trying to be created but gets stuck in a "Starting"
> > state, however the Agent state is "Up". I have these logs in the
> agent.log
> > (sorry for the formatting)
> >
> > 2018-06-12 12:22:06,354 WARN [kvm.resource.LibvirtComputingResource]
> > (Script-8:null) (logid:) Interrupting script.
> > 2018-06-12 12:22:06,355 WARN [kvm.resource.LibvirtComputingResource]
> > (agentRequest-Handler-4:null) (logid:ea9cb55a) Timed out:
> > /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.py
> > -n
> > v-1-VM -p
> > %template=domP%type=consoleproxy%host=1.1.1.1%
> > port=8250%name=v-1-VM%zone=1%pod=1%guid=Proxy.1%proxy_vm=1%
> > disable_rp_filter=true%eth2ip=9.9.9.9%eth2mask=255.255.255.
> > 0%gateway=9.9.9.1%eth0ip=169.254.3.159%eth0mask=255.255.0.
> > 0%eth1ip=6.6.6.6%eth1mask=255.255.255.0%mgmtcidr=
> >
> 9.9.9.0/24%localgw=1.2.3.4%internaldns1=1.2.3.4%dns1=8.8.8.8%dns2=8.8.4.4
> > . Output is:
> > 2018-06-12 12:22:06,355 ERROR [kvm.resource.LibvirtComputingResource]
> > (agentRequest-Handler-4:null) (logid:ea9cb55a) passcmd failed:timeout
> > 2018-06-12 12:22:08,914 WARN [kvm.resource.LibvirtComputingResource]
> > (Script-4:null) (logid:) Interrupting script.
> > 2018-06-12 12:22:08,915 WARN [kvm.resource.LibvirtComputingResource]
> > (agentRequest-Handler-5:null) (logid:8e44093e) Timed out:
> > /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.py
> > -n
> > s-2-VM -p
> > %template=domP%type=secstorage%host=1.1.1.1%port=
> > 8250%name=s-2-VM%zone=1%pod=1%guid=s-2-VM%workers=5%resource=org.apache.
> > cloudstack.storage.resource.NfsSecondaryStorageResource%
> > instance=SecStorage%sslcopy=false%role=templateProcessor%
> > mtu=1500%eth2ip=7.7.7.7%eth2mask=255.255.255.0%gateway=9.9.9.1%public.
> > network.device=eth2%eth0ip=169.254.2.193%eth0mask=255.
> > 255.0.0%eth1ip=10.120.0.61%eth1mask=255.255.255.0%mgmtcidr=
> > 9.9.9.0/24%localgw=1.2.3.4%private.network.device=eth1%
> > internaldns1=1.2.3.4%dns1=8.8.8.8%dns2=8.8.4.4%nfsVersion=null
> > . Output is:
> > 2018-06-12 12:22:08,915 ERROR [kvm.resource.LibvirtComputingResource]
> > (agentRequest-Handler-5:null) (logid:8e44093e) passcmd failed:timeout
> >
> > I have seen this error around but did not really find a solution to it. I
> > am not exactly sure whats "timing" out? I can ping both SSVM's on their
> > private and public interface.
> >
> > I hope someone can help me out here. :)
> >
> > --
> > Thanks,
> > Chris pedersen
> >
>
>
>
> --
> Thanks,
> Chris pedersen
>
Re: SSVM's not starting, timeout for libvirt python script in agent.log
Posted by Ivan Kudryavtsev <ku...@bw-sw.com>.
Increasing command timeouts in global parameters can work here. At least I
met similar behaviour with VR.
вт, 12 июн. 2018 г., 14:39 Christoffer Pedersen <vr...@vrod.dk>:
> Hi Nicolas,
>
> I did a apt show qemu and it gave me this version:
>
> Version: 1:2.5+dfsg-5ubuntu10.29
>
> So I guess tha would be version 2.5?
>
> On Tue, Jun 12, 2018 at 1:04 PM, Nicolas Bouige <n....@dimsi.fr> wrote:
>
> > Hello Christoffer,
> >
> >
> > Could you tell us wich qemu version are you using ?
> >
> > Nicolas Bouige
> > DIMSI
> > cloud.dimsi.fr<http://www.cloud.dimsi.fr>
> > 4, avenue Laurent Cely
> > Tour d’Asnière – 92600 Asnière sur Seine
> > T/ +33 (0)6 28 98 53 40
> >
> >
> > ________________________________
> > De : Christoffer Pedersen <vr...@vrod.dk>
> > Envoyé : mardi 12 juin 2018 12:30:48
> > À : users@cloudstack.apache.org
> > Objet : SSVM's not starting, timeout for libvirt python script in
> agent.log
> >
> > Hi all,
> >
> > I have an issue regarding the system VMs. After deploying an advanced
> zone,
> > the system VMs are trying to be created but gets stuck in a "Starting"
> > state, however the Agent state is "Up". I have these logs in the
> agent.log
> > (sorry for the formatting)
> >
> > 2018-06-12 12:22:06,354 WARN [kvm.resource.LibvirtComputingResource]
> > (Script-8:null) (logid:) Interrupting script.
> > 2018-06-12 12:22:06,355 WARN [kvm.resource.LibvirtComputingResource]
> > (agentRequest-Handler-4:null) (logid:ea9cb55a) Timed out:
> > /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.py
> > -n
> > v-1-VM -p
> > %template=domP%type=consoleproxy%host=1.1.1.1%
> > port=8250%name=v-1-VM%zone=1%pod=1%guid=Proxy.1%proxy_vm=1%
> > disable_rp_filter=true%eth2ip=9.9.9.9%eth2mask=255.255.255.
> > 0%gateway=9.9.9.1%eth0ip=169.254.3.159%eth0mask=255.255.0.
> > 0%eth1ip=6.6.6.6%eth1mask=255.255.255.0%mgmtcidr=
> >
> 9.9.9.0/24%localgw=1.2.3.4%internaldns1=1.2.3.4%dns1=8.8.8.8%dns2=8.8.4.4
> > . Output is:
> > 2018-06-12 12:22:06,355 ERROR [kvm.resource.LibvirtComputingResource]
> > (agentRequest-Handler-4:null) (logid:ea9cb55a) passcmd failed:timeout
> > 2018-06-12 12:22:08,914 WARN [kvm.resource.LibvirtComputingResource]
> > (Script-4:null) (logid:) Interrupting script.
> > 2018-06-12 12:22:08,915 WARN [kvm.resource.LibvirtComputingResource]
> > (agentRequest-Handler-5:null) (logid:8e44093e) Timed out:
> > /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.py
> > -n
> > s-2-VM -p
> > %template=domP%type=secstorage%host=1.1.1.1%port=
> > 8250%name=s-2-VM%zone=1%pod=1%guid=s-2-VM%workers=5%resource=org.apache.
> > cloudstack.storage.resource.NfsSecondaryStorageResource%
> > instance=SecStorage%sslcopy=false%role=templateProcessor%
> > mtu=1500%eth2ip=7.7.7.7%eth2mask=255.255.255.0%gateway=9.9.9.1%public.
> > network.device=eth2%eth0ip=169.254.2.193%eth0mask=255.
> > 255.0.0%eth1ip=10.120.0.61%eth1mask=255.255.255.0%mgmtcidr=
> > 9.9.9.0/24%localgw=1.2.3.4%private.network.device=eth1%
> > internaldns1=1.2.3.4%dns1=8.8.8.8%dns2=8.8.4.4%nfsVersion=null
> > . Output is:
> > 2018-06-12 12:22:08,915 ERROR [kvm.resource.LibvirtComputingResource]
> > (agentRequest-Handler-5:null) (logid:8e44093e) passcmd failed:timeout
> >
> > I have seen this error around but did not really find a solution to it. I
> > am not exactly sure whats "timing" out? I can ping both SSVM's on their
> > private and public interface.
> >
> > I hope someone can help me out here. :)
> >
> > --
> > Thanks,
> > Chris pedersen
> >
>
>
>
> --
> Thanks,
> Chris pedersen
>
Re: SSVM's not starting, timeout for libvirt python script in agent.log
Posted by Christoffer Pedersen <vr...@vrod.dk>.
Hi Nicolas,
I did a apt show qemu and it gave me this version:
Version: 1:2.5+dfsg-5ubuntu10.29
So I guess tha would be version 2.5?
On Tue, Jun 12, 2018 at 1:04 PM, Nicolas Bouige <n....@dimsi.fr> wrote:
> Hello Christoffer,
>
>
> Could you tell us wich qemu version are you using ?
>
> Nicolas Bouige
> DIMSI
> cloud.dimsi.fr<http://www.cloud.dimsi.fr>
> 4, avenue Laurent Cely
> Tour d’Asnière – 92600 Asnière sur Seine
> T/ +33 (0)6 28 98 53 40
>
>
> ________________________________
> De : Christoffer Pedersen <vr...@vrod.dk>
> Envoyé : mardi 12 juin 2018 12:30:48
> À : users@cloudstack.apache.org
> Objet : SSVM's not starting, timeout for libvirt python script in agent.log
>
> Hi all,
>
> I have an issue regarding the system VMs. After deploying an advanced zone,
> the system VMs are trying to be created but gets stuck in a "Starting"
> state, however the Agent state is "Up". I have these logs in the agent.log
> (sorry for the formatting)
>
> 2018-06-12 12:22:06,354 WARN [kvm.resource.LibvirtComputingResource]
> (Script-8:null) (logid:) Interrupting script.
> 2018-06-12 12:22:06,355 WARN [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-4:null) (logid:ea9cb55a) Timed out:
> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.py
> -n
> v-1-VM -p
> %template=domP%type=consoleproxy%host=1.1.1.1%
> port=8250%name=v-1-VM%zone=1%pod=1%guid=Proxy.1%proxy_vm=1%
> disable_rp_filter=true%eth2ip=9.9.9.9%eth2mask=255.255.255.
> 0%gateway=9.9.9.1%eth0ip=169.254.3.159%eth0mask=255.255.0.
> 0%eth1ip=6.6.6.6%eth1mask=255.255.255.0%mgmtcidr=
> 9.9.9.0/24%localgw=1.2.3.4%internaldns1=1.2.3.4%dns1=8.8.8.8%dns2=8.8.4.4
> . Output is:
> 2018-06-12 12:22:06,355 ERROR [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-4:null) (logid:ea9cb55a) passcmd failed:timeout
> 2018-06-12 12:22:08,914 WARN [kvm.resource.LibvirtComputingResource]
> (Script-4:null) (logid:) Interrupting script.
> 2018-06-12 12:22:08,915 WARN [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-5:null) (logid:8e44093e) Timed out:
> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.py
> -n
> s-2-VM -p
> %template=domP%type=secstorage%host=1.1.1.1%port=
> 8250%name=s-2-VM%zone=1%pod=1%guid=s-2-VM%workers=5%resource=org.apache.
> cloudstack.storage.resource.NfsSecondaryStorageResource%
> instance=SecStorage%sslcopy=false%role=templateProcessor%
> mtu=1500%eth2ip=7.7.7.7%eth2mask=255.255.255.0%gateway=9.9.9.1%public.
> network.device=eth2%eth0ip=169.254.2.193%eth0mask=255.
> 255.0.0%eth1ip=10.120.0.61%eth1mask=255.255.255.0%mgmtcidr=
> 9.9.9.0/24%localgw=1.2.3.4%private.network.device=eth1%
> internaldns1=1.2.3.4%dns1=8.8.8.8%dns2=8.8.4.4%nfsVersion=null
> . Output is:
> 2018-06-12 12:22:08,915 ERROR [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-5:null) (logid:8e44093e) passcmd failed:timeout
>
> I have seen this error around but did not really find a solution to it. I
> am not exactly sure whats "timing" out? I can ping both SSVM's on their
> private and public interface.
>
> I hope someone can help me out here. :)
>
> --
> Thanks,
> Chris pedersen
>
--
Thanks,
Chris pedersen
RE: SSVM's not starting, timeout for libvirt python script in
agent.log
Posted by Nicolas Bouige <n....@dimsi.fr>.
Hello Christoffer,
Could you tell us wich qemu version are you using ?
Nicolas Bouige
DIMSI
cloud.dimsi.fr<http://www.cloud.dimsi.fr>
4, avenue Laurent Cely
Tour d’Asnière – 92600 Asnière sur Seine
T/ +33 (0)6 28 98 53 40
________________________________
De : Christoffer Pedersen <vr...@vrod.dk>
Envoyé : mardi 12 juin 2018 12:30:48
À : users@cloudstack.apache.org
Objet : SSVM's not starting, timeout for libvirt python script in agent.log
Hi all,
I have an issue regarding the system VMs. After deploying an advanced zone,
the system VMs are trying to be created but gets stuck in a "Starting"
state, however the Agent state is "Up". I have these logs in the agent.log
(sorry for the formatting)
2018-06-12 12:22:06,354 WARN [kvm.resource.LibvirtComputingResource]
(Script-8:null) (logid:) Interrupting script.
2018-06-12 12:22:06,355 WARN [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-4:null) (logid:ea9cb55a) Timed out:
/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.py -n
v-1-VM -p
%template=domP%type=consoleproxy%host=1.1.1.1%port=8250%name=v-1-VM%zone=1%pod=1%guid=Proxy.1%proxy_vm=1%disable_rp_filter=true%eth2ip=9.9.9.9%eth2mask=255.255.255.0%gateway=9.9.9.1%eth0ip=169.254.3.159%eth0mask=255.255.0.0%eth1ip=6.6.6.6%eth1mask=255.255.255.0%mgmtcidr=
9.9.9.0/24%localgw=1.2.3.4%internaldns1=1.2.3.4%dns1=8.8.8.8%dns2=8.8.4.4
. Output is:
2018-06-12 12:22:06,355 ERROR [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-4:null) (logid:ea9cb55a) passcmd failed:timeout
2018-06-12 12:22:08,914 WARN [kvm.resource.LibvirtComputingResource]
(Script-4:null) (logid:) Interrupting script.
2018-06-12 12:22:08,915 WARN [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-5:null) (logid:8e44093e) Timed out:
/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.py -n
s-2-VM -p
%template=domP%type=secstorage%host=1.1.1.1%port=8250%name=s-2-VM%zone=1%pod=1%guid=s-2-VM%workers=5%resource=org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource%instance=SecStorage%sslcopy=false%role=templateProcessor%mtu=1500%eth2ip=7.7.7.7%eth2mask=255.255.255.0%gateway=9.9.9.1%public.network.device=eth2%eth0ip=169.254.2.193%eth0mask=255.255.0.0%eth1ip=10.120.0.61%eth1mask=255.255.255.0%mgmtcidr=
9.9.9.0/24%localgw=1.2.3.4%private.network.device=eth1%internaldns1=1.2.3.4%dns1=8.8.8.8%dns2=8.8.4.4%nfsVersion=null
. Output is:
2018-06-12 12:22:08,915 ERROR [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-5:null) (logid:8e44093e) passcmd failed:timeout
I have seen this error around but did not really find a solution to it. I
am not exactly sure whats "timing" out? I can ping both SSVM's on their
private and public interface.
I hope someone can help me out here. :)
--
Thanks,
Chris pedersen