Apache CloudStack 4.10 VR/BasicZone/KVM Problem

[Ivan Kudryavtsev <ku...@bw-sw.com>]

Hello, Devs.

I experience VR Start Problem in the fresh ACS 4.10 deployment

Intersting place of logs is here: https://pastebin.com/iBXRBA5N

Basically, the situation looks like:

1. Management Server tries to launch VR
2. It gets from Agent proper VR response with VR details
3. It sends StopCommand without explanation.

I'm trying to figure out what happens inside, but the codebase is huge and
still no positive results. Please, let me know if you have any ideas which
could help me finding the reason. Thanks a lot.

-- 
With best regards, Ivan Kudryavtsev
Bitworks Software, Ltd.
Cell: +7-923-414-1515
WWW: http://bitworks.software/ <http://bw-sw.com/>

Re: Apache CloudStack 4.10 VR/BasicZone/KVM Problem

[Özhan Rüzgar Karaman <or...@gmail.com>]

Hi Ivan, i know that 4.9.x is lts release and 4.10 and 4.11 are not lts
releases. What will happen when new lts release comes, does this release
fork from 4.9.x branch and apply fixes on 4.10 and 4.1x over that or new
lts release will forward fork from 4.11 or 4.1x releases?

I am curious about this procedure and I also ask this question because we
are using a fix ( https://issues.apache.org/jira/browse/CLOUDSTACK-9538 )
which is only on 4.9 branch and not available for 4.1x branches, i am not
developer so i need to find a developer from community to port this fix to
new 4.1x trees, if upcoming lts release will fork from 4.1x releases :)


On Mon, Nov 13, 2017 at 12:41 PM, Ivan Kudryavtsev <kudryavtsev_ia@bw-sw.com
> wrote:

> I don't know if 4.10 will be improved centrally. AFAIK 4.11 WIP right
> now...
>
> 13 нояб. 2017 г. 16:37 пользователь "Özhan Rüzgar Karaman" <
> oruzgarkaraman@gmail.com> написал:
>
> > Hi Ivan;
> > I like to try virtio-scsi, i like to use its benefits like enlarging data
> > disks without rebooting vm and getting unused space in vm back to my
> > central storage, we also waited for long time, lets wait some more for
> 4.10
> > to become more stable :) Thanks for all replies
> >
> > Özhan
> >
> > On Mon, Nov 13, 2017 at 12:29 PM, Ivan Kudryavtsev <
> > kudryavtsev_ia@bw-sw.com
> > > wrote:
> >
> > > Ozhan, I suggest 4.10 only if you need KVM VM snapshots, virtio-scsi
> and
> > > ipv6. Don't see other reasons to go from 4.9 (at least for kvm). But I
> > > waited for 4.10 for a long time...
> > >
> > > 13 нояб. 2017 г. 16:21 пользователь "Özhan Rüzgar Karaman" <
> > > oruzgarkaraman@gmail.com> написал:
> > >
> > > > ok we will try to compile 4.10 from source and try it on our test
> > > > environment.
> > > >
> > > > We currently use 4.9.x on production, i make this test for deciding
> if
> > > 4.10
> > > > is suitable & stable for us, we also use very simple and generic
> setup
> > > for
> > > > this test no ipv6 just simple environment . So for production it
> looks
> > > like
> > > > we need to wait some time for upcoming releases, what do you think
> > about
> > > > that?
> > > >
> > > > There was a thread about this question one month ago and i remember
> > that
> > > > most of people still stick to 4.9 release for their production
> > > > environments...
> > > >
> > > > Thanks
> > > > Özhan
> > > >
> > > > On Mon, Nov 13, 2017 at 12:18 PM, Wei ZHOU <us...@gmail.com>
> > > wrote:
> > > >
> > > > > Hi Ivan,
> > > > >
> > > > > I would suggest you to create jira tickets for each problem you
> found
> > > in
> > > > > your testing, and create a github pull request for a jira ticket.
> > > > > It is convenient for reviewers.
> > > > >
> > > > > Kind regards,
> > > > > Wei
> > > > >
> > > > > 2017-11-13 10:01 GMT+01:00 Ivan Kudryavtsev <
> > kudryavtsev_ia@bw-sw.com
> > > >:
> > > > >
> > > > > > Hello, Ozhan
> > > > > >
> > > > > > https://github.com/apache/cloudstack/pull/2320
> > > > > >
> > > > > > fixes everything I found right now. It enables functioning of
> > > > everything
> > > > > > correctly even if no IPv6 CIDR specified for network (at least
> for
> > > > Ubuntu
> > > > > > 14.04).
> > > > > > For IPv6 configuration instruction please take a look at:
> > > > > > https://github.com/apache/cloudstack/commit/
> > > > > f10c8bfe0c99a762c2606459413a47
> > > > > > 219614e775
> > > > > > (oh my god,I spend several hours trying to find how to configure
> > IPv6
> > > > for
> > > > > > 4.10).
> > > > > >
> > > > > > Please, don't forget to recreate SSVM because there is a fix for
> > > > > templates
> > > > > > too:
> > > > > > https://github.com/apache/cloudstack/pull/2322
> > > > > >
> > > > > >
> > > > > > 2017-11-13 15:51 GMT+07:00 Özhan Rüzgar Karaman <
> > > > > oruzgarkaraman@gmail.com
> > > > > > >:
> > > > > >
> > > > > > > Hi Ivan;
> > > > > > > Does this hotfixes also solve qoutes and shell script
> interprets
> > > > > problem?
> > > > > > > We have no ipv6 setup and today we made similar test with fresh
> > > > install
> > > > > > > 4.10. We noticed that we receive similar error on security
> groups
> > > > stage
> > > > > > > while br_netfilter module is already active on our environment.
> > We
> > > > made
> > > > > > > same tests for Ubuntu 16.04.3 and 14.04.5 kvm hosts
> > > > > > >
> > > > > > > Logs are below:
> > > > > > > 2017-11-13 11:47:41,773 DEBUG [kvm.resource.
> > > > LibvirtComputingResource]
> > > > > > > (agentRequest-Handler-1:null) Executing:
> > > > > > > /usr/share/cloudstack-common/scripts/vm/network/security_
> > group.py
> > > > > > > add_network_rules --vmname i-2-5-VM --vmid 5 --vmip
> 192.168.18.6
> > > > > --vmip6
> > > > > > > null --sig 74a6d8c403af9c3c7b89ecf206e4ac26 --seq 16 --vmmac
> > > > > > > 1e:00:9b:00:00:05 --vif vnet8 --brname breth0-23 --nicsecips 0:
> > > > --rules
> > > > > > > I:tcp:1:65535:
> > > > > > > 0.0.0.0/0,NEXT;I:udp:1:65535:0.0.0.0/0,NEXT;E:tcp:1:65535:
> > > > > 0.0.0.0/0,NEXT
> > > > > > ;
> > > > > > > 2017-11-13 11:47:41,773 WARN  [kvm.resource.
> > > > LibvirtComputingResource]
> > > > > > > (agentRequest-Handler-1:null) Exception:
> > > > > > > /usr/share/cloudstack-common/scripts/vm/network/security_
> > group.py
> > > > > > > add_network_rules --vmname i-2-5-VM --vmid 5 --vmip
> 192.168.18.6
> > > > > --vmip6
> > > > > > > null --sig 74a6d8c403af9c3c7b89ecf206e4ac26 --seq 16 --vmmac
> > > > > > > 1e:00:9b:00:00:05 --vif vnet8 --brname breth0-23 --nicsecips 0:
> > > > --rules
> > > > > > > I:tcp:1:65535:
> > > > > > > 0.0.0.0/0,NEXT;I:udp:1:65535:0.0.0.0/0,NEXT;E:tcp:1:65535:
> > > > > 0.0.0.0/0,NEXT
> > > > > > ;
> > > > > > > java.lang.NullPointerException
> > > > > > > at java.lang.ProcessBuilder.start(ProcessBuilder.java:1012)
> > > > > > > at com.cloud.utils.script.Script.execute(Script.java:214)
> > > > > > > at com.cloud.utils.script.Script.execute(Script.java:182)
> > > > > > > at
> > > > > > > com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.
> > > > > > > addNetworkRules(LibvirtComputingResource.java:3429)
> > > > > > > at
> > > > > > > com.cloud.hypervisor.kvm.resource.wrapper.
> > > > > LibvirtSecurityGroupRulesComma
> > > > > > > ndWrapper.execute(LibvirtSecurityGroupRulesComma
> > ndWrapper.java:57)
> > > > > > > at
> > > > > > > com.cloud.hypervisor.kvm.resource.wrapper.
> > > > > LibvirtSecurityGroupRulesComma
> > > > > > > ndWrapper.execute(LibvirtSecurityGroupRulesComma
> > ndWrapper.java:36)
> > > > > > > at
> > > > > > > com.cloud.hypervisor.kvm.resource.wrapper.
> > > > > LibvirtRequestWrapper.execute(
> > > > > > > LibvirtRequestWrapper.java:75)
> > > > > > > at
> > > > > > > com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.
> > > > > > executeRequest(
> > > > > > > LibvirtComputingResource.java:1369)
> > > > > > > at com.cloud.agent.Agent.processRequest(Agent.java:525)
> > > > > > > at com.cloud.agent.Agent$AgentRequestHandler.doTask(
> > > Agent.java:833)
> > > > > > > at com.cloud.utils.nio.Task.call(Task.java:83)
> > > > > > > at com.cloud.utils.nio.Task.call(Task.java:29)
> > > > > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > > > > > at
> > > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(
> > > > > > > ThreadPoolExecutor.java:1149)
> > > > > > > at
> > > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > > > > > > ThreadPoolExecutor.java:624)
> > > > > > > at java.lang.Thread.run(Thread.java:748)
> > > > > > > 2017-11-13 11:47:41,774 WARN
> > > > > > > [resource.wrapper.LibvirtSecurityGroupRulesCommandWrapper]
> > > > > > > (agentRequest-Handler-1:null) Failed to program network rules
> for
> > > vm
> > > > > > > i-2-5-VM
> > > > > > > 2017-11-13 11:47:41,775 DEBUG [cloud.agent.Agent]
> > > > > > > (agentRequest-Handler-1:null) Seq 1-6412562919422165093:  {
> Ans:
> > ,
> > > > > > MgmtId:
> > > > > > > 345048635880, via: 1, Ver: v1, Flags: 110,
> > > > > > > [{"com.cloud.agent.api.SecurityGroupRuleAnswer":{"
> > > > > > > logSequenceNumber":16,"vmId":5,"reason":"PROGRAMMING_
> > > > > > > FAILED","result":false,"details":"programming
> > > > > > > network rules failed","wait":0}}] }
> > > > > > >
> > > > > > >
> > > > > > > When we execute command with double quotas for rules section
> from
> > > > > command
> > > > > > > line it executes without a problem like below:
> > > > > > > root@kvmt3:/var/log/cloudstack/agent#
> > > > > > > /usr/share/cloudstack-common/scripts/vm/network/security_
> > group.py
> > > > > > > add_network_rules --vmname i-2-5-VM --vmid 5 --vmip
> 192.168.18.6
> > > > > --vmip6
> > > > > > > null --sig 74a6d8c403af9c3c7b89ecf206e4ac26 --seq 16 --vmmac
> > > > > > > 1e:00:9b:00:00:05 --vif vnet8 --brname breth0-23 --nicsecips 0:
> > > > --rules
> > > > > > > "I:tcp:1:65535:
> > > > > > > 0.0.0.0/0,NEXT;I:udp:1:65535:0.0.0.0/0,NEXT;E:tcp:1:65535:
> > > > > 0.0.0.0/0,NEXT
> > > > > > ;"
> > > > > > > root@kvmt3:/var/log/cloudstack/agent# echo $?
> > > > > > > 0
> > > > > > > root@kvmt3:/var/log/cloudstack/agent#
> > > > > > >
> > > > > > > Thanks
> > > > > > > Özhan
> > > > > > >
> > > > > > >
> > > > > > > On Sat, Nov 11, 2017 at 6:59 PM, Ivan Kudryavtsev <
> > > > > > > kudryavtsev_ia@bw-sw.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hello, I implemented some hotfixes for 4.10 to work
> > > > > > > >
> > > > > > > > https://github.com/apache/cloudstack/pull/2319 - to master
> > (load
> > > > > > > > br_netfilter module)
> > > > > > > > https://github.com/apache/cloudstack/pull/2320 - to 4.10
> which
> > > > fixes
> > > > > > SG
> > > > > > > > failures related to ipv6.
> > > > > > > >
> > > > > > > >
> > > > > > > > 2017-11-11 15:51 GMT+07:00 Ivan Kudryavtsev <
> > > > > kudryavtsev_ia@bw-sw.com
> > > > > > >:
> > > > > > > >
> > > > > > > > > Following up with previous question. I managed to make it
> > work
> > > by
> > > > > > > > removing
> > > > > > > > > all and heading to ubuntu 14.04 hypervisor host.
> > > > > > > > >
> > > > > > > > > Also, what I found more:
> > > > > > > > >
> > > > > > > > > 1. when setup databases (management server) if custom port
> is
> > > > > > > specified,
> > > > > > > > > databases themself is not created. If create manually,
> import
> > > > > scripts
> > > > > > > > work
> > > > > > > > > fine.
> > > > > > > > > 2. UI: unable to download ISO to __all__ zones. Have to
> > specify
> > > > > > certain
> > > > > > > > > zone, else UI gives an error.
> > > > > > > > > 3. Ubuntu doesn't load module *br_netfilter* but
> > > > > > > > >
> > > > > > > > > /usr/share/cloudstack-common/scripts/vm/network/security_
> > > > group.py
> > > > > > > > >
> > > > > > > > > uses it and nothing good as a result:
> > > > > > > > >
> > > > > > > > > 2017-11-11 15:38:29,241 - sysctl -w
> > net.bridge.bridge-nf-call-
> > > > > > > > arptables=1
> > > > > > > > > 2017-11-11 15:38:29,244 - sysctl -w
> > net.bridge.bridge-nf-call-
> > > > > > > iptables=1
> > > > > > > > > 2017-11-11 15:38:29,247 - sysctl -w
> > net.bridge.bridge-nf-call-
> > > > > > > > ip6tables=1
> > > > > > > > >
> > > > > > > > > adding br_netfilter to /etc/modules fixes it. I suppose
> it's
> > > > > > necessary
> > > > > > > to
> > > > > > > > > add
> > > > > > > > >
> > > > > > > > > in script something like: modprobe br_netfilter (or smarter
> > > > thing).
> > > > > > > > >
> > > > > > > > > But It doesn't work completely, actually, security groups
> are
> > > > > > > > unfunctional:
> > > > > > > > >
> > > > > > > > > ==> /var/log/cloudstack/agent/agent.log <==
> > > > > > > > > 2017-11-11 15:40:41,326 WARN  [kvm.resource.
> > > > > > LibvirtComputingResource]
> > > > > > > > > (agentRequest-Handler-2:null) (logid:eab9a328) Exception:
> > > > > > > > > /usr/share/cloudstack-common/scripts/vm/network/security_
> > > > group.py
> > > > > > > > > add_network_rules --vmname i-2-7-VM --vmid 7 --vmip
> > > 176.120.28.4
> > > > > > > --vmip6
> > > > > > > > > null --sig d60255deb618b7be9f477eed10d65234 --seq 4
> --vmmac
> > > > > > > > > 1e:00:6f:00:01:01 --vif vnet8 --brname cloudbr0 --nicsecips
> > 0:
> > > > > > --rules
> > > > > > > > > I:icmp:-1:-1:0.0.0.0/0,NEXT;I:tcp:1:65535:0.0.0.0/0,NEXT;I
> :
> > > > > > > > > udp:1:65535:0.0.0.0/0,NEXT;E:icmp:-1:-1:0.0.0.0/0,NEXT;E:
> > > > > > > > > tcp:1:65535:0.0.0.0/0,NEXT;E:udp:1:65535:0.0.0.0/0,NEXT;
> > > > > > > > > java.lang.NullPointerException
> > > > > > > > > at java.lang.ProcessBuilder.start(ProcessBuilder.java:
> 1012)
> > > > > > > > > at com.cloud.utils.script.Script.execute(Script.java:214)
> > > > > > > > > at com.cloud.utils.script.Script.execute(Script.java:182)
> > > > > > > > > at com.cloud.hypervisor.kvm.resource.
> > LibvirtComputingResource.
> > > > > > > > > addNetworkRules(LibvirtComputingResource.java:3429)
> > > > > > > > > at com.cloud.hypervisor.kvm.resource.wrapper.
> > > > > > > > > LibvirtSecurityGroupRulesCommandWrapper.execute(
> > > > > > > > > LibvirtSecurityGroupRulesCommandWrapper.java:57)
> > > > > > > > > at com.cloud.hypervisor.kvm.resource.wrapper.
> > > > > > > > > LibvirtSecurityGroupRulesCommandWrapper.execute(
> > > > > > > > > LibvirtSecurityGroupRulesCommandWrapper.java:36)
> > > > > > > > > at com.cloud.hypervisor.kvm.resource.wrapper.
> > > > > > > > > LibvirtRequestWrapper.execute(
> LibvirtRequestWrapper.java:75)
> > > > > > > > > at com.cloud.hypervisor.kvm.resource.
> > LibvirtComputingResource.
> > > > > > > > > executeRequest(LibvirtComputingResource.java:1369)
> > > > > > > > > at com.cloud.agent.Agent.processRequest(Agent.java:525)
> > > > > > > > > at com.cloud.agent.Agent$AgentRequestHandler.doTask(
> > > > > Agent.java:833)
> > > > > > > > > at com.cloud.utils.nio.Task.call(Task.java:83)
> > > > > > > > > at com.cloud.utils.nio.Task.call(Task.java:29)
> > > > > > > > > at java.util.concurrent.FutureTask.run(FutureTask.
> java:266)
> > > > > > > > > at java.util.concurrent.ThreadPoolExecutor.runWorker(
> > > > > > > > > ThreadPoolExecutor.java:1149)
> > > > > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > > > > > > > > ThreadPoolExecutor.java:624)
> > > > > > > > > at java.lang.Thread.run(Thread.java:748)
> > > > > > > > > 2017-11-11 15:40:41,327 WARN  [resource.wrapper.
> > > > > > > > > LibvirtSecurityGroupRulesCommandWrapper]
> > > > > > (agentRequest-Handler-2:null)
> > > > > > > > > (logid:eab9a328) Failed to program network rules for vm
> > > i-2-7-VM
> > > > > > > > >
> > > > > > > > > So, no rules are actually created. Script doesn't call... I
> > > > suppose
> > > > > > may
> > > > > > > > be
> > > > > > > > > quotes are required because shell interprets ';' as command
> > > > > > separator.
> > > > > > > I
> > > > > > > > > suppose that optimization introduced in 4.10, because in
> 4.9
> > > SGs
> > > > > work
> > > > > > > > like
> > > > > > > > > a charm...
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > 2017-11-11 3:15 GMT+07:00 Paul Angus <
> > paul.angus@shapeblue.com
> > > >:
> > > > > > > > >
> > > > > > > > >> Ivan,
> > > > > > > > >>
> > > > > > > > >> Can you paste a larger section of unfiltered logs.  There
> > > would
> > > > > > always
> > > > > > > > be
> > > > > > > > >> a message explaining why the mgmt. server thought that a
> VR
> > > > should
> > > > > > be
> > > > > > > > shut
> > > > > > > > >> down
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >> Kind regards,
> > > > > > > > >>
> > > > > > > > >> Paul Angus
> > > > > > > > >>
> > > > > > > > >> paul.angus@shapeblue.com
> > > > > > > > >> www.shapeblue.com
> > > > > > > > >> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > > > > > > >> @shapeblue
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >> -----Original Message-----
> > > > > > > > >> From: Simon Weller [mailto:sweller@ena.com.INVALID]
> > > > > > > > >> Sent: 10 November 2017 18:39
> > > > > > > > >> To: dev@cloudstack.apache.org
> > > > > > > > >> Subject: Re: Apache CloudStack 4.10 VR/BasicZone/KVM
> Problem
> > > > > > > > >>
> > > > > > > > >> What VR template image are you using?
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >> ________________________________
> > > > > > > > >> From: Ivan Kudryavtsev <ku...@bw-sw.com>
> > > > > > > > >> Sent: Friday, November 10, 2017 11:59 AM
> > > > > > > > >> To: dev@cloudstack.apache.org
> > > > > > > > >> Subject: Re: Apache CloudStack 4.10 VR/BasicZone/KVM
> Problem
> > > > > > > > >>
> > > > > > > > >> Hi. No, regular NFS. VR starts great, but stopped by ms,
> > other
> > > > > > system
> > > > > > > > vms
> > > > > > > > >> are working. I even added to communication script on
> compute
> > > > node
> > > > > > > "sleep
> > > > > > > > >> 3600" before ssh, so response to management is delayed, I
> > > logged
> > > > > so
> > > > > > to
> > > > > > > > VR,
> > > > > > > > >> all interfaces are up, iptables rules are OK.
> > > > > > > > >>
> > > > > > > > >> So agent rolls vr good, but stops it by management order
> > with
> > > no
> > > > > > > obvious
> > > > > > > > >> reason.
> > > > > > > > >>
> > > > > > > > >> 11 нояб. 2017 г. 0:54 пользователь "Simon Weller"
> > > > > > > > <sweller@ena.com.invalid
> > > > > > > > >> >
> > > > > > > > >> написал:
> > > > > > > > >>
> > > > > > > > >> > Is the storage ceph?
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >> > ________________________________
> > > > > > > > >> > From: Ivan Kudryavtsev <ku...@bw-sw.com>
> > > > > > > > >> > Sent: Friday, November 10, 2017 11:52 AM
> > > > > > > > >> > To: dev@cloudstack.apache.org
> > > > > > > > >> > Subject: Re: Apache CloudStack 4.10 VR/BasicZone/KVM
> > Problem
> > > > > > > > >> >
> > > > > > > > >> > Hi, I did, and it does the things right, I even added
> > "tee"
> > > to
> > > > > ssh
> > > > > > > > >> > 3922 communication script to out vr response to
> additional
> > > log
> > > > > and
> > > > > > > it
> > > > > > > > >> > only receives VR version line and sends all info (the
> same
> > > > from
> > > > > > > > >> > pastebin) to ACS and receives "stop" order.
> > > > > > > > >> >
> > > > > > > > >> > I'll try to provide additional info, but ad you can see,
> > > > > > management
> > > > > > > > >> > receives proper response and sends stop next op. It
> looks
> > > very
> > > > > > > freaky
> > > > > > > > >> > without any notification...
> > > > > > > > >> >
> > > > > > > > >> > 11 нояб. 2017 г. 0:37 пользователь "Simon Weller"
> > > > > > > > >> > <sweller@ena.com.invalid
> > > > > > > > >> > >
> > > > > > > > >> > написал:
> > > > > > > > >> >
> > > > > > > > >> > > Ivan,
> > > > > > > > >> > >
> > > > > > > > >> > >
> > > > > > > > >> > > Can you put the host agents into debug mode? Hopefully
> > > that
> > > > > will
> > > > > > > > >> > > provide more information.
> > > > > > > > >> > >
> > > > > > > > >> > >
> > > > > > > > >> > > https://cwiki.apache.org/
> confluence/display/CLOUDSTACK/
> > > > > > > > KVM+agent+deb
> > > > > > > > >> > > ug
> > > > > > > > >> KVM agent debug - Apache Cloudstack - Apache Software ...<
> > > > > > > > >> https://cwiki.apache.org/confluence/display/CLOUDSTACK/
> > > > > > > KVM+agent+debug>
> > > > > > > > >> cwiki.apache.org
> > > > > > > > >> Steps to debug the KVM agent from eclipse: In KVM agent
> edit
> > > > > > > > >> '/usr/libexec/agent-runner ', add "-Xrunjdwp:transport=dt_
> > > > > > > > socket,address=8787
> > > > > > > > >> ...
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >> > >
> > > > > > > > >> > >
> > > > > > > > >> > > - Si
> > > > > > > > >> > >
> > > > > > > > >> > > ________________________________
> > > > > > > > >> > > From: Ivan Kudryavtsev <ku...@bw-sw.com>
> > > > > > > > >> > > Sent: Friday, November 10, 2017 11:34 AM
> > > > > > > > >> > > To: dev@cloudstack.apache.org
> > > > > > > > >> > > Subject: Apache CloudStack 4.10 VR/BasicZone/KVM
> Problem
> > > > > > > > >> > >
> > > > > > > > >> > > Hello, Devs.
> > > > > > > > >> > >
> > > > > > > > >> > > I experience VR Start Problem in the fresh ACS 4.10
> > > > deployment
> > > > > > > > >> > >
> > > > > > > > >> > > Intersting place of logs is here:
> > > > > https://pastebin.com/iBXRBA5N
> > > > > > > > >> [https://pastebin.com/i/facebook.png]<https://
> > > > > pastebin.com/iBXRBA5N
> > > > > > >
> > > > > > > > >>
> > > > > > > > >> 2017-11-10 23:05:35,853 DEBUG [c.c.a.t.Request]
> > > > > > > > >> (Work-Job-Executor-15:ctx-6fdf61 - Pastebin.com<
> > > > https://pastebin.
> > > > > > > > >> com/iBXRBA5N>
> > > > > > > > >> pastebin.com
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >> > >
> > > > > > > > >> > > Basically, the situation looks like:
> > > > > > > > >> > >
> > > > > > > > >> > > 1. Management Server tries to launch VR 2. It gets
> from
> > > > Agent
> > > > > > > proper
> > > > > > > > >> > > VR response with VR details 3. It sends StopCommand
> > > without
> > > > > > > > >> > > explanation.
> > > > > > > > >> > >
> > > > > > > > >> > > I'm trying to figure out what happens inside, but the
> > > > codebase
> > > > > > is
> > > > > > > > >> > > huge
> > > > > > > > >> > and
> > > > > > > > >> > > still no positive results. Please, let me know if you
> > have
> > > > any
> > > > > > > ideas
> > > > > > > > >> > which
> > > > > > > > >> > > could help me finding the reason. Thanks a lot.
> > > > > > > > >> > >
> > > > > > > > >> > > --
> > > > > > > > >> > > With best regards, Ivan Kudryavtsev
> > > > > > > > >> > > Bitworks Software, Ltd.
> > > > > > > > >> > > Cell: +7-923-414-1515
> > > > > > > > >> > > WWW: http://bitworks.software/ <http://bw-sw.com/>
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > With best regards, Ivan Kudryavtsev
> > > > > > > > > Bitworks Software, Ltd.
> > > > > > > > > Cell: +7-923-414-1515
> > > > > > > > > WWW: http://bitworks.software/ <http://bw-sw.com/>
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > With best regards, Ivan Kudryavtsev
> > > > > > > > Bitworks Software, Ltd.
> > > > > > > > Cell: +7-923-414-1515
> > > > > > > > WWW: http://bitworks.software/ <http://bw-sw.com/>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > With best regards, Ivan Kudryavtsev
> > > > > > Bitworks Software, Ltd.
> > > > > > Cell: +7-923-414-1515
> > > > > > WWW: http://bitworks.software/ <http://bw-sw.com/>
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Apache CloudStack 4.10 VR/BasicZone/KVM Problem

[Ivan Kudryavtsev <ku...@bw-sw.com>]

I don't know if 4.10 will be improved centrally. AFAIK 4.11 WIP right now...

13 нояб. 2017 г. 16:37 пользователь "Özhan Rüzgar Karaman" <
oruzgarkaraman@gmail.com> написал:

> Hi Ivan;
> I like to try virtio-scsi, i like to use its benefits like enlarging data
> disks without rebooting vm and getting unused space in vm back to my
> central storage, we also waited for long time, lets wait some more for 4.10
> to become more stable :) Thanks for all replies
>
> Özhan
>
> On Mon, Nov 13, 2017 at 12:29 PM, Ivan Kudryavtsev <
> kudryavtsev_ia@bw-sw.com
> > wrote:
>
> > Ozhan, I suggest 4.10 only if you need KVM VM snapshots, virtio-scsi and
> > ipv6. Don't see other reasons to go from 4.9 (at least for kvm). But I
> > waited for 4.10 for a long time...
> >
> > 13 нояб. 2017 г. 16:21 пользователь "Özhan Rüzgar Karaman" <
> > oruzgarkaraman@gmail.com> написал:
> >
> > > ok we will try to compile 4.10 from source and try it on our test
> > > environment.
> > >
> > > We currently use 4.9.x on production, i make this test for deciding if
> > 4.10
> > > is suitable & stable for us, we also use very simple and generic setup
> > for
> > > this test no ipv6 just simple environment . So for production it looks
> > like
> > > we need to wait some time for upcoming releases, what do you think
> about
> > > that?
> > >
> > > There was a thread about this question one month ago and i remember
> that
> > > most of people still stick to 4.9 release for their production
> > > environments...
> > >
> > > Thanks
> > > Özhan
> > >
> > > On Mon, Nov 13, 2017 at 12:18 PM, Wei ZHOU <us...@gmail.com>
> > wrote:
> > >
> > > > Hi Ivan,
> > > >
> > > > I would suggest you to create jira tickets for each problem you found
> > in
> > > > your testing, and create a github pull request for a jira ticket.
> > > > It is convenient for reviewers.
> > > >
> > > > Kind regards,
> > > > Wei
> > > >
> > > > 2017-11-13 10:01 GMT+01:00 Ivan Kudryavtsev <
> kudryavtsev_ia@bw-sw.com
> > >:
> > > >
> > > > > Hello, Ozhan
> > > > >
> > > > > https://github.com/apache/cloudstack/pull/2320
> > > > >
> > > > > fixes everything I found right now. It enables functioning of
> > > everything
> > > > > correctly even if no IPv6 CIDR specified for network (at least for
> > > Ubuntu
> > > > > 14.04).
> > > > > For IPv6 configuration instruction please take a look at:
> > > > > https://github.com/apache/cloudstack/commit/
> > > > f10c8bfe0c99a762c2606459413a47
> > > > > 219614e775
> > > > > (oh my god,I spend several hours trying to find how to configure
> IPv6
> > > for
> > > > > 4.10).
> > > > >
> > > > > Please, don't forget to recreate SSVM because there is a fix for
> > > > templates
> > > > > too:
> > > > > https://github.com/apache/cloudstack/pull/2322
> > > > >
> > > > >
> > > > > 2017-11-13 15:51 GMT+07:00 Özhan Rüzgar Karaman <
> > > > oruzgarkaraman@gmail.com
> > > > > >:
> > > > >
> > > > > > Hi Ivan;
> > > > > > Does this hotfixes also solve qoutes and shell script interprets
> > > > problem?
> > > > > > We have no ipv6 setup and today we made similar test with fresh
> > > install
> > > > > > 4.10. We noticed that we receive similar error on security groups
> > > stage
> > > > > > while br_netfilter module is already active on our environment.
> We
> > > made
> > > > > > same tests for Ubuntu 16.04.3 and 14.04.5 kvm hosts
> > > > > >
> > > > > > Logs are below:
> > > > > > 2017-11-13 11:47:41,773 DEBUG [kvm.resource.
> > > LibvirtComputingResource]
> > > > > > (agentRequest-Handler-1:null) Executing:
> > > > > > /usr/share/cloudstack-common/scripts/vm/network/security_
> group.py
> > > > > > add_network_rules --vmname i-2-5-VM --vmid 5 --vmip 192.168.18.6
> > > > --vmip6
> > > > > > null --sig 74a6d8c403af9c3c7b89ecf206e4ac26 --seq 16 --vmmac
> > > > > > 1e:00:9b:00:00:05 --vif vnet8 --brname breth0-23 --nicsecips 0:
> > > --rules
> > > > > > I:tcp:1:65535:
> > > > > > 0.0.0.0/0,NEXT;I:udp:1:65535:0.0.0.0/0,NEXT;E:tcp:1:65535:
> > > > 0.0.0.0/0,NEXT
> > > > > ;
> > > > > > 2017-11-13 11:47:41,773 WARN  [kvm.resource.
> > > LibvirtComputingResource]
> > > > > > (agentRequest-Handler-1:null) Exception:
> > > > > > /usr/share/cloudstack-common/scripts/vm/network/security_
> group.py
> > > > > > add_network_rules --vmname i-2-5-VM --vmid 5 --vmip 192.168.18.6
> > > > --vmip6
> > > > > > null --sig 74a6d8c403af9c3c7b89ecf206e4ac26 --seq 16 --vmmac
> > > > > > 1e:00:9b:00:00:05 --vif vnet8 --brname breth0-23 --nicsecips 0:
> > > --rules
> > > > > > I:tcp:1:65535:
> > > > > > 0.0.0.0/0,NEXT;I:udp:1:65535:0.0.0.0/0,NEXT;E:tcp:1:65535:
> > > > 0.0.0.0/0,NEXT
> > > > > ;
> > > > > > java.lang.NullPointerException
> > > > > > at java.lang.ProcessBuilder.start(ProcessBuilder.java:1012)
> > > > > > at com.cloud.utils.script.Script.execute(Script.java:214)
> > > > > > at com.cloud.utils.script.Script.execute(Script.java:182)
> > > > > > at
> > > > > > com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.
> > > > > > addNetworkRules(LibvirtComputingResource.java:3429)
> > > > > > at
> > > > > > com.cloud.hypervisor.kvm.resource.wrapper.
> > > > LibvirtSecurityGroupRulesComma
> > > > > > ndWrapper.execute(LibvirtSecurityGroupRulesComma
> ndWrapper.java:57)
> > > > > > at
> > > > > > com.cloud.hypervisor.kvm.resource.wrapper.
> > > > LibvirtSecurityGroupRulesComma
> > > > > > ndWrapper.execute(LibvirtSecurityGroupRulesComma
> ndWrapper.java:36)
> > > > > > at
> > > > > > com.cloud.hypervisor.kvm.resource.wrapper.
> > > > LibvirtRequestWrapper.execute(
> > > > > > LibvirtRequestWrapper.java:75)
> > > > > > at
> > > > > > com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.
> > > > > executeRequest(
> > > > > > LibvirtComputingResource.java:1369)
> > > > > > at com.cloud.agent.Agent.processRequest(Agent.java:525)
> > > > > > at com.cloud.agent.Agent$AgentRequestHandler.doTask(
> > Agent.java:833)
> > > > > > at com.cloud.utils.nio.Task.call(Task.java:83)
> > > > > > at com.cloud.utils.nio.Task.call(Task.java:29)
> > > > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > > > > at
> > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(
> > > > > > ThreadPoolExecutor.java:1149)
> > > > > > at
> > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > > > > > ThreadPoolExecutor.java:624)
> > > > > > at java.lang.Thread.run(Thread.java:748)
> > > > > > 2017-11-13 11:47:41,774 WARN
> > > > > > [resource.wrapper.LibvirtSecurityGroupRulesCommandWrapper]
> > > > > > (agentRequest-Handler-1:null) Failed to program network rules for
> > vm
> > > > > > i-2-5-VM
> > > > > > 2017-11-13 11:47:41,775 DEBUG [cloud.agent.Agent]
> > > > > > (agentRequest-Handler-1:null) Seq 1-6412562919422165093:  { Ans:
> ,
> > > > > MgmtId:
> > > > > > 345048635880, via: 1, Ver: v1, Flags: 110,
> > > > > > [{"com.cloud.agent.api.SecurityGroupRuleAnswer":{"
> > > > > > logSequenceNumber":16,"vmId":5,"reason":"PROGRAMMING_
> > > > > > FAILED","result":false,"details":"programming
> > > > > > network rules failed","wait":0}}] }
> > > > > >
> > > > > >
> > > > > > When we execute command with double quotas for rules section from
> > > > command
> > > > > > line it executes without a problem like below:
> > > > > > root@kvmt3:/var/log/cloudstack/agent#
> > > > > > /usr/share/cloudstack-common/scripts/vm/network/security_
> group.py
> > > > > > add_network_rules --vmname i-2-5-VM --vmid 5 --vmip 192.168.18.6
> > > > --vmip6
> > > > > > null --sig 74a6d8c403af9c3c7b89ecf206e4ac26 --seq 16 --vmmac
> > > > > > 1e:00:9b:00:00:05 --vif vnet8 --brname breth0-23 --nicsecips 0:
> > > --rules
> > > > > > "I:tcp:1:65535:
> > > > > > 0.0.0.0/0,NEXT;I:udp:1:65535:0.0.0.0/0,NEXT;E:tcp:1:65535:
> > > > 0.0.0.0/0,NEXT
> > > > > ;"
> > > > > > root@kvmt3:/var/log/cloudstack/agent# echo $?
> > > > > > 0
> > > > > > root@kvmt3:/var/log/cloudstack/agent#
> > > > > >
> > > > > > Thanks
> > > > > > Özhan
> > > > > >
> > > > > >
> > > > > > On Sat, Nov 11, 2017 at 6:59 PM, Ivan Kudryavtsev <
> > > > > > kudryavtsev_ia@bw-sw.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hello, I implemented some hotfixes for 4.10 to work
> > > > > > >
> > > > > > > https://github.com/apache/cloudstack/pull/2319 - to master
> (load
> > > > > > > br_netfilter module)
> > > > > > > https://github.com/apache/cloudstack/pull/2320 - to 4.10 which
> > > fixes
> > > > > SG
> > > > > > > failures related to ipv6.
> > > > > > >
> > > > > > >
> > > > > > > 2017-11-11 15:51 GMT+07:00 Ivan Kudryavtsev <
> > > > kudryavtsev_ia@bw-sw.com
> > > > > >:
> > > > > > >
> > > > > > > > Following up with previous question. I managed to make it
> work
> > by
> > > > > > > removing
> > > > > > > > all and heading to ubuntu 14.04 hypervisor host.
> > > > > > > >
> > > > > > > > Also, what I found more:
> > > > > > > >
> > > > > > > > 1. when setup databases (management server) if custom port is
> > > > > > specified,
> > > > > > > > databases themself is not created. If create manually, import
> > > > scripts
> > > > > > > work
> > > > > > > > fine.
> > > > > > > > 2. UI: unable to download ISO to __all__ zones. Have to
> specify
> > > > > certain
> > > > > > > > zone, else UI gives an error.
> > > > > > > > 3. Ubuntu doesn't load module *br_netfilter* but
> > > > > > > >
> > > > > > > > /usr/share/cloudstack-common/scripts/vm/network/security_
> > > group.py
> > > > > > > >
> > > > > > > > uses it and nothing good as a result:
> > > > > > > >
> > > > > > > > 2017-11-11 15:38:29,241 - sysctl -w
> net.bridge.bridge-nf-call-
> > > > > > > arptables=1
> > > > > > > > 2017-11-11 15:38:29,244 - sysctl -w
> net.bridge.bridge-nf-call-
> > > > > > iptables=1
> > > > > > > > 2017-11-11 15:38:29,247 - sysctl -w
> net.bridge.bridge-nf-call-
> > > > > > > ip6tables=1
> > > > > > > >
> > > > > > > > adding br_netfilter to /etc/modules fixes it. I suppose it's
> > > > > necessary
> > > > > > to
> > > > > > > > add
> > > > > > > >
> > > > > > > > in script something like: modprobe br_netfilter (or smarter
> > > thing).
> > > > > > > >
> > > > > > > > But It doesn't work completely, actually, security groups are
> > > > > > > unfunctional:
> > > > > > > >
> > > > > > > > ==> /var/log/cloudstack/agent/agent.log <==
> > > > > > > > 2017-11-11 15:40:41,326 WARN  [kvm.resource.
> > > > > LibvirtComputingResource]
> > > > > > > > (agentRequest-Handler-2:null) (logid:eab9a328) Exception:
> > > > > > > > /usr/share/cloudstack-common/scripts/vm/network/security_
> > > group.py
> > > > > > > > add_network_rules --vmname i-2-7-VM --vmid 7 --vmip
> > 176.120.28.4
> > > > > > --vmip6
> > > > > > > > null --sig d60255deb618b7be9f477eed10d65234 --seq 4 --vmmac
> > > > > > > > 1e:00:6f:00:01:01 --vif vnet8 --brname cloudbr0 --nicsecips
> 0:
> > > > > --rules
> > > > > > > > I:icmp:-1:-1:0.0.0.0/0,NEXT;I:tcp:1:65535:0.0.0.0/0,NEXT;I:
> > > > > > > > udp:1:65535:0.0.0.0/0,NEXT;E:icmp:-1:-1:0.0.0.0/0,NEXT;E:
> > > > > > > > tcp:1:65535:0.0.0.0/0,NEXT;E:udp:1:65535:0.0.0.0/0,NEXT;
> > > > > > > > java.lang.NullPointerException
> > > > > > > > at java.lang.ProcessBuilder.start(ProcessBuilder.java:1012)
> > > > > > > > at com.cloud.utils.script.Script.execute(Script.java:214)
> > > > > > > > at com.cloud.utils.script.Script.execute(Script.java:182)
> > > > > > > > at com.cloud.hypervisor.kvm.resource.
> LibvirtComputingResource.
> > > > > > > > addNetworkRules(LibvirtComputingResource.java:3429)
> > > > > > > > at com.cloud.hypervisor.kvm.resource.wrapper.
> > > > > > > > LibvirtSecurityGroupRulesCommandWrapper.execute(
> > > > > > > > LibvirtSecurityGroupRulesCommandWrapper.java:57)
> > > > > > > > at com.cloud.hypervisor.kvm.resource.wrapper.
> > > > > > > > LibvirtSecurityGroupRulesCommandWrapper.execute(
> > > > > > > > LibvirtSecurityGroupRulesCommandWrapper.java:36)
> > > > > > > > at com.cloud.hypervisor.kvm.resource.wrapper.
> > > > > > > > LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:75)
> > > > > > > > at com.cloud.hypervisor.kvm.resource.
> LibvirtComputingResource.
> > > > > > > > executeRequest(LibvirtComputingResource.java:1369)
> > > > > > > > at com.cloud.agent.Agent.processRequest(Agent.java:525)
> > > > > > > > at com.cloud.agent.Agent$AgentRequestHandler.doTask(
> > > > Agent.java:833)
> > > > > > > > at com.cloud.utils.nio.Task.call(Task.java:83)
> > > > > > > > at com.cloud.utils.nio.Task.call(Task.java:29)
> > > > > > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > > > > > > at java.util.concurrent.ThreadPoolExecutor.runWorker(
> > > > > > > > ThreadPoolExecutor.java:1149)
> > > > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > > > > > > > ThreadPoolExecutor.java:624)
> > > > > > > > at java.lang.Thread.run(Thread.java:748)
> > > > > > > > 2017-11-11 15:40:41,327 WARN  [resource.wrapper.
> > > > > > > > LibvirtSecurityGroupRulesCommandWrapper]
> > > > > (agentRequest-Handler-2:null)
> > > > > > > > (logid:eab9a328) Failed to program network rules for vm
> > i-2-7-VM
> > > > > > > >
> > > > > > > > So, no rules are actually created. Script doesn't call... I
> > > suppose
> > > > > may
> > > > > > > be
> > > > > > > > quotes are required because shell interprets ';' as command
> > > > > separator.
> > > > > > I
> > > > > > > > suppose that optimization introduced in 4.10, because in 4.9
> > SGs
> > > > work
> > > > > > > like
> > > > > > > > a charm...
> > > > > > > >
> > > > > > > >
> > > > > > > > 2017-11-11 3:15 GMT+07:00 Paul Angus <
> paul.angus@shapeblue.com
> > >:
> > > > > > > >
> > > > > > > >> Ivan,
> > > > > > > >>
> > > > > > > >> Can you paste a larger section of unfiltered logs.  There
> > would
> > > > > always
> > > > > > > be
> > > > > > > >> a message explaining why the mgmt. server thought that a VR
> > > should
> > > > > be
> > > > > > > shut
> > > > > > > >> down
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> Kind regards,
> > > > > > > >>
> > > > > > > >> Paul Angus
> > > > > > > >>
> > > > > > > >> paul.angus@shapeblue.com
> > > > > > > >> www.shapeblue.com
> > > > > > > >> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > > > > > >> @shapeblue
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> -----Original Message-----
> > > > > > > >> From: Simon Weller [mailto:sweller@ena.com.INVALID]
> > > > > > > >> Sent: 10 November 2017 18:39
> > > > > > > >> To: dev@cloudstack.apache.org
> > > > > > > >> Subject: Re: Apache CloudStack 4.10 VR/BasicZone/KVM Problem
> > > > > > > >>
> > > > > > > >> What VR template image are you using?
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> ________________________________
> > > > > > > >> From: Ivan Kudryavtsev <ku...@bw-sw.com>
> > > > > > > >> Sent: Friday, November 10, 2017 11:59 AM
> > > > > > > >> To: dev@cloudstack.apache.org
> > > > > > > >> Subject: Re: Apache CloudStack 4.10 VR/BasicZone/KVM Problem
> > > > > > > >>
> > > > > > > >> Hi. No, regular NFS. VR starts great, but stopped by ms,
> other
> > > > > system
> > > > > > > vms
> > > > > > > >> are working. I even added to communication script on compute
> > > node
> > > > > > "sleep
> > > > > > > >> 3600" before ssh, so response to management is delayed, I
> > logged
> > > > so
> > > > > to
> > > > > > > VR,
> > > > > > > >> all interfaces are up, iptables rules are OK.
> > > > > > > >>
> > > > > > > >> So agent rolls vr good, but stops it by management order
> with
> > no
> > > > > > obvious
> > > > > > > >> reason.
> > > > > > > >>
> > > > > > > >> 11 нояб. 2017 г. 0:54 пользователь "Simon Weller"
> > > > > > > <sweller@ena.com.invalid
> > > > > > > >> >
> > > > > > > >> написал:
> > > > > > > >>
> > > > > > > >> > Is the storage ceph?
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > ________________________________
> > > > > > > >> > From: Ivan Kudryavtsev <ku...@bw-sw.com>
> > > > > > > >> > Sent: Friday, November 10, 2017 11:52 AM
> > > > > > > >> > To: dev@cloudstack.apache.org
> > > > > > > >> > Subject: Re: Apache CloudStack 4.10 VR/BasicZone/KVM
> Problem
> > > > > > > >> >
> > > > > > > >> > Hi, I did, and it does the things right, I even added
> "tee"
> > to
> > > > ssh
> > > > > > > >> > 3922 communication script to out vr response to additional
> > log
> > > > and
> > > > > > it
> > > > > > > >> > only receives VR version line and sends all info (the same
> > > from
> > > > > > > >> > pastebin) to ACS and receives "stop" order.
> > > > > > > >> >
> > > > > > > >> > I'll try to provide additional info, but ad you can see,
> > > > > management
> > > > > > > >> > receives proper response and sends stop next op. It looks
> > very
> > > > > > freaky
> > > > > > > >> > without any notification...
> > > > > > > >> >
> > > > > > > >> > 11 нояб. 2017 г. 0:37 пользователь "Simon Weller"
> > > > > > > >> > <sweller@ena.com.invalid
> > > > > > > >> > >
> > > > > > > >> > написал:
> > > > > > > >> >
> > > > > > > >> > > Ivan,
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> > > Can you put the host agents into debug mode? Hopefully
> > that
> > > > will
> > > > > > > >> > > provide more information.
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> > > https://cwiki.apache.org/confluence/display/CLOUDSTACK/
> > > > > > > KVM+agent+deb
> > > > > > > >> > > ug
> > > > > > > >> KVM agent debug - Apache Cloudstack - Apache Software ...<
> > > > > > > >> https://cwiki.apache.org/confluence/display/CLOUDSTACK/
> > > > > > KVM+agent+debug>
> > > > > > > >> cwiki.apache.org
> > > > > > > >> Steps to debug the KVM agent from eclipse: In KVM agent edit
> > > > > > > >> '/usr/libexec/agent-runner ', add "-Xrunjdwp:transport=dt_
> > > > > > > socket,address=8787
> > > > > > > >> ...
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> > > - Si
> > > > > > > >> > >
> > > > > > > >> > > ________________________________
> > > > > > > >> > > From: Ivan Kudryavtsev <ku...@bw-sw.com>
> > > > > > > >> > > Sent: Friday, November 10, 2017 11:34 AM
> > > > > > > >> > > To: dev@cloudstack.apache.org
> > > > > > > >> > > Subject: Apache CloudStack 4.10 VR/BasicZone/KVM Problem
> > > > > > > >> > >
> > > > > > > >> > > Hello, Devs.
> > > > > > > >> > >
> > > > > > > >> > > I experience VR Start Problem in the fresh ACS 4.10
> > > deployment
> > > > > > > >> > >
> > > > > > > >> > > Intersting place of logs is here:
> > > > https://pastebin.com/iBXRBA5N
> > > > > > > >> [https://pastebin.com/i/facebook.png]<https://
> > > > pastebin.com/iBXRBA5N
> > > > > >
> > > > > > > >>
> > > > > > > >> 2017-11-10 23:05:35,853 DEBUG [c.c.a.t.Request]
> > > > > > > >> (Work-Job-Executor-15:ctx-6fdf61 - Pastebin.com<
> > > https://pastebin.
> > > > > > > >> com/iBXRBA5N>
> > > > > > > >> pastebin.com
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> > >
> > > > > > > >> > > Basically, the situation looks like:
> > > > > > > >> > >
> > > > > > > >> > > 1. Management Server tries to launch VR 2. It gets from
> > > Agent
> > > > > > proper
> > > > > > > >> > > VR response with VR details 3. It sends StopCommand
> > without
> > > > > > > >> > > explanation.
> > > > > > > >> > >
> > > > > > > >> > > I'm trying to figure out what happens inside, but the
> > > codebase
> > > > > is
> > > > > > > >> > > huge
> > > > > > > >> > and
> > > > > > > >> > > still no positive results. Please, let me know if you
> have
> > > any
> > > > > > ideas
> > > > > > > >> > which
> > > > > > > >> > > could help me finding the reason. Thanks a lot.
> > > > > > > >> > >
> > > > > > > >> > > --
> > > > > > > >> > > With best regards, Ivan Kudryavtsev
> > > > > > > >> > > Bitworks Software, Ltd.
> > > > > > > >> > > Cell: +7-923-414-1515
> > > > > > > >> > > WWW: http://bitworks.software/ <http://bw-sw.com/>
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > > >>
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > With best regards, Ivan Kudryavtsev
> > > > > > > > Bitworks Software, Ltd.
> > > > > > > > Cell: +7-923-414-1515
> > > > > > > > WWW: http://bitworks.software/ <http://bw-sw.com/>
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > With best regards, Ivan Kudryavtsev
> > > > > > > Bitworks Software, Ltd.
> > > > > > > Cell: +7-923-414-1515
> > > > > > > WWW: http://bitworks.software/ <http://bw-sw.com/>
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > With best regards, Ivan Kudryavtsev
> > > > > Bitworks Software, Ltd.
> > > > > Cell: +7-923-414-1515
> > > > > WWW: http://bitworks.software/ <http://bw-sw.com/>
> > > > >
> > > >
> > >
> >
>

Re: Apache CloudStack 4.10 VR/BasicZone/KVM Problem

[Özhan Rüzgar Karaman <or...@gmail.com>]

Hi Ivan;
I like to try virtio-scsi, i like to use its benefits like enlarging data
disks without rebooting vm and getting unused space in vm back to my
central storage, we also waited for long time, lets wait some more for 4.10
to become more stable :) Thanks for all replies

Özhan

On Mon, Nov 13, 2017 at 12:29 PM, Ivan Kudryavtsev <kudryavtsev_ia@bw-sw.com
> wrote:

> Ozhan, I suggest 4.10 only if you need KVM VM snapshots, virtio-scsi and
> ipv6. Don't see other reasons to go from 4.9 (at least for kvm). But I
> waited for 4.10 for a long time...
>
> 13 нояб. 2017 г. 16:21 пользователь "Özhan Rüzgar Karaman" <
> oruzgarkaraman@gmail.com> написал:
>
> > ok we will try to compile 4.10 from source and try it on our test
> > environment.
> >
> > We currently use 4.9.x on production, i make this test for deciding if
> 4.10
> > is suitable & stable for us, we also use very simple and generic setup
> for
> > this test no ipv6 just simple environment . So for production it looks
> like
> > we need to wait some time for upcoming releases, what do you think about
> > that?
> >
> > There was a thread about this question one month ago and i remember that
> > most of people still stick to 4.9 release for their production
> > environments...
> >
> > Thanks
> > Özhan
> >
> > On Mon, Nov 13, 2017 at 12:18 PM, Wei ZHOU <us...@gmail.com>
> wrote:
> >
> > > Hi Ivan,
> > >
> > > I would suggest you to create jira tickets for each problem you found
> in
> > > your testing, and create a github pull request for a jira ticket.
> > > It is convenient for reviewers.
> > >
> > > Kind regards,
> > > Wei
> > >
> > > 2017-11-13 10:01 GMT+01:00 Ivan Kudryavtsev <kudryavtsev_ia@bw-sw.com
> >:
> > >
> > > > Hello, Ozhan
> > > >
> > > > https://github.com/apache/cloudstack/pull/2320
> > > >
> > > > fixes everything I found right now. It enables functioning of
> > everything
> > > > correctly even if no IPv6 CIDR specified for network (at least for
> > Ubuntu
> > > > 14.04).
> > > > For IPv6 configuration instruction please take a look at:
> > > > https://github.com/apache/cloudstack/commit/
> > > f10c8bfe0c99a762c2606459413a47
> > > > 219614e775
> > > > (oh my god,I spend several hours trying to find how to configure IPv6
> > for
> > > > 4.10).
> > > >
> > > > Please, don't forget to recreate SSVM because there is a fix for
> > > templates
> > > > too:
> > > > https://github.com/apache/cloudstack/pull/2322
> > > >
> > > >
> > > > 2017-11-13 15:51 GMT+07:00 Özhan Rüzgar Karaman <
> > > oruzgarkaraman@gmail.com
> > > > >:
> > > >
> > > > > Hi Ivan;
> > > > > Does this hotfixes also solve qoutes and shell script interprets
> > > problem?
> > > > > We have no ipv6 setup and today we made similar test with fresh
> > install
> > > > > 4.10. We noticed that we receive similar error on security groups
> > stage
> > > > > while br_netfilter module is already active on our environment. We
> > made
> > > > > same tests for Ubuntu 16.04.3 and 14.04.5 kvm hosts
> > > > >
> > > > > Logs are below:
> > > > > 2017-11-13 11:47:41,773 DEBUG [kvm.resource.
> > LibvirtComputingResource]
> > > > > (agentRequest-Handler-1:null) Executing:
> > > > > /usr/share/cloudstack-common/scripts/vm/network/security_group.py
> > > > > add_network_rules --vmname i-2-5-VM --vmid 5 --vmip 192.168.18.6
> > > --vmip6
> > > > > null --sig 74a6d8c403af9c3c7b89ecf206e4ac26 --seq 16 --vmmac
> > > > > 1e:00:9b:00:00:05 --vif vnet8 --brname breth0-23 --nicsecips 0:
> > --rules
> > > > > I:tcp:1:65535:
> > > > > 0.0.0.0/0,NEXT;I:udp:1:65535:0.0.0.0/0,NEXT;E:tcp:1:65535:
> > > 0.0.0.0/0,NEXT
> > > > ;
> > > > > 2017-11-13 11:47:41,773 WARN  [kvm.resource.
> > LibvirtComputingResource]
> > > > > (agentRequest-Handler-1:null) Exception:
> > > > > /usr/share/cloudstack-common/scripts/vm/network/security_group.py
> > > > > add_network_rules --vmname i-2-5-VM --vmid 5 --vmip 192.168.18.6
> > > --vmip6
> > > > > null --sig 74a6d8c403af9c3c7b89ecf206e4ac26 --seq 16 --vmmac
> > > > > 1e:00:9b:00:00:05 --vif vnet8 --brname breth0-23 --nicsecips 0:
> > --rules
> > > > > I:tcp:1:65535:
> > > > > 0.0.0.0/0,NEXT;I:udp:1:65535:0.0.0.0/0,NEXT;E:tcp:1:65535:
> > > 0.0.0.0/0,NEXT
> > > > ;
> > > > > java.lang.NullPointerException
> > > > > at java.lang.ProcessBuilder.start(ProcessBuilder.java:1012)
> > > > > at com.cloud.utils.script.Script.execute(Script.java:214)
> > > > > at com.cloud.utils.script.Script.execute(Script.java:182)
> > > > > at
> > > > > com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.
> > > > > addNetworkRules(LibvirtComputingResource.java:3429)
> > > > > at
> > > > > com.cloud.hypervisor.kvm.resource.wrapper.
> > > LibvirtSecurityGroupRulesComma
> > > > > ndWrapper.execute(LibvirtSecurityGroupRulesCommandWrapper.java:57)
> > > > > at
> > > > > com.cloud.hypervisor.kvm.resource.wrapper.
> > > LibvirtSecurityGroupRulesComma
> > > > > ndWrapper.execute(LibvirtSecurityGroupRulesCommandWrapper.java:36)
> > > > > at
> > > > > com.cloud.hypervisor.kvm.resource.wrapper.
> > > LibvirtRequestWrapper.execute(
> > > > > LibvirtRequestWrapper.java:75)
> > > > > at
> > > > > com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.
> > > > executeRequest(
> > > > > LibvirtComputingResource.java:1369)
> > > > > at com.cloud.agent.Agent.processRequest(Agent.java:525)
> > > > > at com.cloud.agent.Agent$AgentRequestHandler.doTask(
> Agent.java:833)
> > > > > at com.cloud.utils.nio.Task.call(Task.java:83)
> > > > > at com.cloud.utils.nio.Task.call(Task.java:29)
> > > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > > > at
> > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(
> > > > > ThreadPoolExecutor.java:1149)
> > > > > at
> > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > > > > ThreadPoolExecutor.java:624)
> > > > > at java.lang.Thread.run(Thread.java:748)
> > > > > 2017-11-13 11:47:41,774 WARN
> > > > > [resource.wrapper.LibvirtSecurityGroupRulesCommandWrapper]
> > > > > (agentRequest-Handler-1:null) Failed to program network rules for
> vm
> > > > > i-2-5-VM
> > > > > 2017-11-13 11:47:41,775 DEBUG [cloud.agent.Agent]
> > > > > (agentRequest-Handler-1:null) Seq 1-6412562919422165093:  { Ans: ,
> > > > MgmtId:
> > > > > 345048635880, via: 1, Ver: v1, Flags: 110,
> > > > > [{"com.cloud.agent.api.SecurityGroupRuleAnswer":{"
> > > > > logSequenceNumber":16,"vmId":5,"reason":"PROGRAMMING_
> > > > > FAILED","result":false,"details":"programming
> > > > > network rules failed","wait":0}}] }
> > > > >
> > > > >
> > > > > When we execute command with double quotas for rules section from
> > > command
> > > > > line it executes without a problem like below:
> > > > > root@kvmt3:/var/log/cloudstack/agent#
> > > > > /usr/share/cloudstack-common/scripts/vm/network/security_group.py
> > > > > add_network_rules --vmname i-2-5-VM --vmid 5 --vmip 192.168.18.6
> > > --vmip6
> > > > > null --sig 74a6d8c403af9c3c7b89ecf206e4ac26 --seq 16 --vmmac
> > > > > 1e:00:9b:00:00:05 --vif vnet8 --brname breth0-23 --nicsecips 0:
> > --rules
> > > > > "I:tcp:1:65535:
> > > > > 0.0.0.0/0,NEXT;I:udp:1:65535:0.0.0.0/0,NEXT;E:tcp:1:65535:
> > > 0.0.0.0/0,NEXT
> > > > ;"
> > > > > root@kvmt3:/var/log/cloudstack/agent# echo $?
> > > > > 0
> > > > > root@kvmt3:/var/log/cloudstack/agent#
> > > > >
> > > > > Thanks
> > > > > Özhan
> > > > >
> > > > >
> > > > > On Sat, Nov 11, 2017 at 6:59 PM, Ivan Kudryavtsev <
> > > > > kudryavtsev_ia@bw-sw.com>
> > > > > wrote:
> > > > >
> > > > > > Hello, I implemented some hotfixes for 4.10 to work
> > > > > >
> > > > > > https://github.com/apache/cloudstack/pull/2319 - to master (load
> > > > > > br_netfilter module)
> > > > > > https://github.com/apache/cloudstack/pull/2320 - to 4.10 which
> > fixes
> > > > SG
> > > > > > failures related to ipv6.
> > > > > >
> > > > > >
> > > > > > 2017-11-11 15:51 GMT+07:00 Ivan Kudryavtsev <
> > > kudryavtsev_ia@bw-sw.com
> > > > >:
> > > > > >
> > > > > > > Following up with previous question. I managed to make it work
> by
> > > > > > removing
> > > > > > > all and heading to ubuntu 14.04 hypervisor host.
> > > > > > >
> > > > > > > Also, what I found more:
> > > > > > >
> > > > > > > 1. when setup databases (management server) if custom port is
> > > > > specified,
> > > > > > > databases themself is not created. If create manually, import
> > > scripts
> > > > > > work
> > > > > > > fine.
> > > > > > > 2. UI: unable to download ISO to __all__ zones. Have to specify
> > > > certain
> > > > > > > zone, else UI gives an error.
> > > > > > > 3. Ubuntu doesn't load module *br_netfilter* but
> > > > > > >
> > > > > > > /usr/share/cloudstack-common/scripts/vm/network/security_
> > group.py
> > > > > > >
> > > > > > > uses it and nothing good as a result:
> > > > > > >
> > > > > > > 2017-11-11 15:38:29,241 - sysctl -w net.bridge.bridge-nf-call-
> > > > > > arptables=1
> > > > > > > 2017-11-11 15:38:29,244 - sysctl -w net.bridge.bridge-nf-call-
> > > > > iptables=1
> > > > > > > 2017-11-11 15:38:29,247 - sysctl -w net.bridge.bridge-nf-call-
> > > > > > ip6tables=1
> > > > > > >
> > > > > > > adding br_netfilter to /etc/modules fixes it. I suppose it's
> > > > necessary
> > > > > to
> > > > > > > add
> > > > > > >
> > > > > > > in script something like: modprobe br_netfilter (or smarter
> > thing).
> > > > > > >
> > > > > > > But It doesn't work completely, actually, security groups are
> > > > > > unfunctional:
> > > > > > >
> > > > > > > ==> /var/log/cloudstack/agent/agent.log <==
> > > > > > > 2017-11-11 15:40:41,326 WARN  [kvm.resource.
> > > > LibvirtComputingResource]
> > > > > > > (agentRequest-Handler-2:null) (logid:eab9a328) Exception:
> > > > > > > /usr/share/cloudstack-common/scripts/vm/network/security_
> > group.py
> > > > > > > add_network_rules --vmname i-2-7-VM --vmid 7 --vmip
> 176.120.28.4
> > > > > --vmip6
> > > > > > > null --sig d60255deb618b7be9f477eed10d65234 --seq 4 --vmmac
> > > > > > > 1e:00:6f:00:01:01 --vif vnet8 --brname cloudbr0 --nicsecips 0:
> > > > --rules
> > > > > > > I:icmp:-1:-1:0.0.0.0/0,NEXT;I:tcp:1:65535:0.0.0.0/0,NEXT;I:
> > > > > > > udp:1:65535:0.0.0.0/0,NEXT;E:icmp:-1:-1:0.0.0.0/0,NEXT;E:
> > > > > > > tcp:1:65535:0.0.0.0/0,NEXT;E:udp:1:65535:0.0.0.0/0,NEXT;
> > > > > > > java.lang.NullPointerException
> > > > > > > at java.lang.ProcessBuilder.start(ProcessBuilder.java:1012)
> > > > > > > at com.cloud.utils.script.Script.execute(Script.java:214)
> > > > > > > at com.cloud.utils.script.Script.execute(Script.java:182)
> > > > > > > at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.
> > > > > > > addNetworkRules(LibvirtComputingResource.java:3429)
> > > > > > > at com.cloud.hypervisor.kvm.resource.wrapper.
> > > > > > > LibvirtSecurityGroupRulesCommandWrapper.execute(
> > > > > > > LibvirtSecurityGroupRulesCommandWrapper.java:57)
> > > > > > > at com.cloud.hypervisor.kvm.resource.wrapper.
> > > > > > > LibvirtSecurityGroupRulesCommandWrapper.execute(
> > > > > > > LibvirtSecurityGroupRulesCommandWrapper.java:36)
> > > > > > > at com.cloud.hypervisor.kvm.resource.wrapper.
> > > > > > > LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:75)
> > > > > > > at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.
> > > > > > > executeRequest(LibvirtComputingResource.java:1369)
> > > > > > > at com.cloud.agent.Agent.processRequest(Agent.java:525)
> > > > > > > at com.cloud.agent.Agent$AgentRequestHandler.doTask(
> > > Agent.java:833)
> > > > > > > at com.cloud.utils.nio.Task.call(Task.java:83)
> > > > > > > at com.cloud.utils.nio.Task.call(Task.java:29)
> > > > > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > > > > > at java.util.concurrent.ThreadPoolExecutor.runWorker(
> > > > > > > ThreadPoolExecutor.java:1149)
> > > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > > > > > > ThreadPoolExecutor.java:624)
> > > > > > > at java.lang.Thread.run(Thread.java:748)
> > > > > > > 2017-11-11 15:40:41,327 WARN  [resource.wrapper.
> > > > > > > LibvirtSecurityGroupRulesCommandWrapper]
> > > > (agentRequest-Handler-2:null)
> > > > > > > (logid:eab9a328) Failed to program network rules for vm
> i-2-7-VM
> > > > > > >
> > > > > > > So, no rules are actually created. Script doesn't call... I
> > suppose
> > > > may
> > > > > > be
> > > > > > > quotes are required because shell interprets ';' as command
> > > > separator.
> > > > > I
> > > > > > > suppose that optimization introduced in 4.10, because in 4.9
> SGs
> > > work
> > > > > > like
> > > > > > > a charm...
> > > > > > >
> > > > > > >
> > > > > > > 2017-11-11 3:15 GMT+07:00 Paul Angus <paul.angus@shapeblue.com
> >:
> > > > > > >
> > > > > > >> Ivan,
> > > > > > >>
> > > > > > >> Can you paste a larger section of unfiltered logs.  There
> would
> > > > always
> > > > > > be
> > > > > > >> a message explaining why the mgmt. server thought that a VR
> > should
> > > > be
> > > > > > shut
> > > > > > >> down
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >> Kind regards,
> > > > > > >>
> > > > > > >> Paul Angus
> > > > > > >>
> > > > > > >> paul.angus@shapeblue.com
> > > > > > >> www.shapeblue.com
> > > > > > >> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > > > > >> @shapeblue
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >> -----Original Message-----
> > > > > > >> From: Simon Weller [mailto:sweller@ena.com.INVALID]
> > > > > > >> Sent: 10 November 2017 18:39
> > > > > > >> To: dev@cloudstack.apache.org
> > > > > > >> Subject: Re: Apache CloudStack 4.10 VR/BasicZone/KVM Problem
> > > > > > >>
> > > > > > >> What VR template image are you using?
> > > > > > >>
> > > > > > >>
> > > > > > >> ________________________________
> > > > > > >> From: Ivan Kudryavtsev <ku...@bw-sw.com>
> > > > > > >> Sent: Friday, November 10, 2017 11:59 AM
> > > > > > >> To: dev@cloudstack.apache.org
> > > > > > >> Subject: Re: Apache CloudStack 4.10 VR/BasicZone/KVM Problem
> > > > > > >>
> > > > > > >> Hi. No, regular NFS. VR starts great, but stopped by ms, other
> > > > system
> > > > > > vms
> > > > > > >> are working. I even added to communication script on compute
> > node
> > > > > "sleep
> > > > > > >> 3600" before ssh, so response to management is delayed, I
> logged
> > > so
> > > > to
> > > > > > VR,
> > > > > > >> all interfaces are up, iptables rules are OK.
> > > > > > >>
> > > > > > >> So agent rolls vr good, but stops it by management order with
> no
> > > > > obvious
> > > > > > >> reason.
> > > > > > >>
> > > > > > >> 11 нояб. 2017 г. 0:54 пользователь "Simon Weller"
> > > > > > <sweller@ena.com.invalid
> > > > > > >> >
> > > > > > >> написал:
> > > > > > >>
> > > > > > >> > Is the storage ceph?
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > ________________________________
> > > > > > >> > From: Ivan Kudryavtsev <ku...@bw-sw.com>
> > > > > > >> > Sent: Friday, November 10, 2017 11:52 AM
> > > > > > >> > To: dev@cloudstack.apache.org
> > > > > > >> > Subject: Re: Apache CloudStack 4.10 VR/BasicZone/KVM Problem
> > > > > > >> >
> > > > > > >> > Hi, I did, and it does the things right, I even added "tee"
> to
> > > ssh
> > > > > > >> > 3922 communication script to out vr response to additional
> log
> > > and
> > > > > it
> > > > > > >> > only receives VR version line and sends all info (the same
> > from
> > > > > > >> > pastebin) to ACS and receives "stop" order.
> > > > > > >> >
> > > > > > >> > I'll try to provide additional info, but ad you can see,
> > > > management
> > > > > > >> > receives proper response and sends stop next op. It looks
> very
> > > > > freaky
> > > > > > >> > without any notification...
> > > > > > >> >
> > > > > > >> > 11 нояб. 2017 г. 0:37 пользователь "Simon Weller"
> > > > > > >> > <sweller@ena.com.invalid
> > > > > > >> > >
> > > > > > >> > написал:
> > > > > > >> >
> > > > > > >> > > Ivan,
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > > Can you put the host agents into debug mode? Hopefully
> that
> > > will
> > > > > > >> > > provide more information.
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > > https://cwiki.apache.org/confluence/display/CLOUDSTACK/
> > > > > > KVM+agent+deb
> > > > > > >> > > ug
> > > > > > >> KVM agent debug - Apache Cloudstack - Apache Software ...<
> > > > > > >> https://cwiki.apache.org/confluence/display/CLOUDSTACK/
> > > > > KVM+agent+debug>
> > > > > > >> cwiki.apache.org
> > > > > > >> Steps to debug the KVM agent from eclipse: In KVM agent edit
> > > > > > >> '/usr/libexec/agent-runner ', add "-Xrunjdwp:transport=dt_
> > > > > > socket,address=8787
> > > > > > >> ...
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > > - Si
> > > > > > >> > >
> > > > > > >> > > ________________________________
> > > > > > >> > > From: Ivan Kudryavtsev <ku...@bw-sw.com>
> > > > > > >> > > Sent: Friday, November 10, 2017 11:34 AM
> > > > > > >> > > To: dev@cloudstack.apache.org
> > > > > > >> > > Subject: Apache CloudStack 4.10 VR/BasicZone/KVM Problem
> > > > > > >> > >
> > > > > > >> > > Hello, Devs.
> > > > > > >> > >
> > > > > > >> > > I experience VR Start Problem in the fresh ACS 4.10
> > deployment
> > > > > > >> > >
> > > > > > >> > > Intersting place of logs is here:
> > > https://pastebin.com/iBXRBA5N
> > > > > > >> [https://pastebin.com/i/facebook.png]<https://
> > > pastebin.com/iBXRBA5N
> > > > >
> > > > > > >>
> > > > > > >> 2017-11-10 23:05:35,853 DEBUG [c.c.a.t.Request]
> > > > > > >> (Work-Job-Executor-15:ctx-6fdf61 - Pastebin.com<
> > https://pastebin.
> > > > > > >> com/iBXRBA5N>
> > > > > > >> pastebin.com
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >> > >
> > > > > > >> > > Basically, the situation looks like:
> > > > > > >> > >
> > > > > > >> > > 1. Management Server tries to launch VR 2. It gets from
> > Agent
> > > > > proper
> > > > > > >> > > VR response with VR details 3. It sends StopCommand
> without
> > > > > > >> > > explanation.
> > > > > > >> > >
> > > > > > >> > > I'm trying to figure out what happens inside, but the
> > codebase
> > > > is
> > > > > > >> > > huge
> > > > > > >> > and
> > > > > > >> > > still no positive results. Please, let me know if you have
> > any
> > > > > ideas
> > > > > > >> > which
> > > > > > >> > > could help me finding the reason. Thanks a lot.
> > > > > > >> > >
> > > > > > >> > > --
> > > > > > >> > > With best regards, Ivan Kudryavtsev
> > > > > > >> > > Bitworks Software, Ltd.
> > > > > > >> > > Cell: +7-923-414-1515
> > > > > > >> > > WWW: http://bitworks.software/ <http://bw-sw.com/>
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > > >>
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > With best regards, Ivan Kudryavtsev
> > > > > > > Bitworks Software, Ltd.
> > > > > > > Cell: +7-923-414-1515
> > > > > > > WWW: http://bitworks.software/ <http://bw-sw.com/>
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > With best regards, Ivan Kudryavtsev
> > > > > > Bitworks Software, Ltd.
> > > > > > Cell: +7-923-414-1515
> > > > > > WWW: http://bitworks.software/ <http://bw-sw.com/>
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > With best regards, Ivan Kudryavtsev
> > > > Bitworks Software, Ltd.
> > > > Cell: +7-923-414-1515
> > > > WWW: http://bitworks.software/ <http://bw-sw.com/>
> > > >
> > >
> >
>

Re: Apache CloudStack 4.10 VR/BasicZone/KVM Problem

[Ivan Kudryavtsev <ku...@bw-sw.com>]

Ozhan, I suggest 4.10 only if you need KVM VM snapshots, virtio-scsi and
ipv6. Don't see other reasons to go from 4.9 (at least for kvm). But I
waited for 4.10 for a long time...

13 нояб. 2017 г. 16:21 пользователь "Özhan Rüzgar Karaman" <
oruzgarkaraman@gmail.com> написал:

> ok we will try to compile 4.10 from source and try it on our test
> environment.
>
> We currently use 4.9.x on production, i make this test for deciding if 4.10
> is suitable & stable for us, we also use very simple and generic setup for
> this test no ipv6 just simple environment . So for production it looks like
> we need to wait some time for upcoming releases, what do you think about
> that?
>
> There was a thread about this question one month ago and i remember that
> most of people still stick to 4.9 release for their production
> environments...
>
> Thanks
> Özhan
>
> On Mon, Nov 13, 2017 at 12:18 PM, Wei ZHOU <us...@gmail.com> wrote:
>
> > Hi Ivan,
> >
> > I would suggest you to create jira tickets for each problem you found in
> > your testing, and create a github pull request for a jira ticket.
> > It is convenient for reviewers.
> >
> > Kind regards,
> > Wei
> >
> > 2017-11-13 10:01 GMT+01:00 Ivan Kudryavtsev <ku...@bw-sw.com>:
> >
> > > Hello, Ozhan
> > >
> > > https://github.com/apache/cloudstack/pull/2320
> > >
> > > fixes everything I found right now. It enables functioning of
> everything
> > > correctly even if no IPv6 CIDR specified for network (at least for
> Ubuntu
> > > 14.04).
> > > For IPv6 configuration instruction please take a look at:
> > > https://github.com/apache/cloudstack/commit/
> > f10c8bfe0c99a762c2606459413a47
> > > 219614e775
> > > (oh my god,I spend several hours trying to find how to configure IPv6
> for
> > > 4.10).
> > >
> > > Please, don't forget to recreate SSVM because there is a fix for
> > templates
> > > too:
> > > https://github.com/apache/cloudstack/pull/2322
> > >
> > >
> > > 2017-11-13 15:51 GMT+07:00 Özhan Rüzgar Karaman <
> > oruzgarkaraman@gmail.com
> > > >:
> > >
> > > > Hi Ivan;
> > > > Does this hotfixes also solve qoutes and shell script interprets
> > problem?
> > > > We have no ipv6 setup and today we made similar test with fresh
> install
> > > > 4.10. We noticed that we receive similar error on security groups
> stage
> > > > while br_netfilter module is already active on our environment. We
> made
> > > > same tests for Ubuntu 16.04.3 and 14.04.5 kvm hosts
> > > >
> > > > Logs are below:
> > > > 2017-11-13 11:47:41,773 DEBUG [kvm.resource.
> LibvirtComputingResource]
> > > > (agentRequest-Handler-1:null) Executing:
> > > > /usr/share/cloudstack-common/scripts/vm/network/security_group.py
> > > > add_network_rules --vmname i-2-5-VM --vmid 5 --vmip 192.168.18.6
> > --vmip6
> > > > null --sig 74a6d8c403af9c3c7b89ecf206e4ac26 --seq 16 --vmmac
> > > > 1e:00:9b:00:00:05 --vif vnet8 --brname breth0-23 --nicsecips 0:
> --rules
> > > > I:tcp:1:65535:
> > > > 0.0.0.0/0,NEXT;I:udp:1:65535:0.0.0.0/0,NEXT;E:tcp:1:65535:
> > 0.0.0.0/0,NEXT
> > > ;
> > > > 2017-11-13 11:47:41,773 WARN  [kvm.resource.
> LibvirtComputingResource]
> > > > (agentRequest-Handler-1:null) Exception:
> > > > /usr/share/cloudstack-common/scripts/vm/network/security_group.py
> > > > add_network_rules --vmname i-2-5-VM --vmid 5 --vmip 192.168.18.6
> > --vmip6
> > > > null --sig 74a6d8c403af9c3c7b89ecf206e4ac26 --seq 16 --vmmac
> > > > 1e:00:9b:00:00:05 --vif vnet8 --brname breth0-23 --nicsecips 0:
> --rules
> > > > I:tcp:1:65535:
> > > > 0.0.0.0/0,NEXT;I:udp:1:65535:0.0.0.0/0,NEXT;E:tcp:1:65535:
> > 0.0.0.0/0,NEXT
> > > ;
> > > > java.lang.NullPointerException
> > > > at java.lang.ProcessBuilder.start(ProcessBuilder.java:1012)
> > > > at com.cloud.utils.script.Script.execute(Script.java:214)
> > > > at com.cloud.utils.script.Script.execute(Script.java:182)
> > > > at
> > > > com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.
> > > > addNetworkRules(LibvirtComputingResource.java:3429)
> > > > at
> > > > com.cloud.hypervisor.kvm.resource.wrapper.
> > LibvirtSecurityGroupRulesComma
> > > > ndWrapper.execute(LibvirtSecurityGroupRulesCommandWrapper.java:57)
> > > > at
> > > > com.cloud.hypervisor.kvm.resource.wrapper.
> > LibvirtSecurityGroupRulesComma
> > > > ndWrapper.execute(LibvirtSecurityGroupRulesCommandWrapper.java:36)
> > > > at
> > > > com.cloud.hypervisor.kvm.resource.wrapper.
> > LibvirtRequestWrapper.execute(
> > > > LibvirtRequestWrapper.java:75)
> > > > at
> > > > com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.
> > > executeRequest(
> > > > LibvirtComputingResource.java:1369)
> > > > at com.cloud.agent.Agent.processRequest(Agent.java:525)
> > > > at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:833)
> > > > at com.cloud.utils.nio.Task.call(Task.java:83)
> > > > at com.cloud.utils.nio.Task.call(Task.java:29)
> > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > > at
> > > > java.util.concurrent.ThreadPoolExecutor.runWorker(
> > > > ThreadPoolExecutor.java:1149)
> > > > at
> > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > > > ThreadPoolExecutor.java:624)
> > > > at java.lang.Thread.run(Thread.java:748)
> > > > 2017-11-13 11:47:41,774 WARN
> > > > [resource.wrapper.LibvirtSecurityGroupRulesCommandWrapper]
> > > > (agentRequest-Handler-1:null) Failed to program network rules for vm
> > > > i-2-5-VM
> > > > 2017-11-13 11:47:41,775 DEBUG [cloud.agent.Agent]
> > > > (agentRequest-Handler-1:null) Seq 1-6412562919422165093:  { Ans: ,
> > > MgmtId:
> > > > 345048635880, via: 1, Ver: v1, Flags: 110,
> > > > [{"com.cloud.agent.api.SecurityGroupRuleAnswer":{"
> > > > logSequenceNumber":16,"vmId":5,"reason":"PROGRAMMING_
> > > > FAILED","result":false,"details":"programming
> > > > network rules failed","wait":0}}] }
> > > >
> > > >
> > > > When we execute command with double quotas for rules section from
> > command
> > > > line it executes without a problem like below:
> > > > root@kvmt3:/var/log/cloudstack/agent#
> > > > /usr/share/cloudstack-common/scripts/vm/network/security_group.py
> > > > add_network_rules --vmname i-2-5-VM --vmid 5 --vmip 192.168.18.6
> > --vmip6
> > > > null --sig 74a6d8c403af9c3c7b89ecf206e4ac26 --seq 16 --vmmac
> > > > 1e:00:9b:00:00:05 --vif vnet8 --brname breth0-23 --nicsecips 0:
> --rules
> > > > "I:tcp:1:65535:
> > > > 0.0.0.0/0,NEXT;I:udp:1:65535:0.0.0.0/0,NEXT;E:tcp:1:65535:
> > 0.0.0.0/0,NEXT
> > > ;"
> > > > root@kvmt3:/var/log/cloudstack/agent# echo $?
> > > > 0
> > > > root@kvmt3:/var/log/cloudstack/agent#
> > > >
> > > > Thanks
> > > > Özhan
> > > >
> > > >
> > > > On Sat, Nov 11, 2017 at 6:59 PM, Ivan Kudryavtsev <
> > > > kudryavtsev_ia@bw-sw.com>
> > > > wrote:
> > > >
> > > > > Hello, I implemented some hotfixes for 4.10 to work
> > > > >
> > > > > https://github.com/apache/cloudstack/pull/2319 - to master (load
> > > > > br_netfilter module)
> > > > > https://github.com/apache/cloudstack/pull/2320 - to 4.10 which
> fixes
> > > SG
> > > > > failures related to ipv6.
> > > > >
> > > > >
> > > > > 2017-11-11 15:51 GMT+07:00 Ivan Kudryavtsev <
> > kudryavtsev_ia@bw-sw.com
> > > >:
> > > > >
> > > > > > Following up with previous question. I managed to make it work by
> > > > > removing
> > > > > > all and heading to ubuntu 14.04 hypervisor host.
> > > > > >
> > > > > > Also, what I found more:
> > > > > >
> > > > > > 1. when setup databases (management server) if custom port is
> > > > specified,
> > > > > > databases themself is not created. If create manually, import
> > scripts
> > > > > work
> > > > > > fine.
> > > > > > 2. UI: unable to download ISO to __all__ zones. Have to specify
> > > certain
> > > > > > zone, else UI gives an error.
> > > > > > 3. Ubuntu doesn't load module *br_netfilter* but
> > > > > >
> > > > > > /usr/share/cloudstack-common/scripts/vm/network/security_
> group.py
> > > > > >
> > > > > > uses it and nothing good as a result:
> > > > > >
> > > > > > 2017-11-11 15:38:29,241 - sysctl -w net.bridge.bridge-nf-call-
> > > > > arptables=1
> > > > > > 2017-11-11 15:38:29,244 - sysctl -w net.bridge.bridge-nf-call-
> > > > iptables=1
> > > > > > 2017-11-11 15:38:29,247 - sysctl -w net.bridge.bridge-nf-call-
> > > > > ip6tables=1
> > > > > >
> > > > > > adding br_netfilter to /etc/modules fixes it. I suppose it's
> > > necessary
> > > > to
> > > > > > add
> > > > > >
> > > > > > in script something like: modprobe br_netfilter (or smarter
> thing).
> > > > > >
> > > > > > But It doesn't work completely, actually, security groups are
> > > > > unfunctional:
> > > > > >
> > > > > > ==> /var/log/cloudstack/agent/agent.log <==
> > > > > > 2017-11-11 15:40:41,326 WARN  [kvm.resource.
> > > LibvirtComputingResource]
> > > > > > (agentRequest-Handler-2:null) (logid:eab9a328) Exception:
> > > > > > /usr/share/cloudstack-common/scripts/vm/network/security_
> group.py
> > > > > > add_network_rules --vmname i-2-7-VM --vmid 7 --vmip 176.120.28.4
> > > > --vmip6
> > > > > > null --sig d60255deb618b7be9f477eed10d65234 --seq 4 --vmmac
> > > > > > 1e:00:6f:00:01:01 --vif vnet8 --brname cloudbr0 --nicsecips 0:
> > > --rules
> > > > > > I:icmp:-1:-1:0.0.0.0/0,NEXT;I:tcp:1:65535:0.0.0.0/0,NEXT;I:
> > > > > > udp:1:65535:0.0.0.0/0,NEXT;E:icmp:-1:-1:0.0.0.0/0,NEXT;E:
> > > > > > tcp:1:65535:0.0.0.0/0,NEXT;E:udp:1:65535:0.0.0.0/0,NEXT;
> > > > > > java.lang.NullPointerException
> > > > > > at java.lang.ProcessBuilder.start(ProcessBuilder.java:1012)
> > > > > > at com.cloud.utils.script.Script.execute(Script.java:214)
> > > > > > at com.cloud.utils.script.Script.execute(Script.java:182)
> > > > > > at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.
> > > > > > addNetworkRules(LibvirtComputingResource.java:3429)
> > > > > > at com.cloud.hypervisor.kvm.resource.wrapper.
> > > > > > LibvirtSecurityGroupRulesCommandWrapper.execute(
> > > > > > LibvirtSecurityGroupRulesCommandWrapper.java:57)
> > > > > > at com.cloud.hypervisor.kvm.resource.wrapper.
> > > > > > LibvirtSecurityGroupRulesCommandWrapper.execute(
> > > > > > LibvirtSecurityGroupRulesCommandWrapper.java:36)
> > > > > > at com.cloud.hypervisor.kvm.resource.wrapper.
> > > > > > LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:75)
> > > > > > at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.
> > > > > > executeRequest(LibvirtComputingResource.java:1369)
> > > > > > at com.cloud.agent.Agent.processRequest(Agent.java:525)
> > > > > > at com.cloud.agent.Agent$AgentRequestHandler.doTask(
> > Agent.java:833)
> > > > > > at com.cloud.utils.nio.Task.call(Task.java:83)
> > > > > > at com.cloud.utils.nio.Task.call(Task.java:29)
> > > > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.runWorker(
> > > > > > ThreadPoolExecutor.java:1149)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > > > > > ThreadPoolExecutor.java:624)
> > > > > > at java.lang.Thread.run(Thread.java:748)
> > > > > > 2017-11-11 15:40:41,327 WARN  [resource.wrapper.
> > > > > > LibvirtSecurityGroupRulesCommandWrapper]
> > > (agentRequest-Handler-2:null)
> > > > > > (logid:eab9a328) Failed to program network rules for vm i-2-7-VM
> > > > > >
> > > > > > So, no rules are actually created. Script doesn't call... I
> suppose
> > > may
> > > > > be
> > > > > > quotes are required because shell interprets ';' as command
> > > separator.
> > > > I
> > > > > > suppose that optimization introduced in 4.10, because in 4.9 SGs
> > work
> > > > > like
> > > > > > a charm...
> > > > > >
> > > > > >
> > > > > > 2017-11-11 3:15 GMT+07:00 Paul Angus <pa...@shapeblue.com>:
> > > > > >
> > > > > >> Ivan,
> > > > > >>
> > > > > >> Can you paste a larger section of unfiltered logs.  There would
> > > always
> > > > > be
> > > > > >> a message explaining why the mgmt. server thought that a VR
> should
> > > be
> > > > > shut
> > > > > >> down
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> Kind regards,
> > > > > >>
> > > > > >> Paul Angus
> > > > > >>
> > > > > >> paul.angus@shapeblue.com
> > > > > >> www.shapeblue.com
> > > > > >> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > > > >> @shapeblue
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> -----Original Message-----
> > > > > >> From: Simon Weller [mailto:sweller@ena.com.INVALID]
> > > > > >> Sent: 10 November 2017 18:39
> > > > > >> To: dev@cloudstack.apache.org
> > > > > >> Subject: Re: Apache CloudStack 4.10 VR/BasicZone/KVM Problem
> > > > > >>
> > > > > >> What VR template image are you using?
> > > > > >>
> > > > > >>
> > > > > >> ________________________________
> > > > > >> From: Ivan Kudryavtsev <ku...@bw-sw.com>
> > > > > >> Sent: Friday, November 10, 2017 11:59 AM
> > > > > >> To: dev@cloudstack.apache.org
> > > > > >> Subject: Re: Apache CloudStack 4.10 VR/BasicZone/KVM Problem
> > > > > >>
> > > > > >> Hi. No, regular NFS. VR starts great, but stopped by ms, other
> > > system
> > > > > vms
> > > > > >> are working. I even added to communication script on compute
> node
> > > > "sleep
> > > > > >> 3600" before ssh, so response to management is delayed, I logged
> > so
> > > to
> > > > > VR,
> > > > > >> all interfaces are up, iptables rules are OK.
> > > > > >>
> > > > > >> So agent rolls vr good, but stops it by management order with no
> > > > obvious
> > > > > >> reason.
> > > > > >>
> > > > > >> 11 нояб. 2017 г. 0:54 пользователь "Simon Weller"
> > > > > <sweller@ena.com.invalid
> > > > > >> >
> > > > > >> написал:
> > > > > >>
> > > > > >> > Is the storage ceph?
> > > > > >> >
> > > > > >> >
> > > > > >> > ________________________________
> > > > > >> > From: Ivan Kudryavtsev <ku...@bw-sw.com>
> > > > > >> > Sent: Friday, November 10, 2017 11:52 AM
> > > > > >> > To: dev@cloudstack.apache.org
> > > > > >> > Subject: Re: Apache CloudStack 4.10 VR/BasicZone/KVM Problem
> > > > > >> >
> > > > > >> > Hi, I did, and it does the things right, I even added "tee" to
> > ssh
> > > > > >> > 3922 communication script to out vr response to additional log
> > and
> > > > it
> > > > > >> > only receives VR version line and sends all info (the same
> from
> > > > > >> > pastebin) to ACS and receives "stop" order.
> > > > > >> >
> > > > > >> > I'll try to provide additional info, but ad you can see,
> > > management
> > > > > >> > receives proper response and sends stop next op. It looks very
> > > > freaky
> > > > > >> > without any notification...
> > > > > >> >
> > > > > >> > 11 нояб. 2017 г. 0:37 пользователь "Simon Weller"
> > > > > >> > <sweller@ena.com.invalid
> > > > > >> > >
> > > > > >> > написал:
> > > > > >> >
> > > > > >> > > Ivan,
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > Can you put the host agents into debug mode? Hopefully that
> > will
> > > > > >> > > provide more information.
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > https://cwiki.apache.org/confluence/display/CLOUDSTACK/
> > > > > KVM+agent+deb
> > > > > >> > > ug
> > > > > >> KVM agent debug - Apache Cloudstack - Apache Software ...<
> > > > > >> https://cwiki.apache.org/confluence/display/CLOUDSTACK/
> > > > KVM+agent+debug>
> > > > > >> cwiki.apache.org
> > > > > >> Steps to debug the KVM agent from eclipse: In KVM agent edit
> > > > > >> '/usr/libexec/agent-runner ', add "-Xrunjdwp:transport=dt_
> > > > > socket,address=8787
> > > > > >> ...
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > - Si
> > > > > >> > >
> > > > > >> > > ________________________________
> > > > > >> > > From: Ivan Kudryavtsev <ku...@bw-sw.com>
> > > > > >> > > Sent: Friday, November 10, 2017 11:34 AM
> > > > > >> > > To: dev@cloudstack.apache.org
> > > > > >> > > Subject: Apache CloudStack 4.10 VR/BasicZone/KVM Problem
> > > > > >> > >
> > > > > >> > > Hello, Devs.
> > > > > >> > >
> > > > > >> > > I experience VR Start Problem in the fresh ACS 4.10
> deployment
> > > > > >> > >
> > > > > >> > > Intersting place of logs is here:
> > https://pastebin.com/iBXRBA5N
> > > > > >> [https://pastebin.com/i/facebook.png]<https://
> > pastebin.com/iBXRBA5N
> > > >
> > > > > >>
> > > > > >> 2017-11-10 23:05:35,853 DEBUG [c.c.a.t.Request]
> > > > > >> (Work-Job-Executor-15:ctx-6fdf61 - Pastebin.com<
> https://pastebin.
> > > > > >> com/iBXRBA5N>
> > > > > >> pastebin.com
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> > >
> > > > > >> > > Basically, the situation looks like:
> > > > > >> > >
> > > > > >> > > 1. Management Server tries to launch VR 2. It gets from
> Agent
> > > > proper
> > > > > >> > > VR response with VR details 3. It sends StopCommand without
> > > > > >> > > explanation.
> > > > > >> > >
> > > > > >> > > I'm trying to figure out what happens inside, but the
> codebase
> > > is
> > > > > >> > > huge
> > > > > >> > and
> > > > > >> > > still no positive results. Please, let me know if you have
> any
> > > > ideas
> > > > > >> > which
> > > > > >> > > could help me finding the reason. Thanks a lot.
> > > > > >> > >
> > > > > >> > > --
> > > > > >> > > With best regards, Ivan Kudryavtsev
> > > > > >> > > Bitworks Software, Ltd.
> > > > > >> > > Cell: +7-923-414-1515
> > > > > >> > > WWW: http://bitworks.software/ <http://bw-sw.com/>
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > > >>
> > > > > >
> > > > > >
> > > > > > --
> > > > > > With best regards, Ivan Kudryavtsev
> > > > > > Bitworks Software, Ltd.
> > > > > > Cell: +7-923-414-1515
> > > > > > WWW: http://bitworks.software/ <http://bw-sw.com/>
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > With best regards, Ivan Kudryavtsev
> > > > > Bitworks Software, Ltd.
> > > > > Cell: +7-923-414-1515
> > > > > WWW: http://bitworks.software/ <http://bw-sw.com/>
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > With best regards, Ivan Kudryavtsev
> > > Bitworks Software, Ltd.
> > > Cell: +7-923-414-1515
> > > WWW: http://bitworks.software/ <http://bw-sw.com/>
> > >
> >
>

Re: Apache CloudStack 4.10 VR/BasicZone/KVM Problem

[Özhan Rüzgar Karaman <or...@gmail.com>]

ok we will try to compile 4.10 from source and try it on our test
environment.

We currently use 4.9.x on production, i make this test for deciding if 4.10
is suitable & stable for us, we also use very simple and generic setup for
this test no ipv6 just simple environment . So for production it looks like
we need to wait some time for upcoming releases, what do you think about
that?

There was a thread about this question one month ago and i remember that
most of people still stick to 4.9 release for their production
environments...

Thanks
Özhan

On Mon, Nov 13, 2017 at 12:18 PM, Wei ZHOU <us...@gmail.com> wrote:

> Hi Ivan,
>
> I would suggest you to create jira tickets for each problem you found in
> your testing, and create a github pull request for a jira ticket.
> It is convenient for reviewers.
>
> Kind regards,
> Wei
>
> 2017-11-13 10:01 GMT+01:00 Ivan Kudryavtsev <ku...@bw-sw.com>:
>
> > Hello, Ozhan
> >
> > https://github.com/apache/cloudstack/pull/2320
> >
> > fixes everything I found right now. It enables functioning of everything
> > correctly even if no IPv6 CIDR specified for network (at least for Ubuntu
> > 14.04).
> > For IPv6 configuration instruction please take a look at:
> > https://github.com/apache/cloudstack/commit/
> f10c8bfe0c99a762c2606459413a47
> > 219614e775
> > (oh my god,I spend several hours trying to find how to configure IPv6 for
> > 4.10).
> >
> > Please, don't forget to recreate SSVM because there is a fix for
> templates
> > too:
> > https://github.com/apache/cloudstack/pull/2322
> >
> >
> > 2017-11-13 15:51 GMT+07:00 Özhan Rüzgar Karaman <
> oruzgarkaraman@gmail.com
> > >:
> >
> > > Hi Ivan;
> > > Does this hotfixes also solve qoutes and shell script interprets
> problem?
> > > We have no ipv6 setup and today we made similar test with fresh install
> > > 4.10. We noticed that we receive similar error on security groups stage
> > > while br_netfilter module is already active on our environment. We made
> > > same tests for Ubuntu 16.04.3 and 14.04.5 kvm hosts
> > >
> > > Logs are below:
> > > 2017-11-13 11:47:41,773 DEBUG [kvm.resource.LibvirtComputingResource]
> > > (agentRequest-Handler-1:null) Executing:
> > > /usr/share/cloudstack-common/scripts/vm/network/security_group.py
> > > add_network_rules --vmname i-2-5-VM --vmid 5 --vmip 192.168.18.6
> --vmip6
> > > null --sig 74a6d8c403af9c3c7b89ecf206e4ac26 --seq 16 --vmmac
> > > 1e:00:9b:00:00:05 --vif vnet8 --brname breth0-23 --nicsecips 0: --rules
> > > I:tcp:1:65535:
> > > 0.0.0.0/0,NEXT;I:udp:1:65535:0.0.0.0/0,NEXT;E:tcp:1:65535:
> 0.0.0.0/0,NEXT
> > ;
> > > 2017-11-13 11:47:41,773 WARN  [kvm.resource.LibvirtComputingResource]
> > > (agentRequest-Handler-1:null) Exception:
> > > /usr/share/cloudstack-common/scripts/vm/network/security_group.py
> > > add_network_rules --vmname i-2-5-VM --vmid 5 --vmip 192.168.18.6
> --vmip6
> > > null --sig 74a6d8c403af9c3c7b89ecf206e4ac26 --seq 16 --vmmac
> > > 1e:00:9b:00:00:05 --vif vnet8 --brname breth0-23 --nicsecips 0: --rules
> > > I:tcp:1:65535:
> > > 0.0.0.0/0,NEXT;I:udp:1:65535:0.0.0.0/0,NEXT;E:tcp:1:65535:
> 0.0.0.0/0,NEXT
> > ;
> > > java.lang.NullPointerException
> > > at java.lang.ProcessBuilder.start(ProcessBuilder.java:1012)
> > > at com.cloud.utils.script.Script.execute(Script.java:214)
> > > at com.cloud.utils.script.Script.execute(Script.java:182)
> > > at
> > > com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.
> > > addNetworkRules(LibvirtComputingResource.java:3429)
> > > at
> > > com.cloud.hypervisor.kvm.resource.wrapper.
> LibvirtSecurityGroupRulesComma
> > > ndWrapper.execute(LibvirtSecurityGroupRulesCommandWrapper.java:57)
> > > at
> > > com.cloud.hypervisor.kvm.resource.wrapper.
> LibvirtSecurityGroupRulesComma
> > > ndWrapper.execute(LibvirtSecurityGroupRulesCommandWrapper.java:36)
> > > at
> > > com.cloud.hypervisor.kvm.resource.wrapper.
> LibvirtRequestWrapper.execute(
> > > LibvirtRequestWrapper.java:75)
> > > at
> > > com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.
> > executeRequest(
> > > LibvirtComputingResource.java:1369)
> > > at com.cloud.agent.Agent.processRequest(Agent.java:525)
> > > at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:833)
> > > at com.cloud.utils.nio.Task.call(Task.java:83)
> > > at com.cloud.utils.nio.Task.call(Task.java:29)
> > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > at
> > > java.util.concurrent.ThreadPoolExecutor.runWorker(
> > > ThreadPoolExecutor.java:1149)
> > > at
> > > java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > > ThreadPoolExecutor.java:624)
> > > at java.lang.Thread.run(Thread.java:748)
> > > 2017-11-13 11:47:41,774 WARN
> > > [resource.wrapper.LibvirtSecurityGroupRulesCommandWrapper]
> > > (agentRequest-Handler-1:null) Failed to program network rules for vm
> > > i-2-5-VM
> > > 2017-11-13 11:47:41,775 DEBUG [cloud.agent.Agent]
> > > (agentRequest-Handler-1:null) Seq 1-6412562919422165093:  { Ans: ,
> > MgmtId:
> > > 345048635880, via: 1, Ver: v1, Flags: 110,
> > > [{"com.cloud.agent.api.SecurityGroupRuleAnswer":{"
> > > logSequenceNumber":16,"vmId":5,"reason":"PROGRAMMING_
> > > FAILED","result":false,"details":"programming
> > > network rules failed","wait":0}}] }
> > >
> > >
> > > When we execute command with double quotas for rules section from
> command
> > > line it executes without a problem like below:
> > > root@kvmt3:/var/log/cloudstack/agent#
> > > /usr/share/cloudstack-common/scripts/vm/network/security_group.py
> > > add_network_rules --vmname i-2-5-VM --vmid 5 --vmip 192.168.18.6
> --vmip6
> > > null --sig 74a6d8c403af9c3c7b89ecf206e4ac26 --seq 16 --vmmac
> > > 1e:00:9b:00:00:05 --vif vnet8 --brname breth0-23 --nicsecips 0: --rules
> > > "I:tcp:1:65535:
> > > 0.0.0.0/0,NEXT;I:udp:1:65535:0.0.0.0/0,NEXT;E:tcp:1:65535:
> 0.0.0.0/0,NEXT
> > ;"
> > > root@kvmt3:/var/log/cloudstack/agent# echo $?
> > > 0
> > > root@kvmt3:/var/log/cloudstack/agent#
> > >
> > > Thanks
> > > Özhan
> > >
> > >
> > > On Sat, Nov 11, 2017 at 6:59 PM, Ivan Kudryavtsev <
> > > kudryavtsev_ia@bw-sw.com>
> > > wrote:
> > >
> > > > Hello, I implemented some hotfixes for 4.10 to work
> > > >
> > > > https://github.com/apache/cloudstack/pull/2319 - to master (load
> > > > br_netfilter module)
> > > > https://github.com/apache/cloudstack/pull/2320 - to 4.10 which fixes
> > SG
> > > > failures related to ipv6.
> > > >
> > > >
> > > > 2017-11-11 15:51 GMT+07:00 Ivan Kudryavtsev <
> kudryavtsev_ia@bw-sw.com
> > >:
> > > >
> > > > > Following up with previous question. I managed to make it work by
> > > > removing
> > > > > all and heading to ubuntu 14.04 hypervisor host.
> > > > >
> > > > > Also, what I found more:
> > > > >
> > > > > 1. when setup databases (management server) if custom port is
> > > specified,
> > > > > databases themself is not created. If create manually, import
> scripts
> > > > work
> > > > > fine.
> > > > > 2. UI: unable to download ISO to __all__ zones. Have to specify
> > certain
> > > > > zone, else UI gives an error.
> > > > > 3. Ubuntu doesn't load module *br_netfilter* but
> > > > >
> > > > > /usr/share/cloudstack-common/scripts/vm/network/security_group.py
> > > > >
> > > > > uses it and nothing good as a result:
> > > > >
> > > > > 2017-11-11 15:38:29,241 - sysctl -w net.bridge.bridge-nf-call-
> > > > arptables=1
> > > > > 2017-11-11 15:38:29,244 - sysctl -w net.bridge.bridge-nf-call-
> > > iptables=1
> > > > > 2017-11-11 15:38:29,247 - sysctl -w net.bridge.bridge-nf-call-
> > > > ip6tables=1
> > > > >
> > > > > adding br_netfilter to /etc/modules fixes it. I suppose it's
> > necessary
> > > to
> > > > > add
> > > > >
> > > > > in script something like: modprobe br_netfilter (or smarter thing).
> > > > >
> > > > > But It doesn't work completely, actually, security groups are
> > > > unfunctional:
> > > > >
> > > > > ==> /var/log/cloudstack/agent/agent.log <==
> > > > > 2017-11-11 15:40:41,326 WARN  [kvm.resource.
> > LibvirtComputingResource]
> > > > > (agentRequest-Handler-2:null) (logid:eab9a328) Exception:
> > > > > /usr/share/cloudstack-common/scripts/vm/network/security_group.py
> > > > > add_network_rules --vmname i-2-7-VM --vmid 7 --vmip 176.120.28.4
> > > --vmip6
> > > > > null --sig d60255deb618b7be9f477eed10d65234 --seq 4 --vmmac
> > > > > 1e:00:6f:00:01:01 --vif vnet8 --brname cloudbr0 --nicsecips 0:
> > --rules
> > > > > I:icmp:-1:-1:0.0.0.0/0,NEXT;I:tcp:1:65535:0.0.0.0/0,NEXT;I:
> > > > > udp:1:65535:0.0.0.0/0,NEXT;E:icmp:-1:-1:0.0.0.0/0,NEXT;E:
> > > > > tcp:1:65535:0.0.0.0/0,NEXT;E:udp:1:65535:0.0.0.0/0,NEXT;
> > > > > java.lang.NullPointerException
> > > > > at java.lang.ProcessBuilder.start(ProcessBuilder.java:1012)
> > > > > at com.cloud.utils.script.Script.execute(Script.java:214)
> > > > > at com.cloud.utils.script.Script.execute(Script.java:182)
> > > > > at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.
> > > > > addNetworkRules(LibvirtComputingResource.java:3429)
> > > > > at com.cloud.hypervisor.kvm.resource.wrapper.
> > > > > LibvirtSecurityGroupRulesCommandWrapper.execute(
> > > > > LibvirtSecurityGroupRulesCommandWrapper.java:57)
> > > > > at com.cloud.hypervisor.kvm.resource.wrapper.
> > > > > LibvirtSecurityGroupRulesCommandWrapper.execute(
> > > > > LibvirtSecurityGroupRulesCommandWrapper.java:36)
> > > > > at com.cloud.hypervisor.kvm.resource.wrapper.
> > > > > LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:75)
> > > > > at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.
> > > > > executeRequest(LibvirtComputingResource.java:1369)
> > > > > at com.cloud.agent.Agent.processRequest(Agent.java:525)
> > > > > at com.cloud.agent.Agent$AgentRequestHandler.doTask(
> Agent.java:833)
> > > > > at com.cloud.utils.nio.Task.call(Task.java:83)
> > > > > at com.cloud.utils.nio.Task.call(Task.java:29)
> > > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > > > at java.util.concurrent.ThreadPoolExecutor.runWorker(
> > > > > ThreadPoolExecutor.java:1149)
> > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > > > > ThreadPoolExecutor.java:624)
> > > > > at java.lang.Thread.run(Thread.java:748)
> > > > > 2017-11-11 15:40:41,327 WARN  [resource.wrapper.
> > > > > LibvirtSecurityGroupRulesCommandWrapper]
> > (agentRequest-Handler-2:null)
> > > > > (logid:eab9a328) Failed to program network rules for vm i-2-7-VM
> > > > >
> > > > > So, no rules are actually created. Script doesn't call... I suppose
> > may
> > > > be
> > > > > quotes are required because shell interprets ';' as command
> > separator.
> > > I
> > > > > suppose that optimization introduced in 4.10, because in 4.9 SGs
> work
> > > > like
> > > > > a charm...
> > > > >
> > > > >
> > > > > 2017-11-11 3:15 GMT+07:00 Paul Angus <pa...@shapeblue.com>:
> > > > >
> > > > >> Ivan,
> > > > >>
> > > > >> Can you paste a larger section of unfiltered logs.  There would
> > always
> > > > be
> > > > >> a message explaining why the mgmt. server thought that a VR should
> > be
> > > > shut
> > > > >> down
> > > > >>
> > > > >>
> > > > >>
> > > > >> Kind regards,
> > > > >>
> > > > >> Paul Angus
> > > > >>
> > > > >> paul.angus@shapeblue.com
> > > > >> www.shapeblue.com
> > > > >> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > > >> @shapeblue
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >> -----Original Message-----
> > > > >> From: Simon Weller [mailto:sweller@ena.com.INVALID]
> > > > >> Sent: 10 November 2017 18:39
> > > > >> To: dev@cloudstack.apache.org
> > > > >> Subject: Re: Apache CloudStack 4.10 VR/BasicZone/KVM Problem
> > > > >>
> > > > >> What VR template image are you using?
> > > > >>
> > > > >>
> > > > >> ________________________________
> > > > >> From: Ivan Kudryavtsev <ku...@bw-sw.com>
> > > > >> Sent: Friday, November 10, 2017 11:59 AM
> > > > >> To: dev@cloudstack.apache.org
> > > > >> Subject: Re: Apache CloudStack 4.10 VR/BasicZone/KVM Problem
> > > > >>
> > > > >> Hi. No, regular NFS. VR starts great, but stopped by ms, other
> > system
> > > > vms
> > > > >> are working. I even added to communication script on compute node
> > > "sleep
> > > > >> 3600" before ssh, so response to management is delayed, I logged
> so
> > to
> > > > VR,
> > > > >> all interfaces are up, iptables rules are OK.
> > > > >>
> > > > >> So agent rolls vr good, but stops it by management order with no
> > > obvious
> > > > >> reason.
> > > > >>
> > > > >> 11 нояб. 2017 г. 0:54 пользователь "Simon Weller"
> > > > <sweller@ena.com.invalid
> > > > >> >
> > > > >> написал:
> > > > >>
> > > > >> > Is the storage ceph?
> > > > >> >
> > > > >> >
> > > > >> > ________________________________
> > > > >> > From: Ivan Kudryavtsev <ku...@bw-sw.com>
> > > > >> > Sent: Friday, November 10, 2017 11:52 AM
> > > > >> > To: dev@cloudstack.apache.org
> > > > >> > Subject: Re: Apache CloudStack 4.10 VR/BasicZone/KVM Problem
> > > > >> >
> > > > >> > Hi, I did, and it does the things right, I even added "tee" to
> ssh
> > > > >> > 3922 communication script to out vr response to additional log
> and
> > > it
> > > > >> > only receives VR version line and sends all info (the same from
> > > > >> > pastebin) to ACS and receives "stop" order.
> > > > >> >
> > > > >> > I'll try to provide additional info, but ad you can see,
> > management
> > > > >> > receives proper response and sends stop next op. It looks very
> > > freaky
> > > > >> > without any notification...
> > > > >> >
> > > > >> > 11 нояб. 2017 г. 0:37 пользователь "Simon Weller"
> > > > >> > <sweller@ena.com.invalid
> > > > >> > >
> > > > >> > написал:
> > > > >> >
> > > > >> > > Ivan,
> > > > >> > >
> > > > >> > >
> > > > >> > > Can you put the host agents into debug mode? Hopefully that
> will
> > > > >> > > provide more information.
> > > > >> > >
> > > > >> > >
> > > > >> > > https://cwiki.apache.org/confluence/display/CLOUDSTACK/
> > > > KVM+agent+deb
> > > > >> > > ug
> > > > >> KVM agent debug - Apache Cloudstack - Apache Software ...<
> > > > >> https://cwiki.apache.org/confluence/display/CLOUDSTACK/
> > > KVM+agent+debug>
> > > > >> cwiki.apache.org
> > > > >> Steps to debug the KVM agent from eclipse: In KVM agent edit
> > > > >> '/usr/libexec/agent-runner ', add "-Xrunjdwp:transport=dt_
> > > > socket,address=8787
> > > > >> ...
> > > > >>
> > > > >>
> > > > >>
> > > > >> > >
> > > > >> > >
> > > > >> > > - Si
> > > > >> > >
> > > > >> > > ________________________________
> > > > >> > > From: Ivan Kudryavtsev <ku...@bw-sw.com>
> > > > >> > > Sent: Friday, November 10, 2017 11:34 AM
> > > > >> > > To: dev@cloudstack.apache.org
> > > > >> > > Subject: Apache CloudStack 4.10 VR/BasicZone/KVM Problem
> > > > >> > >
> > > > >> > > Hello, Devs.
> > > > >> > >
> > > > >> > > I experience VR Start Problem in the fresh ACS 4.10 deployment
> > > > >> > >
> > > > >> > > Intersting place of logs is here:
> https://pastebin.com/iBXRBA5N
> > > > >> [https://pastebin.com/i/facebook.png]<https://
> pastebin.com/iBXRBA5N
> > >
> > > > >>
> > > > >> 2017-11-10 23:05:35,853 DEBUG [c.c.a.t.Request]
> > > > >> (Work-Job-Executor-15:ctx-6fdf61 - Pastebin.com<https://pastebin.
> > > > >> com/iBXRBA5N>
> > > > >> pastebin.com
> > > > >>
> > > > >>
> > > > >>
> > > > >> > >
> > > > >> > > Basically, the situation looks like:
> > > > >> > >
> > > > >> > > 1. Management Server tries to launch VR 2. It gets from Agent
> > > proper
> > > > >> > > VR response with VR details 3. It sends StopCommand without
> > > > >> > > explanation.
> > > > >> > >
> > > > >> > > I'm trying to figure out what happens inside, but the codebase
> > is
> > > > >> > > huge
> > > > >> > and
> > > > >> > > still no positive results. Please, let me know if you have any
> > > ideas
> > > > >> > which
> > > > >> > > could help me finding the reason. Thanks a lot.
> > > > >> > >
> > > > >> > > --
> > > > >> > > With best regards, Ivan Kudryavtsev
> > > > >> > > Bitworks Software, Ltd.
> > > > >> > > Cell: +7-923-414-1515
> > > > >> > > WWW: http://bitworks.software/ <http://bw-sw.com/>
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >>
> > > > >
> > > > >
> > > > > --
> > > > > With best regards, Ivan Kudryavtsev
> > > > > Bitworks Software, Ltd.
> > > > > Cell: +7-923-414-1515
> > > > > WWW: http://bitworks.software/ <http://bw-sw.com/>
> > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > With best regards, Ivan Kudryavtsev
> > > > Bitworks Software, Ltd.
> > > > Cell: +7-923-414-1515
> > > > WWW: http://bitworks.software/ <http://bw-sw.com/>
> > > >
> > >
> >
> >
> >
> > --
> > With best regards, Ivan Kudryavtsev
> > Bitworks Software, Ltd.
> > Cell: +7-923-414-1515
> > WWW: http://bitworks.software/ <http://bw-sw.com/>
> >
>

Re: Apache CloudStack 4.10 VR/BasicZone/KVM Problem

[Ivan Kudryavtsev <ku...@bw-sw.com>]

Hi, I did separate Jira issues and PRs to master, but everything in one
related to to 4.10, because 4.10 is already out and I don't want to do
duplicate PRs to different branches.

13 нояб. 2017 г. 16:18 пользователь "Wei ZHOU" <us...@gmail.com>
написал:

> Hi Ivan,
>
> I would suggest you to create jira tickets for each problem you found in
> your testing, and create a github pull request for a jira ticket.
> It is convenient for reviewers.
>
> Kind regards,
> Wei
>
> 2017-11-13 10:01 GMT+01:00 Ivan Kudryavtsev <ku...@bw-sw.com>:
>
> > Hello, Ozhan
> >
> > https://github.com/apache/cloudstack/pull/2320
> >
> > fixes everything I found right now. It enables functioning of everything
> > correctly even if no IPv6 CIDR specified for network (at least for Ubuntu
> > 14.04).
> > For IPv6 configuration instruction please take a look at:
> > https://github.com/apache/cloudstack/commit/
> f10c8bfe0c99a762c2606459413a47
> > 219614e775
> > (oh my god,I spend several hours trying to find how to configure IPv6 for
> > 4.10).
> >
> > Please, don't forget to recreate SSVM because there is a fix for
> templates
> > too:
> > https://github.com/apache/cloudstack/pull/2322
> >
> >
> > 2017-11-13 15:51 GMT+07:00 Özhan Rüzgar Karaman <
> oruzgarkaraman@gmail.com
> > >:
> >
> > > Hi Ivan;
> > > Does this hotfixes also solve qoutes and shell script interprets
> problem?
> > > We have no ipv6 setup and today we made similar test with fresh install
> > > 4.10. We noticed that we receive similar error on security groups stage
> > > while br_netfilter module is already active on our environment. We made
> > > same tests for Ubuntu 16.04.3 and 14.04.5 kvm hosts
> > >
> > > Logs are below:
> > > 2017-11-13 11:47:41,773 DEBUG [kvm.resource.LibvirtComputingResource]
> > > (agentRequest-Handler-1:null) Executing:
> > > /usr/share/cloudstack-common/scripts/vm/network/security_group.py
> > > add_network_rules --vmname i-2-5-VM --vmid 5 --vmip 192.168.18.6
> --vmip6
> > > null --sig 74a6d8c403af9c3c7b89ecf206e4ac26 --seq 16 --vmmac
> > > 1e:00:9b:00:00:05 --vif vnet8 --brname breth0-23 --nicsecips 0: --rules
> > > I:tcp:1:65535:
> > > 0.0.0.0/0,NEXT;I:udp:1:65535:0.0.0.0/0,NEXT;E:tcp:1:65535:
> 0.0.0.0/0,NEXT
> > ;
> > > 2017-11-13 11:47:41,773 WARN  [kvm.resource.LibvirtComputingResource]
> > > (agentRequest-Handler-1:null) Exception:
> > > /usr/share/cloudstack-common/scripts/vm/network/security_group.py
> > > add_network_rules --vmname i-2-5-VM --vmid 5 --vmip 192.168.18.6
> --vmip6
> > > null --sig 74a6d8c403af9c3c7b89ecf206e4ac26 --seq 16 --vmmac
> > > 1e:00:9b:00:00:05 --vif vnet8 --brname breth0-23 --nicsecips 0: --rules
> > > I:tcp:1:65535:
> > > 0.0.0.0/0,NEXT;I:udp:1:65535:0.0.0.0/0,NEXT;E:tcp:1:65535:
> 0.0.0.0/0,NEXT
> > ;
> > > java.lang.NullPointerException
> > > at java.lang.ProcessBuilder.start(ProcessBuilder.java:1012)
> > > at com.cloud.utils.script.Script.execute(Script.java:214)
> > > at com.cloud.utils.script.Script.execute(Script.java:182)
> > > at
> > > com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.
> > > addNetworkRules(LibvirtComputingResource.java:3429)
> > > at
> > > com.cloud.hypervisor.kvm.resource.wrapper.
> LibvirtSecurityGroupRulesComma
> > > ndWrapper.execute(LibvirtSecurityGroupRulesCommandWrapper.java:57)
> > > at
> > > com.cloud.hypervisor.kvm.resource.wrapper.
> LibvirtSecurityGroupRulesComma
> > > ndWrapper.execute(LibvirtSecurityGroupRulesCommandWrapper.java:36)
> > > at
> > > com.cloud.hypervisor.kvm.resource.wrapper.
> LibvirtRequestWrapper.execute(
> > > LibvirtRequestWrapper.java:75)
> > > at
> > > com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.
> > executeRequest(
> > > LibvirtComputingResource.java:1369)
> > > at com.cloud.agent.Agent.processRequest(Agent.java:525)
> > > at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:833)
> > > at com.cloud.utils.nio.Task.call(Task.java:83)
> > > at com.cloud.utils.nio.Task.call(Task.java:29)
> > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > at
> > > java.util.concurrent.ThreadPoolExecutor.runWorker(
> > > ThreadPoolExecutor.java:1149)
> > > at
> > > java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > > ThreadPoolExecutor.java:624)
> > > at java.lang.Thread.run(Thread.java:748)
> > > 2017-11-13 11:47:41,774 WARN
> > > [resource.wrapper.LibvirtSecurityGroupRulesCommandWrapper]
> > > (agentRequest-Handler-1:null) Failed to program network rules for vm
> > > i-2-5-VM
> > > 2017-11-13 11:47:41,775 DEBUG [cloud.agent.Agent]
> > > (agentRequest-Handler-1:null) Seq 1-6412562919422165093:  { Ans: ,
> > MgmtId:
> > > 345048635880, via: 1, Ver: v1, Flags: 110,
> > > [{"com.cloud.agent.api.SecurityGroupRuleAnswer":{"
> > > logSequenceNumber":16,"vmId":5,"reason":"PROGRAMMING_
> > > FAILED","result":false,"details":"programming
> > > network rules failed","wait":0}}] }
> > >
> > >
> > > When we execute command with double quotas for rules section from
> command
> > > line it executes without a problem like below:
> > > root@kvmt3:/var/log/cloudstack/agent#
> > > /usr/share/cloudstack-common/scripts/vm/network/security_group.py
> > > add_network_rules --vmname i-2-5-VM --vmid 5 --vmip 192.168.18.6
> --vmip6
> > > null --sig 74a6d8c403af9c3c7b89ecf206e4ac26 --seq 16 --vmmac
> > > 1e:00:9b:00:00:05 --vif vnet8 --brname breth0-23 --nicsecips 0: --rules
> > > "I:tcp:1:65535:
> > > 0.0.0.0/0,NEXT;I:udp:1:65535:0.0.0.0/0,NEXT;E:tcp:1:65535:
> 0.0.0.0/0,NEXT
> > ;"
> > > root@kvmt3:/var/log/cloudstack/agent# echo $?
> > > 0
> > > root@kvmt3:/var/log/cloudstack/agent#
> > >
> > > Thanks
> > > Özhan
> > >
> > >
> > > On Sat, Nov 11, 2017 at 6:59 PM, Ivan Kudryavtsev <
> > > kudryavtsev_ia@bw-sw.com>
> > > wrote:
> > >
> > > > Hello, I implemented some hotfixes for 4.10 to work
> > > >
> > > > https://github.com/apache/cloudstack/pull/2319 - to master (load
> > > > br_netfilter module)
> > > > https://github.com/apache/cloudstack/pull/2320 - to 4.10 which fixes
> > SG
> > > > failures related to ipv6.
> > > >
> > > >
> > > > 2017-11-11 15:51 GMT+07:00 Ivan Kudryavtsev <
> kudryavtsev_ia@bw-sw.com
> > >:
> > > >
> > > > > Following up with previous question. I managed to make it work by
> > > > removing
> > > > > all and heading to ubuntu 14.04 hypervisor host.
> > > > >
> > > > > Also, what I found more:
> > > > >
> > > > > 1. when setup databases (management server) if custom port is
> > > specified,
> > > > > databases themself is not created. If create manually, import
> scripts
> > > > work
> > > > > fine.
> > > > > 2. UI: unable to download ISO to __all__ zones. Have to specify
> > certain
> > > > > zone, else UI gives an error.
> > > > > 3. Ubuntu doesn't load module *br_netfilter* but
> > > > >
> > > > > /usr/share/cloudstack-common/scripts/vm/network/security_group.py
> > > > >
> > > > > uses it and nothing good as a result:
> > > > >
> > > > > 2017-11-11 15:38:29,241 - sysctl -w net.bridge.bridge-nf-call-
> > > > arptables=1
> > > > > 2017-11-11 15:38:29,244 - sysctl -w net.bridge.bridge-nf-call-
> > > iptables=1
> > > > > 2017-11-11 15:38:29,247 - sysctl -w net.bridge.bridge-nf-call-
> > > > ip6tables=1
> > > > >
> > > > > adding br_netfilter to /etc/modules fixes it. I suppose it's
> > necessary
> > > to
> > > > > add
> > > > >
> > > > > in script something like: modprobe br_netfilter (or smarter thing).
> > > > >
> > > > > But It doesn't work completely, actually, security groups are
> > > > unfunctional:
> > > > >
> > > > > ==> /var/log/cloudstack/agent/agent.log <==
> > > > > 2017-11-11 15:40:41,326 WARN  [kvm.resource.
> > LibvirtComputingResource]
> > > > > (agentRequest-Handler-2:null) (logid:eab9a328) Exception:
> > > > > /usr/share/cloudstack-common/scripts/vm/network/security_group.py
> > > > > add_network_rules --vmname i-2-7-VM --vmid 7 --vmip 176.120.28.4
> > > --vmip6
> > > > > null --sig d60255deb618b7be9f477eed10d65234 --seq 4 --vmmac
> > > > > 1e:00:6f:00:01:01 --vif vnet8 --brname cloudbr0 --nicsecips 0:
> > --rules
> > > > > I:icmp:-1:-1:0.0.0.0/0,NEXT;I:tcp:1:65535:0.0.0.0/0,NEXT;I:
> > > > > udp:1:65535:0.0.0.0/0,NEXT;E:icmp:-1:-1:0.0.0.0/0,NEXT;E:
> > > > > tcp:1:65535:0.0.0.0/0,NEXT;E:udp:1:65535:0.0.0.0/0,NEXT;
> > > > > java.lang.NullPointerException
> > > > > at java.lang.ProcessBuilder.start(ProcessBuilder.java:1012)
> > > > > at com.cloud.utils.script.Script.execute(Script.java:214)
> > > > > at com.cloud.utils.script.Script.execute(Script.java:182)
> > > > > at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.
> > > > > addNetworkRules(LibvirtComputingResource.java:3429)
> > > > > at com.cloud.hypervisor.kvm.resource.wrapper.
> > > > > LibvirtSecurityGroupRulesCommandWrapper.execute(
> > > > > LibvirtSecurityGroupRulesCommandWrapper.java:57)
> > > > > at com.cloud.hypervisor.kvm.resource.wrapper.
> > > > > LibvirtSecurityGroupRulesCommandWrapper.execute(
> > > > > LibvirtSecurityGroupRulesCommandWrapper.java:36)
> > > > > at com.cloud.hypervisor.kvm.resource.wrapper.
> > > > > LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:75)
> > > > > at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.
> > > > > executeRequest(LibvirtComputingResource.java:1369)
> > > > > at com.cloud.agent.Agent.processRequest(Agent.java:525)
> > > > > at com.cloud.agent.Agent$AgentRequestHandler.doTask(
> Agent.java:833)
> > > > > at com.cloud.utils.nio.Task.call(Task.java:83)
> > > > > at com.cloud.utils.nio.Task.call(Task.java:29)
> > > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > > > at java.util.concurrent.ThreadPoolExecutor.runWorker(
> > > > > ThreadPoolExecutor.java:1149)
> > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > > > > ThreadPoolExecutor.java:624)
> > > > > at java.lang.Thread.run(Thread.java:748)
> > > > > 2017-11-11 15:40:41,327 WARN  [resource.wrapper.
> > > > > LibvirtSecurityGroupRulesCommandWrapper]
> > (agentRequest-Handler-2:null)
> > > > > (logid:eab9a328) Failed to program network rules for vm i-2-7-VM
> > > > >
> > > > > So, no rules are actually created. Script doesn't call... I suppose
> > may
> > > > be
> > > > > quotes are required because shell interprets ';' as command
> > separator.
> > > I
> > > > > suppose that optimization introduced in 4.10, because in 4.9 SGs
> work
> > > > like
> > > > > a charm...
> > > > >
> > > > >
> > > > > 2017-11-11 3:15 GMT+07:00 Paul Angus <pa...@shapeblue.com>:
> > > > >
> > > > >> Ivan,
> > > > >>
> > > > >> Can you paste a larger section of unfiltered logs.  There would
> > always
> > > > be
> > > > >> a message explaining why the mgmt. server thought that a VR should
> > be
> > > > shut
> > > > >> down
> > > > >>
> > > > >>
> > > > >>
> > > > >> Kind regards,
> > > > >>
> > > > >> Paul Angus
> > > > >>
> > > > >> paul.angus@shapeblue.com
> > > > >> www.shapeblue.com
> > > > >> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > > >> @shapeblue
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >> -----Original Message-----
> > > > >> From: Simon Weller [mailto:sweller@ena.com.INVALID]
> > > > >> Sent: 10 November 2017 18:39
> > > > >> To: dev@cloudstack.apache.org
> > > > >> Subject: Re: Apache CloudStack 4.10 VR/BasicZone/KVM Problem
> > > > >>
> > > > >> What VR template image are you using?
> > > > >>
> > > > >>
> > > > >> ________________________________
> > > > >> From: Ivan Kudryavtsev <ku...@bw-sw.com>
> > > > >> Sent: Friday, November 10, 2017 11:59 AM
> > > > >> To: dev@cloudstack.apache.org
> > > > >> Subject: Re: Apache CloudStack 4.10 VR/BasicZone/KVM Problem
> > > > >>
> > > > >> Hi. No, regular NFS. VR starts great, but stopped by ms, other
> > system
> > > > vms
> > > > >> are working. I even added to communication script on compute node
> > > "sleep
> > > > >> 3600" before ssh, so response to management is delayed, I logged
> so
> > to
> > > > VR,
> > > > >> all interfaces are up, iptables rules are OK.
> > > > >>
> > > > >> So agent rolls vr good, but stops it by management order with no
> > > obvious
> > > > >> reason.
> > > > >>
> > > > >> 11 нояб. 2017 г. 0:54 пользователь "Simon Weller"
> > > > <sweller@ena.com.invalid
> > > > >> >
> > > > >> написал:
> > > > >>
> > > > >> > Is the storage ceph?
> > > > >> >
> > > > >> >
> > > > >> > ________________________________
> > > > >> > From: Ivan Kudryavtsev <ku...@bw-sw.com>
> > > > >> > Sent: Friday, November 10, 2017 11:52 AM
> > > > >> > To: dev@cloudstack.apache.org
> > > > >> > Subject: Re: Apache CloudStack 4.10 VR/BasicZone/KVM Problem
> > > > >> >
> > > > >> > Hi, I did, and it does the things right, I even added "tee" to
> ssh
> > > > >> > 3922 communication script to out vr response to additional log
> and
> > > it
> > > > >> > only receives VR version line and sends all info (the same from
> > > > >> > pastebin) to ACS and receives "stop" order.
> > > > >> >
> > > > >> > I'll try to provide additional info, but ad you can see,
> > management
> > > > >> > receives proper response and sends stop next op. It looks very
> > > freaky
> > > > >> > without any notification...
> > > > >> >
> > > > >> > 11 нояб. 2017 г. 0:37 пользователь "Simon Weller"
> > > > >> > <sweller@ena.com.invalid
> > > > >> > >
> > > > >> > написал:
> > > > >> >
> > > > >> > > Ivan,
> > > > >> > >
> > > > >> > >
> > > > >> > > Can you put the host agents into debug mode? Hopefully that
> will
> > > > >> > > provide more information.
> > > > >> > >
> > > > >> > >
> > > > >> > > https://cwiki.apache.org/confluence/display/CLOUDSTACK/
> > > > KVM+agent+deb
> > > > >> > > ug
> > > > >> KVM agent debug - Apache Cloudstack - Apache Software ...<
> > > > >> https://cwiki.apache.org/confluence/display/CLOUDSTACK/
> > > KVM+agent+debug>
> > > > >> cwiki.apache.org
> > > > >> Steps to debug the KVM agent from eclipse: In KVM agent edit
> > > > >> '/usr/libexec/agent-runner ', add "-Xrunjdwp:transport=dt_
> > > > socket,address=8787
> > > > >> ...
> > > > >>
> > > > >>
> > > > >>
> > > > >> > >
> > > > >> > >
> > > > >> > > - Si
> > > > >> > >
> > > > >> > > ________________________________
> > > > >> > > From: Ivan Kudryavtsev <ku...@bw-sw.com>
> > > > >> > > Sent: Friday, November 10, 2017 11:34 AM
> > > > >> > > To: dev@cloudstack.apache.org
> > > > >> > > Subject: Apache CloudStack 4.10 VR/BasicZone/KVM Problem
> > > > >> > >
> > > > >> > > Hello, Devs.
> > > > >> > >
> > > > >> > > I experience VR Start Problem in the fresh ACS 4.10 deployment
> > > > >> > >
> > > > >> > > Intersting place of logs is here:
> https://pastebin.com/iBXRBA5N
> > > > >> [https://pastebin.com/i/facebook.png]<https://
> pastebin.com/iBXRBA5N
> > >
> > > > >>
> > > > >> 2017-11-10 23:05:35,853 DEBUG [c.c.a.t.Request]
> > > > >> (Work-Job-Executor-15:ctx-6fdf61 - Pastebin.com<https://pastebin.
> > > > >> com/iBXRBA5N>
> > > > >> pastebin.com
> > > > >>
> > > > >>
> > > > >>
> > > > >> > >
> > > > >> > > Basically, the situation looks like:
> > > > >> > >
> > > > >> > > 1. Management Server tries to launch VR 2. It gets from Agent
> > > proper
> > > > >> > > VR response with VR details 3. It sends StopCommand without
> > > > >> > > explanation.
> > > > >> > >
> > > > >> > > I'm trying to figure out what happens inside, but the codebase
> > is
> > > > >> > > huge
> > > > >> > and
> > > > >> > > still no positive results. Please, let me know if you have any
> > > ideas
> > > > >> > which
> > > > >> > > could help me finding the reason. Thanks a lot.
> > > > >> > >
> > > > >> > > --
> > > > >> > > With best regards, Ivan Kudryavtsev
> > > > >> > > Bitworks Software, Ltd.
> > > > >> > > Cell: +7-923-414-1515
> > > > >> > > WWW: http://bitworks.software/ <http://bw-sw.com/>
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >>
> > > > >
> > > > >
> > > > > --
> > > > > With best regards, Ivan Kudryavtsev
> > > > > Bitworks Software, Ltd.
> > > > > Cell: +7-923-414-1515
> > > > > WWW: http://bitworks.software/ <http://bw-sw.com/>
> > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > With best regards, Ivan Kudryavtsev
> > > > Bitworks Software, Ltd.
> > > > Cell: +7-923-414-1515
> > > > WWW: http://bitworks.software/ <http://bw-sw.com/>
> > > >
> > >
> >
> >
> >
> > --
> > With best regards, Ivan Kudryavtsev
> > Bitworks Software, Ltd.
> > Cell: +7-923-414-1515
> > WWW: http://bitworks.software/ <http://bw-sw.com/>
> >
>

Re: Apache CloudStack 4.10 VR/BasicZone/KVM Problem

[Wei ZHOU <us...@gmail.com>]

Hi Ivan,

I would suggest you to create jira tickets for each problem you found in
your testing, and create a github pull request for a jira ticket.
It is convenient for reviewers.

Kind regards,
Wei

2017-11-13 10:01 GMT+01:00 Ivan Kudryavtsev <ku...@bw-sw.com>:

> Hello, Ozhan
>
> https://github.com/apache/cloudstack/pull/2320
>
> fixes everything I found right now. It enables functioning of everything
> correctly even if no IPv6 CIDR specified for network (at least for Ubuntu
> 14.04).
> For IPv6 configuration instruction please take a look at:
> https://github.com/apache/cloudstack/commit/f10c8bfe0c99a762c2606459413a47
> 219614e775
> (oh my god,I spend several hours trying to find how to configure IPv6 for
> 4.10).
>
> Please, don't forget to recreate SSVM because there is a fix for templates
> too:
> https://github.com/apache/cloudstack/pull/2322
>
>
> 2017-11-13 15:51 GMT+07:00 Özhan Rüzgar Karaman <oruzgarkaraman@gmail.com
> >:
>
> > Hi Ivan;
> > Does this hotfixes also solve qoutes and shell script interprets problem?
> > We have no ipv6 setup and today we made similar test with fresh install
> > 4.10. We noticed that we receive similar error on security groups stage
> > while br_netfilter module is already active on our environment. We made
> > same tests for Ubuntu 16.04.3 and 14.04.5 kvm hosts
> >
> > Logs are below:
> > 2017-11-13 11:47:41,773 DEBUG [kvm.resource.LibvirtComputingResource]
> > (agentRequest-Handler-1:null) Executing:
> > /usr/share/cloudstack-common/scripts/vm/network/security_group.py
> > add_network_rules --vmname i-2-5-VM --vmid 5 --vmip 192.168.18.6 --vmip6
> > null --sig 74a6d8c403af9c3c7b89ecf206e4ac26 --seq 16 --vmmac
> > 1e:00:9b:00:00:05 --vif vnet8 --brname breth0-23 --nicsecips 0: --rules
> > I:tcp:1:65535:
> > 0.0.0.0/0,NEXT;I:udp:1:65535:0.0.0.0/0,NEXT;E:tcp:1:65535:0.0.0.0/0,NEXT
> ;
> > 2017-11-13 11:47:41,773 WARN  [kvm.resource.LibvirtComputingResource]
> > (agentRequest-Handler-1:null) Exception:
> > /usr/share/cloudstack-common/scripts/vm/network/security_group.py
> > add_network_rules --vmname i-2-5-VM --vmid 5 --vmip 192.168.18.6 --vmip6
> > null --sig 74a6d8c403af9c3c7b89ecf206e4ac26 --seq 16 --vmmac
> > 1e:00:9b:00:00:05 --vif vnet8 --brname breth0-23 --nicsecips 0: --rules
> > I:tcp:1:65535:
> > 0.0.0.0/0,NEXT;I:udp:1:65535:0.0.0.0/0,NEXT;E:tcp:1:65535:0.0.0.0/0,NEXT
> ;
> > java.lang.NullPointerException
> > at java.lang.ProcessBuilder.start(ProcessBuilder.java:1012)
> > at com.cloud.utils.script.Script.execute(Script.java:214)
> > at com.cloud.utils.script.Script.execute(Script.java:182)
> > at
> > com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.
> > addNetworkRules(LibvirtComputingResource.java:3429)
> > at
> > com.cloud.hypervisor.kvm.resource.wrapper.LibvirtSecurityGroupRulesComma
> > ndWrapper.execute(LibvirtSecurityGroupRulesCommandWrapper.java:57)
> > at
> > com.cloud.hypervisor.kvm.resource.wrapper.LibvirtSecurityGroupRulesComma
> > ndWrapper.execute(LibvirtSecurityGroupRulesCommandWrapper.java:36)
> > at
> > com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(
> > LibvirtRequestWrapper.java:75)
> > at
> > com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.
> executeRequest(
> > LibvirtComputingResource.java:1369)
> > at com.cloud.agent.Agent.processRequest(Agent.java:525)
> > at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:833)
> > at com.cloud.utils.nio.Task.call(Task.java:83)
> > at com.cloud.utils.nio.Task.call(Task.java:29)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(
> > ThreadPoolExecutor.java:1149)
> > at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > ThreadPoolExecutor.java:624)
> > at java.lang.Thread.run(Thread.java:748)
> > 2017-11-13 11:47:41,774 WARN
> > [resource.wrapper.LibvirtSecurityGroupRulesCommandWrapper]
> > (agentRequest-Handler-1:null) Failed to program network rules for vm
> > i-2-5-VM
> > 2017-11-13 11:47:41,775 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-1:null) Seq 1-6412562919422165093:  { Ans: ,
> MgmtId:
> > 345048635880, via: 1, Ver: v1, Flags: 110,
> > [{"com.cloud.agent.api.SecurityGroupRuleAnswer":{"
> > logSequenceNumber":16,"vmId":5,"reason":"PROGRAMMING_
> > FAILED","result":false,"details":"programming
> > network rules failed","wait":0}}] }
> >
> >
> > When we execute command with double quotas for rules section from command
> > line it executes without a problem like below:
> > root@kvmt3:/var/log/cloudstack/agent#
> > /usr/share/cloudstack-common/scripts/vm/network/security_group.py
> > add_network_rules --vmname i-2-5-VM --vmid 5 --vmip 192.168.18.6 --vmip6
> > null --sig 74a6d8c403af9c3c7b89ecf206e4ac26 --seq 16 --vmmac
> > 1e:00:9b:00:00:05 --vif vnet8 --brname breth0-23 --nicsecips 0: --rules
> > "I:tcp:1:65535:
> > 0.0.0.0/0,NEXT;I:udp:1:65535:0.0.0.0/0,NEXT;E:tcp:1:65535:0.0.0.0/0,NEXT
> ;"
> > root@kvmt3:/var/log/cloudstack/agent# echo $?
> > 0
> > root@kvmt3:/var/log/cloudstack/agent#
> >
> > Thanks
> > Özhan
> >
> >
> > On Sat, Nov 11, 2017 at 6:59 PM, Ivan Kudryavtsev <
> > kudryavtsev_ia@bw-sw.com>
> > wrote:
> >
> > > Hello, I implemented some hotfixes for 4.10 to work
> > >
> > > https://github.com/apache/cloudstack/pull/2319 - to master (load
> > > br_netfilter module)
> > > https://github.com/apache/cloudstack/pull/2320 - to 4.10 which fixes
> SG
> > > failures related to ipv6.
> > >
> > >
> > > 2017-11-11 15:51 GMT+07:00 Ivan Kudryavtsev <kudryavtsev_ia@bw-sw.com
> >:
> > >
> > > > Following up with previous question. I managed to make it work by
> > > removing
> > > > all and heading to ubuntu 14.04 hypervisor host.
> > > >
> > > > Also, what I found more:
> > > >
> > > > 1. when setup databases (management server) if custom port is
> > specified,
> > > > databases themself is not created. If create manually, import scripts
> > > work
> > > > fine.
> > > > 2. UI: unable to download ISO to __all__ zones. Have to specify
> certain
> > > > zone, else UI gives an error.
> > > > 3. Ubuntu doesn't load module *br_netfilter* but
> > > >
> > > > /usr/share/cloudstack-common/scripts/vm/network/security_group.py
> > > >
> > > > uses it and nothing good as a result:
> > > >
> > > > 2017-11-11 15:38:29,241 - sysctl -w net.bridge.bridge-nf-call-
> > > arptables=1
> > > > 2017-11-11 15:38:29,244 - sysctl -w net.bridge.bridge-nf-call-
> > iptables=1
> > > > 2017-11-11 15:38:29,247 - sysctl -w net.bridge.bridge-nf-call-
> > > ip6tables=1
> > > >
> > > > adding br_netfilter to /etc/modules fixes it. I suppose it's
> necessary
> > to
> > > > add
> > > >
> > > > in script something like: modprobe br_netfilter (or smarter thing).
> > > >
> > > > But It doesn't work completely, actually, security groups are
> > > unfunctional:
> > > >
> > > > ==> /var/log/cloudstack/agent/agent.log <==
> > > > 2017-11-11 15:40:41,326 WARN  [kvm.resource.
> LibvirtComputingResource]
> > > > (agentRequest-Handler-2:null) (logid:eab9a328) Exception:
> > > > /usr/share/cloudstack-common/scripts/vm/network/security_group.py
> > > > add_network_rules --vmname i-2-7-VM --vmid 7 --vmip 176.120.28.4
> > --vmip6
> > > > null --sig d60255deb618b7be9f477eed10d65234 --seq 4 --vmmac
> > > > 1e:00:6f:00:01:01 --vif vnet8 --brname cloudbr0 --nicsecips 0:
> --rules
> > > > I:icmp:-1:-1:0.0.0.0/0,NEXT;I:tcp:1:65535:0.0.0.0/0,NEXT;I:
> > > > udp:1:65535:0.0.0.0/0,NEXT;E:icmp:-1:-1:0.0.0.0/0,NEXT;E:
> > > > tcp:1:65535:0.0.0.0/0,NEXT;E:udp:1:65535:0.0.0.0/0,NEXT;
> > > > java.lang.NullPointerException
> > > > at java.lang.ProcessBuilder.start(ProcessBuilder.java:1012)
> > > > at com.cloud.utils.script.Script.execute(Script.java:214)
> > > > at com.cloud.utils.script.Script.execute(Script.java:182)
> > > > at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.
> > > > addNetworkRules(LibvirtComputingResource.java:3429)
> > > > at com.cloud.hypervisor.kvm.resource.wrapper.
> > > > LibvirtSecurityGroupRulesCommandWrapper.execute(
> > > > LibvirtSecurityGroupRulesCommandWrapper.java:57)
> > > > at com.cloud.hypervisor.kvm.resource.wrapper.
> > > > LibvirtSecurityGroupRulesCommandWrapper.execute(
> > > > LibvirtSecurityGroupRulesCommandWrapper.java:36)
> > > > at com.cloud.hypervisor.kvm.resource.wrapper.
> > > > LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:75)
> > > > at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.
> > > > executeRequest(LibvirtComputingResource.java:1369)
> > > > at com.cloud.agent.Agent.processRequest(Agent.java:525)
> > > > at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:833)
> > > > at com.cloud.utils.nio.Task.call(Task.java:83)
> > > > at com.cloud.utils.nio.Task.call(Task.java:29)
> > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > > at java.util.concurrent.ThreadPoolExecutor.runWorker(
> > > > ThreadPoolExecutor.java:1149)
> > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > > > ThreadPoolExecutor.java:624)
> > > > at java.lang.Thread.run(Thread.java:748)
> > > > 2017-11-11 15:40:41,327 WARN  [resource.wrapper.
> > > > LibvirtSecurityGroupRulesCommandWrapper]
> (agentRequest-Handler-2:null)
> > > > (logid:eab9a328) Failed to program network rules for vm i-2-7-VM
> > > >
> > > > So, no rules are actually created. Script doesn't call... I suppose
> may
> > > be
> > > > quotes are required because shell interprets ';' as command
> separator.
> > I
> > > > suppose that optimization introduced in 4.10, because in 4.9 SGs work
> > > like
> > > > a charm...
> > > >
> > > >
> > > > 2017-11-11 3:15 GMT+07:00 Paul Angus <pa...@shapeblue.com>:
> > > >
> > > >> Ivan,
> > > >>
> > > >> Can you paste a larger section of unfiltered logs.  There would
> always
> > > be
> > > >> a message explaining why the mgmt. server thought that a VR should
> be
> > > shut
> > > >> down
> > > >>
> > > >>
> > > >>
> > > >> Kind regards,
> > > >>
> > > >> Paul Angus
> > > >>
> > > >> paul.angus@shapeblue.com
> > > >> www.shapeblue.com
> > > >> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > >> @shapeblue
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> -----Original Message-----
> > > >> From: Simon Weller [mailto:sweller@ena.com.INVALID]
> > > >> Sent: 10 November 2017 18:39
> > > >> To: dev@cloudstack.apache.org
> > > >> Subject: Re: Apache CloudStack 4.10 VR/BasicZone/KVM Problem
> > > >>
> > > >> What VR template image are you using?
> > > >>
> > > >>
> > > >> ________________________________
> > > >> From: Ivan Kudryavtsev <ku...@bw-sw.com>
> > > >> Sent: Friday, November 10, 2017 11:59 AM
> > > >> To: dev@cloudstack.apache.org
> > > >> Subject: Re: Apache CloudStack 4.10 VR/BasicZone/KVM Problem
> > > >>
> > > >> Hi. No, regular NFS. VR starts great, but stopped by ms, other
> system
> > > vms
> > > >> are working. I even added to communication script on compute node
> > "sleep
> > > >> 3600" before ssh, so response to management is delayed, I logged so
> to
> > > VR,
> > > >> all interfaces are up, iptables rules are OK.
> > > >>
> > > >> So agent rolls vr good, but stops it by management order with no
> > obvious
> > > >> reason.
> > > >>
> > > >> 11 нояб. 2017 г. 0:54 пользователь "Simon Weller"
> > > <sweller@ena.com.invalid
> > > >> >
> > > >> написал:
> > > >>
> > > >> > Is the storage ceph?
> > > >> >
> > > >> >
> > > >> > ________________________________
> > > >> > From: Ivan Kudryavtsev <ku...@bw-sw.com>
> > > >> > Sent: Friday, November 10, 2017 11:52 AM
> > > >> > To: dev@cloudstack.apache.org
> > > >> > Subject: Re: Apache CloudStack 4.10 VR/BasicZone/KVM Problem
> > > >> >
> > > >> > Hi, I did, and it does the things right, I even added "tee" to ssh
> > > >> > 3922 communication script to out vr response to additional log and
> > it
> > > >> > only receives VR version line and sends all info (the same from
> > > >> > pastebin) to ACS and receives "stop" order.
> > > >> >
> > > >> > I'll try to provide additional info, but ad you can see,
> management
> > > >> > receives proper response and sends stop next op. It looks very
> > freaky
> > > >> > without any notification...
> > > >> >
> > > >> > 11 нояб. 2017 г. 0:37 пользователь "Simon Weller"
> > > >> > <sweller@ena.com.invalid
> > > >> > >
> > > >> > написал:
> > > >> >
> > > >> > > Ivan,
> > > >> > >
> > > >> > >
> > > >> > > Can you put the host agents into debug mode? Hopefully that will
> > > >> > > provide more information.
> > > >> > >
> > > >> > >
> > > >> > > https://cwiki.apache.org/confluence/display/CLOUDSTACK/
> > > KVM+agent+deb
> > > >> > > ug
> > > >> KVM agent debug - Apache Cloudstack - Apache Software ...<
> > > >> https://cwiki.apache.org/confluence/display/CLOUDSTACK/
> > KVM+agent+debug>
> > > >> cwiki.apache.org
> > > >> Steps to debug the KVM agent from eclipse: In KVM agent edit
> > > >> '/usr/libexec/agent-runner ', add "-Xrunjdwp:transport=dt_
> > > socket,address=8787
> > > >> ...
> > > >>
> > > >>
> > > >>
> > > >> > >
> > > >> > >
> > > >> > > - Si
> > > >> > >
> > > >> > > ________________________________
> > > >> > > From: Ivan Kudryavtsev <ku...@bw-sw.com>
> > > >> > > Sent: Friday, November 10, 2017 11:34 AM
> > > >> > > To: dev@cloudstack.apache.org
> > > >> > > Subject: Apache CloudStack 4.10 VR/BasicZone/KVM Problem
> > > >> > >
> > > >> > > Hello, Devs.
> > > >> > >
> > > >> > > I experience VR Start Problem in the fresh ACS 4.10 deployment
> > > >> > >
> > > >> > > Intersting place of logs is here: https://pastebin.com/iBXRBA5N
> > > >> [https://pastebin.com/i/facebook.png]<https://pastebin.com/iBXRBA5N
> >
> > > >>
> > > >> 2017-11-10 23:05:35,853 DEBUG [c.c.a.t.Request]
> > > >> (Work-Job-Executor-15:ctx-6fdf61 - Pastebin.com<https://pastebin.
> > > >> com/iBXRBA5N>
> > > >> pastebin.com
> > > >>
> > > >>
> > > >>
> > > >> > >
> > > >> > > Basically, the situation looks like:
> > > >> > >
> > > >> > > 1. Management Server tries to launch VR 2. It gets from Agent
> > proper
> > > >> > > VR response with VR details 3. It sends StopCommand without
> > > >> > > explanation.
> > > >> > >
> > > >> > > I'm trying to figure out what happens inside, but the codebase
> is
> > > >> > > huge
> > > >> > and
> > > >> > > still no positive results. Please, let me know if you have any
> > ideas
> > > >> > which
> > > >> > > could help me finding the reason. Thanks a lot.
> > > >> > >
> > > >> > > --
> > > >> > > With best regards, Ivan Kudryavtsev
> > > >> > > Bitworks Software, Ltd.
> > > >> > > Cell: +7-923-414-1515
> > > >> > > WWW: http://bitworks.software/ <http://bw-sw.com/>
> > > >> > >
> > > >> >
> > > >>
> > > >>
> > > >
> > > >
> > > > --
> > > > With best regards, Ivan Kudryavtsev
> > > > Bitworks Software, Ltd.
> > > > Cell: +7-923-414-1515
> > > > WWW: http://bitworks.software/ <http://bw-sw.com/>
> > > >
> > > >
> > >
> > >
> > > --
> > > With best regards, Ivan Kudryavtsev
> > > Bitworks Software, Ltd.
> > > Cell: +7-923-414-1515
> > > WWW: http://bitworks.software/ <http://bw-sw.com/>
> > >
> >
>
>
>
> --
> With best regards, Ivan Kudryavtsev
> Bitworks Software, Ltd.
> Cell: +7-923-414-1515
> WWW: http://bitworks.software/ <http://bw-sw.com/>
>

Re: Apache CloudStack 4.10 VR/BasicZone/KVM Problem

[Ivan Kudryavtsev <ku...@bw-sw.com>]

Hello, Ozhan

https://github.com/apache/cloudstack/pull/2320

fixes everything I found right now. It enables functioning of everything
correctly even if no IPv6 CIDR specified for network (at least for Ubuntu
14.04).
For IPv6 configuration instruction please take a look at:
https://github.com/apache/cloudstack/commit/f10c8bfe0c99a762c2606459413a47219614e775
(oh my god,I spend several hours trying to find how to configure IPv6 for
4.10).

Please, don't forget to recreate SSVM because there is a fix for templates
too:
https://github.com/apache/cloudstack/pull/2322


2017-11-13 15:51 GMT+07:00 Özhan Rüzgar Karaman <or...@gmail.com>:

> Hi Ivan;
> Does this hotfixes also solve qoutes and shell script interprets problem?
> We have no ipv6 setup and today we made similar test with fresh install
> 4.10. We noticed that we receive similar error on security groups stage
> while br_netfilter module is already active on our environment. We made
> same tests for Ubuntu 16.04.3 and 14.04.5 kvm hosts
>
> Logs are below:
> 2017-11-13 11:47:41,773 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-1:null) Executing:
> /usr/share/cloudstack-common/scripts/vm/network/security_group.py
> add_network_rules --vmname i-2-5-VM --vmid 5 --vmip 192.168.18.6 --vmip6
> null --sig 74a6d8c403af9c3c7b89ecf206e4ac26 --seq 16 --vmmac
> 1e:00:9b:00:00:05 --vif vnet8 --brname breth0-23 --nicsecips 0: --rules
> I:tcp:1:65535:
> 0.0.0.0/0,NEXT;I:udp:1:65535:0.0.0.0/0,NEXT;E:tcp:1:65535:0.0.0.0/0,NEXT;
> 2017-11-13 11:47:41,773 WARN  [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-1:null) Exception:
> /usr/share/cloudstack-common/scripts/vm/network/security_group.py
> add_network_rules --vmname i-2-5-VM --vmid 5 --vmip 192.168.18.6 --vmip6
> null --sig 74a6d8c403af9c3c7b89ecf206e4ac26 --seq 16 --vmmac
> 1e:00:9b:00:00:05 --vif vnet8 --brname breth0-23 --nicsecips 0: --rules
> I:tcp:1:65535:
> 0.0.0.0/0,NEXT;I:udp:1:65535:0.0.0.0/0,NEXT;E:tcp:1:65535:0.0.0.0/0,NEXT;
> java.lang.NullPointerException
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:1012)
> at com.cloud.utils.script.Script.execute(Script.java:214)
> at com.cloud.utils.script.Script.execute(Script.java:182)
> at
> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.
> addNetworkRules(LibvirtComputingResource.java:3429)
> at
> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtSecurityGroupRulesComma
> ndWrapper.execute(LibvirtSecurityGroupRulesCommandWrapper.java:57)
> at
> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtSecurityGroupRulesComma
> ndWrapper.execute(LibvirtSecurityGroupRulesCommandWrapper.java:36)
> at
> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(
> LibvirtRequestWrapper.java:75)
> at
> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(
> LibvirtComputingResource.java:1369)
> at com.cloud.agent.Agent.processRequest(Agent.java:525)
> at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:833)
> at com.cloud.utils.nio.Task.call(Task.java:83)
> at com.cloud.utils.nio.Task.call(Task.java:29)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2017-11-13 11:47:41,774 WARN
> [resource.wrapper.LibvirtSecurityGroupRulesCommandWrapper]
> (agentRequest-Handler-1:null) Failed to program network rules for vm
> i-2-5-VM
> 2017-11-13 11:47:41,775 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-1:null) Seq 1-6412562919422165093:  { Ans: , MgmtId:
> 345048635880, via: 1, Ver: v1, Flags: 110,
> [{"com.cloud.agent.api.SecurityGroupRuleAnswer":{"
> logSequenceNumber":16,"vmId":5,"reason":"PROGRAMMING_
> FAILED","result":false,"details":"programming
> network rules failed","wait":0}}] }
>
>
> When we execute command with double quotas for rules section from command
> line it executes without a problem like below:
> root@kvmt3:/var/log/cloudstack/agent#
> /usr/share/cloudstack-common/scripts/vm/network/security_group.py
> add_network_rules --vmname i-2-5-VM --vmid 5 --vmip 192.168.18.6 --vmip6
> null --sig 74a6d8c403af9c3c7b89ecf206e4ac26 --seq 16 --vmmac
> 1e:00:9b:00:00:05 --vif vnet8 --brname breth0-23 --nicsecips 0: --rules
> "I:tcp:1:65535:
> 0.0.0.0/0,NEXT;I:udp:1:65535:0.0.0.0/0,NEXT;E:tcp:1:65535:0.0.0.0/0,NEXT;"
> root@kvmt3:/var/log/cloudstack/agent# echo $?
> 0
> root@kvmt3:/var/log/cloudstack/agent#
>
> Thanks
> Özhan
>
>
> On Sat, Nov 11, 2017 at 6:59 PM, Ivan Kudryavtsev <
> kudryavtsev_ia@bw-sw.com>
> wrote:
>
> > Hello, I implemented some hotfixes for 4.10 to work
> >
> > https://github.com/apache/cloudstack/pull/2319 - to master (load
> > br_netfilter module)
> > https://github.com/apache/cloudstack/pull/2320 - to 4.10 which fixes SG
> > failures related to ipv6.
> >
> >
> > 2017-11-11 15:51 GMT+07:00 Ivan Kudryavtsev <ku...@bw-sw.com>:
> >
> > > Following up with previous question. I managed to make it work by
> > removing
> > > all and heading to ubuntu 14.04 hypervisor host.
> > >
> > > Also, what I found more:
> > >
> > > 1. when setup databases (management server) if custom port is
> specified,
> > > databases themself is not created. If create manually, import scripts
> > work
> > > fine.
> > > 2. UI: unable to download ISO to __all__ zones. Have to specify certain
> > > zone, else UI gives an error.
> > > 3. Ubuntu doesn't load module *br_netfilter* but
> > >
> > > /usr/share/cloudstack-common/scripts/vm/network/security_group.py
> > >
> > > uses it and nothing good as a result:
> > >
> > > 2017-11-11 15:38:29,241 - sysctl -w net.bridge.bridge-nf-call-
> > arptables=1
> > > 2017-11-11 15:38:29,244 - sysctl -w net.bridge.bridge-nf-call-
> iptables=1
> > > 2017-11-11 15:38:29,247 - sysctl -w net.bridge.bridge-nf-call-
> > ip6tables=1
> > >
> > > adding br_netfilter to /etc/modules fixes it. I suppose it's necessary
> to
> > > add
> > >
> > > in script something like: modprobe br_netfilter (or smarter thing).
> > >
> > > But It doesn't work completely, actually, security groups are
> > unfunctional:
> > >
> > > ==> /var/log/cloudstack/agent/agent.log <==
> > > 2017-11-11 15:40:41,326 WARN  [kvm.resource.LibvirtComputingResource]
> > > (agentRequest-Handler-2:null) (logid:eab9a328) Exception:
> > > /usr/share/cloudstack-common/scripts/vm/network/security_group.py
> > > add_network_rules --vmname i-2-7-VM --vmid 7 --vmip 176.120.28.4
> --vmip6
> > > null --sig d60255deb618b7be9f477eed10d65234 --seq 4 --vmmac
> > > 1e:00:6f:00:01:01 --vif vnet8 --brname cloudbr0 --nicsecips 0: --rules
> > > I:icmp:-1:-1:0.0.0.0/0,NEXT;I:tcp:1:65535:0.0.0.0/0,NEXT;I:
> > > udp:1:65535:0.0.0.0/0,NEXT;E:icmp:-1:-1:0.0.0.0/0,NEXT;E:
> > > tcp:1:65535:0.0.0.0/0,NEXT;E:udp:1:65535:0.0.0.0/0,NEXT;
> > > java.lang.NullPointerException
> > > at java.lang.ProcessBuilder.start(ProcessBuilder.java:1012)
> > > at com.cloud.utils.script.Script.execute(Script.java:214)
> > > at com.cloud.utils.script.Script.execute(Script.java:182)
> > > at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.
> > > addNetworkRules(LibvirtComputingResource.java:3429)
> > > at com.cloud.hypervisor.kvm.resource.wrapper.
> > > LibvirtSecurityGroupRulesCommandWrapper.execute(
> > > LibvirtSecurityGroupRulesCommandWrapper.java:57)
> > > at com.cloud.hypervisor.kvm.resource.wrapper.
> > > LibvirtSecurityGroupRulesCommandWrapper.execute(
> > > LibvirtSecurityGroupRulesCommandWrapper.java:36)
> > > at com.cloud.hypervisor.kvm.resource.wrapper.
> > > LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:75)
> > > at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.
> > > executeRequest(LibvirtComputingResource.java:1369)
> > > at com.cloud.agent.Agent.processRequest(Agent.java:525)
> > > at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:833)
> > > at com.cloud.utils.nio.Task.call(Task.java:83)
> > > at com.cloud.utils.nio.Task.call(Task.java:29)
> > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > at java.util.concurrent.ThreadPoolExecutor.runWorker(
> > > ThreadPoolExecutor.java:1149)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > > ThreadPoolExecutor.java:624)
> > > at java.lang.Thread.run(Thread.java:748)
> > > 2017-11-11 15:40:41,327 WARN  [resource.wrapper.
> > > LibvirtSecurityGroupRulesCommandWrapper] (agentRequest-Handler-2:null)
> > > (logid:eab9a328) Failed to program network rules for vm i-2-7-VM
> > >
> > > So, no rules are actually created. Script doesn't call... I suppose may
> > be
> > > quotes are required because shell interprets ';' as command separator.
> I
> > > suppose that optimization introduced in 4.10, because in 4.9 SGs work
> > like
> > > a charm...
> > >
> > >
> > > 2017-11-11 3:15 GMT+07:00 Paul Angus <pa...@shapeblue.com>:
> > >
> > >> Ivan,
> > >>
> > >> Can you paste a larger section of unfiltered logs.  There would always
> > be
> > >> a message explaining why the mgmt. server thought that a VR should be
> > shut
> > >> down
> > >>
> > >>
> > >>
> > >> Kind regards,
> > >>
> > >> Paul Angus
> > >>
> > >> paul.angus@shapeblue.com
> > >> www.shapeblue.com
> > >> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > >> @shapeblue
> > >>
> > >>
> > >>
> > >>
> > >> -----Original Message-----
> > >> From: Simon Weller [mailto:sweller@ena.com.INVALID]
> > >> Sent: 10 November 2017 18:39
> > >> To: dev@cloudstack.apache.org
> > >> Subject: Re: Apache CloudStack 4.10 VR/BasicZone/KVM Problem
> > >>
> > >> What VR template image are you using?
> > >>
> > >>
> > >> ________________________________
> > >> From: Ivan Kudryavtsev <ku...@bw-sw.com>
> > >> Sent: Friday, November 10, 2017 11:59 AM
> > >> To: dev@cloudstack.apache.org
> > >> Subject: Re: Apache CloudStack 4.10 VR/BasicZone/KVM Problem
> > >>
> > >> Hi. No, regular NFS. VR starts great, but stopped by ms, other system
> > vms
> > >> are working. I even added to communication script on compute node
> "sleep
> > >> 3600" before ssh, so response to management is delayed, I logged so to
> > VR,
> > >> all interfaces are up, iptables rules are OK.
> > >>
> > >> So agent rolls vr good, but stops it by management order with no
> obvious
> > >> reason.
> > >>
> > >> 11 нояб. 2017 г. 0:54 пользователь "Simon Weller"
> > <sweller@ena.com.invalid
> > >> >
> > >> написал:
> > >>
> > >> > Is the storage ceph?
> > >> >
> > >> >
> > >> > ________________________________
> > >> > From: Ivan Kudryavtsev <ku...@bw-sw.com>
> > >> > Sent: Friday, November 10, 2017 11:52 AM
> > >> > To: dev@cloudstack.apache.org
> > >> > Subject: Re: Apache CloudStack 4.10 VR/BasicZone/KVM Problem
> > >> >
> > >> > Hi, I did, and it does the things right, I even added "tee" to ssh
> > >> > 3922 communication script to out vr response to additional log and
> it
> > >> > only receives VR version line and sends all info (the same from
> > >> > pastebin) to ACS and receives "stop" order.
> > >> >
> > >> > I'll try to provide additional info, but ad you can see, management
> > >> > receives proper response and sends stop next op. It looks very
> freaky
> > >> > without any notification...
> > >> >
> > >> > 11 нояб. 2017 г. 0:37 пользователь "Simon Weller"
> > >> > <sweller@ena.com.invalid
> > >> > >
> > >> > написал:
> > >> >
> > >> > > Ivan,
> > >> > >
> > >> > >
> > >> > > Can you put the host agents into debug mode? Hopefully that will
> > >> > > provide more information.
> > >> > >
> > >> > >
> > >> > > https://cwiki.apache.org/confluence/display/CLOUDSTACK/
> > KVM+agent+deb
> > >> > > ug
> > >> KVM agent debug - Apache Cloudstack - Apache Software ...<
> > >> https://cwiki.apache.org/confluence/display/CLOUDSTACK/
> KVM+agent+debug>
> > >> cwiki.apache.org
> > >> Steps to debug the KVM agent from eclipse: In KVM agent edit
> > >> '/usr/libexec/agent-runner ', add "-Xrunjdwp:transport=dt_
> > socket,address=8787
> > >> ...
> > >>
> > >>
> > >>
> > >> > >
> > >> > >
> > >> > > - Si
> > >> > >
> > >> > > ________________________________
> > >> > > From: Ivan Kudryavtsev <ku...@bw-sw.com>
> > >> > > Sent: Friday, November 10, 2017 11:34 AM
> > >> > > To: dev@cloudstack.apache.org
> > >> > > Subject: Apache CloudStack 4.10 VR/BasicZone/KVM Problem
> > >> > >
> > >> > > Hello, Devs.
> > >> > >
> > >> > > I experience VR Start Problem in the fresh ACS 4.10 deployment
> > >> > >
> > >> > > Intersting place of logs is here: https://pastebin.com/iBXRBA5N
> > >> [https://pastebin.com/i/facebook.png]<https://pastebin.com/iBXRBA5N>
> > >>
> > >> 2017-11-10 23:05:35,853 DEBUG [c.c.a.t.Request]
> > >> (Work-Job-Executor-15:ctx-6fdf61 - Pastebin.com<https://pastebin.
> > >> com/iBXRBA5N>
> > >> pastebin.com
> > >>
> > >>
> > >>
> > >> > >
> > >> > > Basically, the situation looks like:
> > >> > >
> > >> > > 1. Management Server tries to launch VR 2. It gets from Agent
> proper
> > >> > > VR response with VR details 3. It sends StopCommand without
> > >> > > explanation.
> > >> > >
> > >> > > I'm trying to figure out what happens inside, but the codebase is
> > >> > > huge
> > >> > and
> > >> > > still no positive results. Please, let me know if you have any
> ideas
> > >> > which
> > >> > > could help me finding the reason. Thanks a lot.
> > >> > >
> > >> > > --
> > >> > > With best regards, Ivan Kudryavtsev
> > >> > > Bitworks Software, Ltd.
> > >> > > Cell: +7-923-414-1515
> > >> > > WWW: http://bitworks.software/ <http://bw-sw.com/>
> > >> > >
> > >> >
> > >>
> > >>
> > >
> > >
> > > --
> > > With best regards, Ivan Kudryavtsev
> > > Bitworks Software, Ltd.
> > > Cell: +7-923-414-1515
> > > WWW: http://bitworks.software/ <http://bw-sw.com/>
> > >
> > >
> >
> >
> > --
> > With best regards, Ivan Kudryavtsev
> > Bitworks Software, Ltd.
> > Cell: +7-923-414-1515
> > WWW: http://bitworks.software/ <http://bw-sw.com/>
> >
>



-- 
With best regards, Ivan Kudryavtsev
Bitworks Software, Ltd.
Cell: +7-923-414-1515
WWW: http://bitworks.software/ <http://bw-sw.com/>

Re: Apache CloudStack 4.10 VR/BasicZone/KVM Problem

[Özhan Rüzgar Karaman <or...@gmail.com>]

Hi Ivan;
Does this hotfixes also solve qoutes and shell script interprets problem?
We have no ipv6 setup and today we made similar test with fresh install
4.10. We noticed that we receive similar error on security groups stage
while br_netfilter module is already active on our environment. We made
same tests for Ubuntu 16.04.3 and 14.04.5 kvm hosts

Logs are below:
2017-11-13 11:47:41,773 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-1:null) Executing:
/usr/share/cloudstack-common/scripts/vm/network/security_group.py
add_network_rules --vmname i-2-5-VM --vmid 5 --vmip 192.168.18.6 --vmip6
null --sig 74a6d8c403af9c3c7b89ecf206e4ac26 --seq 16 --vmmac
1e:00:9b:00:00:05 --vif vnet8 --brname breth0-23 --nicsecips 0: --rules
I:tcp:1:65535:
0.0.0.0/0,NEXT;I:udp:1:65535:0.0.0.0/0,NEXT;E:tcp:1:65535:0.0.0.0/0,NEXT;
2017-11-13 11:47:41,773 WARN  [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-1:null) Exception:
/usr/share/cloudstack-common/scripts/vm/network/security_group.py
add_network_rules --vmname i-2-5-VM --vmid 5 --vmip 192.168.18.6 --vmip6
null --sig 74a6d8c403af9c3c7b89ecf206e4ac26 --seq 16 --vmmac
1e:00:9b:00:00:05 --vif vnet8 --brname breth0-23 --nicsecips 0: --rules
I:tcp:1:65535:
0.0.0.0/0,NEXT;I:udp:1:65535:0.0.0.0/0,NEXT;E:tcp:1:65535:0.0.0.0/0,NEXT;
java.lang.NullPointerException
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1012)
at com.cloud.utils.script.Script.execute(Script.java:214)
at com.cloud.utils.script.Script.execute(Script.java:182)
at
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.addNetworkRules(LibvirtComputingResource.java:3429)
at
com.cloud.hypervisor.kvm.resource.wrapper.LibvirtSecurityGroupRulesCommandWrapper.execute(LibvirtSecurityGroupRulesCommandWrapper.java:57)
at
com.cloud.hypervisor.kvm.resource.wrapper.LibvirtSecurityGroupRulesCommandWrapper.execute(LibvirtSecurityGroupRulesCommandWrapper.java:36)
at
com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:75)
at
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1369)
at com.cloud.agent.Agent.processRequest(Agent.java:525)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:833)
at com.cloud.utils.nio.Task.call(Task.java:83)
at com.cloud.utils.nio.Task.call(Task.java:29)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2017-11-13 11:47:41,774 WARN
[resource.wrapper.LibvirtSecurityGroupRulesCommandWrapper]
(agentRequest-Handler-1:null) Failed to program network rules for vm
i-2-5-VM
2017-11-13 11:47:41,775 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-1:null) Seq 1-6412562919422165093:  { Ans: , MgmtId:
345048635880, via: 1, Ver: v1, Flags: 110,
[{"com.cloud.agent.api.SecurityGroupRuleAnswer":{"logSequenceNumber":16,"vmId":5,"reason":"PROGRAMMING_FAILED","result":false,"details":"programming
network rules failed","wait":0}}] }


When we execute command with double quotas for rules section from command
line it executes without a problem like below:
root@kvmt3:/var/log/cloudstack/agent#
/usr/share/cloudstack-common/scripts/vm/network/security_group.py
add_network_rules --vmname i-2-5-VM --vmid 5 --vmip 192.168.18.6 --vmip6
null --sig 74a6d8c403af9c3c7b89ecf206e4ac26 --seq 16 --vmmac
1e:00:9b:00:00:05 --vif vnet8 --brname breth0-23 --nicsecips 0: --rules
"I:tcp:1:65535:
0.0.0.0/0,NEXT;I:udp:1:65535:0.0.0.0/0,NEXT;E:tcp:1:65535:0.0.0.0/0,NEXT;"
root@kvmt3:/var/log/cloudstack/agent# echo $?
0
root@kvmt3:/var/log/cloudstack/agent#

Thanks
Özhan


On Sat, Nov 11, 2017 at 6:59 PM, Ivan Kudryavtsev <ku...@bw-sw.com>
wrote:

> Hello, I implemented some hotfixes for 4.10 to work
>
> https://github.com/apache/cloudstack/pull/2319 - to master (load
> br_netfilter module)
> https://github.com/apache/cloudstack/pull/2320 - to 4.10 which fixes SG
> failures related to ipv6.
>
>
> 2017-11-11 15:51 GMT+07:00 Ivan Kudryavtsev <ku...@bw-sw.com>:
>
> > Following up with previous question. I managed to make it work by
> removing
> > all and heading to ubuntu 14.04 hypervisor host.
> >
> > Also, what I found more:
> >
> > 1. when setup databases (management server) if custom port is specified,
> > databases themself is not created. If create manually, import scripts
> work
> > fine.
> > 2. UI: unable to download ISO to __all__ zones. Have to specify certain
> > zone, else UI gives an error.
> > 3. Ubuntu doesn't load module *br_netfilter* but
> >
> > /usr/share/cloudstack-common/scripts/vm/network/security_group.py
> >
> > uses it and nothing good as a result:
> >
> > 2017-11-11 15:38:29,241 - sysctl -w net.bridge.bridge-nf-call-
> arptables=1
> > 2017-11-11 15:38:29,244 - sysctl -w net.bridge.bridge-nf-call-iptables=1
> > 2017-11-11 15:38:29,247 - sysctl -w net.bridge.bridge-nf-call-
> ip6tables=1
> >
> > adding br_netfilter to /etc/modules fixes it. I suppose it's necessary to
> > add
> >
> > in script something like: modprobe br_netfilter (or smarter thing).
> >
> > But It doesn't work completely, actually, security groups are
> unfunctional:
> >
> > ==> /var/log/cloudstack/agent/agent.log <==
> > 2017-11-11 15:40:41,326 WARN  [kvm.resource.LibvirtComputingResource]
> > (agentRequest-Handler-2:null) (logid:eab9a328) Exception:
> > /usr/share/cloudstack-common/scripts/vm/network/security_group.py
> > add_network_rules --vmname i-2-7-VM --vmid 7 --vmip 176.120.28.4 --vmip6
> > null --sig d60255deb618b7be9f477eed10d65234 --seq 4 --vmmac
> > 1e:00:6f:00:01:01 --vif vnet8 --brname cloudbr0 --nicsecips 0: --rules
> > I:icmp:-1:-1:0.0.0.0/0,NEXT;I:tcp:1:65535:0.0.0.0/0,NEXT;I:
> > udp:1:65535:0.0.0.0/0,NEXT;E:icmp:-1:-1:0.0.0.0/0,NEXT;E:
> > tcp:1:65535:0.0.0.0/0,NEXT;E:udp:1:65535:0.0.0.0/0,NEXT;
> > java.lang.NullPointerException
> > at java.lang.ProcessBuilder.start(ProcessBuilder.java:1012)
> > at com.cloud.utils.script.Script.execute(Script.java:214)
> > at com.cloud.utils.script.Script.execute(Script.java:182)
> > at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.
> > addNetworkRules(LibvirtComputingResource.java:3429)
> > at com.cloud.hypervisor.kvm.resource.wrapper.
> > LibvirtSecurityGroupRulesCommandWrapper.execute(
> > LibvirtSecurityGroupRulesCommandWrapper.java:57)
> > at com.cloud.hypervisor.kvm.resource.wrapper.
> > LibvirtSecurityGroupRulesCommandWrapper.execute(
> > LibvirtSecurityGroupRulesCommandWrapper.java:36)
> > at com.cloud.hypervisor.kvm.resource.wrapper.
> > LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:75)
> > at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.
> > executeRequest(LibvirtComputingResource.java:1369)
> > at com.cloud.agent.Agent.processRequest(Agent.java:525)
> > at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:833)
> > at com.cloud.utils.nio.Task.call(Task.java:83)
> > at com.cloud.utils.nio.Task.call(Task.java:29)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > at java.util.concurrent.ThreadPoolExecutor.runWorker(
> > ThreadPoolExecutor.java:1149)
> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > ThreadPoolExecutor.java:624)
> > at java.lang.Thread.run(Thread.java:748)
> > 2017-11-11 15:40:41,327 WARN  [resource.wrapper.
> > LibvirtSecurityGroupRulesCommandWrapper] (agentRequest-Handler-2:null)
> > (logid:eab9a328) Failed to program network rules for vm i-2-7-VM
> >
> > So, no rules are actually created. Script doesn't call... I suppose may
> be
> > quotes are required because shell interprets ';' as command separator. I
> > suppose that optimization introduced in 4.10, because in 4.9 SGs work
> like
> > a charm...
> >
> >
> > 2017-11-11 3:15 GMT+07:00 Paul Angus <pa...@shapeblue.com>:
> >
> >> Ivan,
> >>
> >> Can you paste a larger section of unfiltered logs.  There would always
> be
> >> a message explaining why the mgmt. server thought that a VR should be
> shut
> >> down
> >>
> >>
> >>
> >> Kind regards,
> >>
> >> Paul Angus
> >>
> >> paul.angus@shapeblue.com
> >> www.shapeblue.com
> >> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> >> @shapeblue
> >>
> >>
> >>
> >>
> >> -----Original Message-----
> >> From: Simon Weller [mailto:sweller@ena.com.INVALID]
> >> Sent: 10 November 2017 18:39
> >> To: dev@cloudstack.apache.org
> >> Subject: Re: Apache CloudStack 4.10 VR/BasicZone/KVM Problem
> >>
> >> What VR template image are you using?
> >>
> >>
> >> ________________________________
> >> From: Ivan Kudryavtsev <ku...@bw-sw.com>
> >> Sent: Friday, November 10, 2017 11:59 AM
> >> To: dev@cloudstack.apache.org
> >> Subject: Re: Apache CloudStack 4.10 VR/BasicZone/KVM Problem
> >>
> >> Hi. No, regular NFS. VR starts great, but stopped by ms, other system
> vms
> >> are working. I even added to communication script on compute node "sleep
> >> 3600" before ssh, so response to management is delayed, I logged so to
> VR,
> >> all interfaces are up, iptables rules are OK.
> >>
> >> So agent rolls vr good, but stops it by management order with no obvious
> >> reason.
> >>
> >> 11 нояб. 2017 г. 0:54 пользователь "Simon Weller"
> <sweller@ena.com.invalid
> >> >
> >> написал:
> >>
> >> > Is the storage ceph?
> >> >
> >> >
> >> > ________________________________
> >> > From: Ivan Kudryavtsev <ku...@bw-sw.com>
> >> > Sent: Friday, November 10, 2017 11:52 AM
> >> > To: dev@cloudstack.apache.org
> >> > Subject: Re: Apache CloudStack 4.10 VR/BasicZone/KVM Problem
> >> >
> >> > Hi, I did, and it does the things right, I even added "tee" to ssh
> >> > 3922 communication script to out vr response to additional log and it
> >> > only receives VR version line and sends all info (the same from
> >> > pastebin) to ACS and receives "stop" order.
> >> >
> >> > I'll try to provide additional info, but ad you can see, management
> >> > receives proper response and sends stop next op. It looks very freaky
> >> > without any notification...
> >> >
> >> > 11 нояб. 2017 г. 0:37 пользователь "Simon Weller"
> >> > <sweller@ena.com.invalid
> >> > >
> >> > написал:
> >> >
> >> > > Ivan,
> >> > >
> >> > >
> >> > > Can you put the host agents into debug mode? Hopefully that will
> >> > > provide more information.
> >> > >
> >> > >
> >> > > https://cwiki.apache.org/confluence/display/CLOUDSTACK/
> KVM+agent+deb
> >> > > ug
> >> KVM agent debug - Apache Cloudstack - Apache Software ...<
> >> https://cwiki.apache.org/confluence/display/CLOUDSTACK/KVM+agent+debug>
> >> cwiki.apache.org
> >> Steps to debug the KVM agent from eclipse: In KVM agent edit
> >> '/usr/libexec/agent-runner ', add "-Xrunjdwp:transport=dt_
> socket,address=8787
> >> ...
> >>
> >>
> >>
> >> > >
> >> > >
> >> > > - Si
> >> > >
> >> > > ________________________________
> >> > > From: Ivan Kudryavtsev <ku...@bw-sw.com>
> >> > > Sent: Friday, November 10, 2017 11:34 AM
> >> > > To: dev@cloudstack.apache.org
> >> > > Subject: Apache CloudStack 4.10 VR/BasicZone/KVM Problem
> >> > >
> >> > > Hello, Devs.
> >> > >
> >> > > I experience VR Start Problem in the fresh ACS 4.10 deployment
> >> > >
> >> > > Intersting place of logs is here: https://pastebin.com/iBXRBA5N
> >> [https://pastebin.com/i/facebook.png]<https://pastebin.com/iBXRBA5N>
> >>
> >> 2017-11-10 23:05:35,853 DEBUG [c.c.a.t.Request]
> >> (Work-Job-Executor-15:ctx-6fdf61 - Pastebin.com<https://pastebin.
> >> com/iBXRBA5N>
> >> pastebin.com
> >>
> >>
> >>
> >> > >
> >> > > Basically, the situation looks like:
> >> > >
> >> > > 1. Management Server tries to launch VR 2. It gets from Agent proper
> >> > > VR response with VR details 3. It sends StopCommand without
> >> > > explanation.
> >> > >
> >> > > I'm trying to figure out what happens inside, but the codebase is
> >> > > huge
> >> > and
> >> > > still no positive results. Please, let me know if you have any ideas
> >> > which
> >> > > could help me finding the reason. Thanks a lot.
> >> > >
> >> > > --
> >> > > With best regards, Ivan Kudryavtsev
> >> > > Bitworks Software, Ltd.
> >> > > Cell: +7-923-414-1515
> >> > > WWW: http://bitworks.software/ <http://bw-sw.com/>
> >> > >
> >> >
> >>
> >>
> >
> >
> > --
> > With best regards, Ivan Kudryavtsev
> > Bitworks Software, Ltd.
> > Cell: +7-923-414-1515
> > WWW: http://bitworks.software/ <http://bw-sw.com/>
> >
> >
>
>
> --
> With best regards, Ivan Kudryavtsev
> Bitworks Software, Ltd.
> Cell: +7-923-414-1515
> WWW: http://bitworks.software/ <http://bw-sw.com/>
>

Re: Apache CloudStack 4.10 VR/BasicZone/KVM Problem

[Ivan Kudryavtsev <ku...@bw-sw.com>]

Hello, I implemented some hotfixes for 4.10 to work

https://github.com/apache/cloudstack/pull/2319 - to master (load
br_netfilter module)
https://github.com/apache/cloudstack/pull/2320 - to 4.10 which fixes SG
failures related to ipv6.


2017-11-11 15:51 GMT+07:00 Ivan Kudryavtsev <ku...@bw-sw.com>:

> Following up with previous question. I managed to make it work by removing
> all and heading to ubuntu 14.04 hypervisor host.
>
> Also, what I found more:
>
> 1. when setup databases (management server) if custom port is specified,
> databases themself is not created. If create manually, import scripts work
> fine.
> 2. UI: unable to download ISO to __all__ zones. Have to specify certain
> zone, else UI gives an error.
> 3. Ubuntu doesn't load module *br_netfilter* but
>
> /usr/share/cloudstack-common/scripts/vm/network/security_group.py
>
> uses it and nothing good as a result:
>
> 2017-11-11 15:38:29,241 - sysctl -w net.bridge.bridge-nf-call-arptables=1
> 2017-11-11 15:38:29,244 - sysctl -w net.bridge.bridge-nf-call-iptables=1
> 2017-11-11 15:38:29,247 - sysctl -w net.bridge.bridge-nf-call-ip6tables=1
>
> adding br_netfilter to /etc/modules fixes it. I suppose it's necessary to
> add
>
> in script something like: modprobe br_netfilter (or smarter thing).
>
> But It doesn't work completely, actually, security groups are unfunctional:
>
> ==> /var/log/cloudstack/agent/agent.log <==
> 2017-11-11 15:40:41,326 WARN  [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-2:null) (logid:eab9a328) Exception:
> /usr/share/cloudstack-common/scripts/vm/network/security_group.py
> add_network_rules --vmname i-2-7-VM --vmid 7 --vmip 176.120.28.4 --vmip6
> null --sig d60255deb618b7be9f477eed10d65234 --seq 4 --vmmac
> 1e:00:6f:00:01:01 --vif vnet8 --brname cloudbr0 --nicsecips 0: --rules
> I:icmp:-1:-1:0.0.0.0/0,NEXT;I:tcp:1:65535:0.0.0.0/0,NEXT;I:
> udp:1:65535:0.0.0.0/0,NEXT;E:icmp:-1:-1:0.0.0.0/0,NEXT;E:
> tcp:1:65535:0.0.0.0/0,NEXT;E:udp:1:65535:0.0.0.0/0,NEXT;
> java.lang.NullPointerException
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:1012)
> at com.cloud.utils.script.Script.execute(Script.java:214)
> at com.cloud.utils.script.Script.execute(Script.java:182)
> at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.
> addNetworkRules(LibvirtComputingResource.java:3429)
> at com.cloud.hypervisor.kvm.resource.wrapper.
> LibvirtSecurityGroupRulesCommandWrapper.execute(
> LibvirtSecurityGroupRulesCommandWrapper.java:57)
> at com.cloud.hypervisor.kvm.resource.wrapper.
> LibvirtSecurityGroupRulesCommandWrapper.execute(
> LibvirtSecurityGroupRulesCommandWrapper.java:36)
> at com.cloud.hypervisor.kvm.resource.wrapper.
> LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:75)
> at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.
> executeRequest(LibvirtComputingResource.java:1369)
> at com.cloud.agent.Agent.processRequest(Agent.java:525)
> at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:833)
> at com.cloud.utils.nio.Task.call(Task.java:83)
> at com.cloud.utils.nio.Task.call(Task.java:29)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2017-11-11 15:40:41,327 WARN  [resource.wrapper.
> LibvirtSecurityGroupRulesCommandWrapper] (agentRequest-Handler-2:null)
> (logid:eab9a328) Failed to program network rules for vm i-2-7-VM
>
> So, no rules are actually created. Script doesn't call... I suppose may be
> quotes are required because shell interprets ';' as command separator. I
> suppose that optimization introduced in 4.10, because in 4.9 SGs work like
> a charm...
>
>
> 2017-11-11 3:15 GMT+07:00 Paul Angus <pa...@shapeblue.com>:
>
>> Ivan,
>>
>> Can you paste a larger section of unfiltered logs.  There would always be
>> a message explaining why the mgmt. server thought that a VR should be shut
>> down
>>
>>
>>
>> Kind regards,
>>
>> Paul Angus
>>
>> paul.angus@shapeblue.com
>> www.shapeblue.com
>> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>> @shapeblue
>>
>>
>>
>>
>> -----Original Message-----
>> From: Simon Weller [mailto:sweller@ena.com.INVALID]
>> Sent: 10 November 2017 18:39
>> To: dev@cloudstack.apache.org
>> Subject: Re: Apache CloudStack 4.10 VR/BasicZone/KVM Problem
>>
>> What VR template image are you using?
>>
>>
>> ________________________________
>> From: Ivan Kudryavtsev <ku...@bw-sw.com>
>> Sent: Friday, November 10, 2017 11:59 AM
>> To: dev@cloudstack.apache.org
>> Subject: Re: Apache CloudStack 4.10 VR/BasicZone/KVM Problem
>>
>> Hi. No, regular NFS. VR starts great, but stopped by ms, other system vms
>> are working. I even added to communication script on compute node "sleep
>> 3600" before ssh, so response to management is delayed, I logged so to VR,
>> all interfaces are up, iptables rules are OK.
>>
>> So agent rolls vr good, but stops it by management order with no obvious
>> reason.
>>
>> 11 нояб. 2017 г. 0:54 пользователь "Simon Weller" <sweller@ena.com.invalid
>> >
>> написал:
>>
>> > Is the storage ceph?
>> >
>> >
>> > ________________________________
>> > From: Ivan Kudryavtsev <ku...@bw-sw.com>
>> > Sent: Friday, November 10, 2017 11:52 AM
>> > To: dev@cloudstack.apache.org
>> > Subject: Re: Apache CloudStack 4.10 VR/BasicZone/KVM Problem
>> >
>> > Hi, I did, and it does the things right, I even added "tee" to ssh
>> > 3922 communication script to out vr response to additional log and it
>> > only receives VR version line and sends all info (the same from
>> > pastebin) to ACS and receives "stop" order.
>> >
>> > I'll try to provide additional info, but ad you can see, management
>> > receives proper response and sends stop next op. It looks very freaky
>> > without any notification...
>> >
>> > 11 нояб. 2017 г. 0:37 пользователь "Simon Weller"
>> > <sweller@ena.com.invalid
>> > >
>> > написал:
>> >
>> > > Ivan,
>> > >
>> > >
>> > > Can you put the host agents into debug mode? Hopefully that will
>> > > provide more information.
>> > >
>> > >
>> > > https://cwiki.apache.org/confluence/display/CLOUDSTACK/KVM+agent+deb
>> > > ug
>> KVM agent debug - Apache Cloudstack - Apache Software ...<
>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/KVM+agent+debug>
>> cwiki.apache.org
>> Steps to debug the KVM agent from eclipse: In KVM agent edit
>> '/usr/libexec/agent-runner ', add "-Xrunjdwp:transport=dt_socket,address=8787
>> ...
>>
>>
>>
>> > >
>> > >
>> > > - Si
>> > >
>> > > ________________________________
>> > > From: Ivan Kudryavtsev <ku...@bw-sw.com>
>> > > Sent: Friday, November 10, 2017 11:34 AM
>> > > To: dev@cloudstack.apache.org
>> > > Subject: Apache CloudStack 4.10 VR/BasicZone/KVM Problem
>> > >
>> > > Hello, Devs.
>> > >
>> > > I experience VR Start Problem in the fresh ACS 4.10 deployment
>> > >
>> > > Intersting place of logs is here: https://pastebin.com/iBXRBA5N
>> [https://pastebin.com/i/facebook.png]<https://pastebin.com/iBXRBA5N>
>>
>> 2017-11-10 23:05:35,853 DEBUG [c.c.a.t.Request]
>> (Work-Job-Executor-15:ctx-6fdf61 - Pastebin.com<https://pastebin.
>> com/iBXRBA5N>
>> pastebin.com
>>
>>
>>
>> > >
>> > > Basically, the situation looks like:
>> > >
>> > > 1. Management Server tries to launch VR 2. It gets from Agent proper
>> > > VR response with VR details 3. It sends StopCommand without
>> > > explanation.
>> > >
>> > > I'm trying to figure out what happens inside, but the codebase is
>> > > huge
>> > and
>> > > still no positive results. Please, let me know if you have any ideas
>> > which
>> > > could help me finding the reason. Thanks a lot.
>> > >
>> > > --
>> > > With best regards, Ivan Kudryavtsev
>> > > Bitworks Software, Ltd.
>> > > Cell: +7-923-414-1515
>> > > WWW: http://bitworks.software/ <http://bw-sw.com/>
>> > >
>> >
>>
>>
>
>
> --
> With best regards, Ivan Kudryavtsev
> Bitworks Software, Ltd.
> Cell: +7-923-414-1515
> WWW: http://bitworks.software/ <http://bw-sw.com/>
>
>


-- 
With best regards, Ivan Kudryavtsev
Bitworks Software, Ltd.
Cell: +7-923-414-1515
WWW: http://bitworks.software/ <http://bw-sw.com/>

Re: Apache CloudStack 4.10 VR/BasicZone/KVM Problem

[Ivan Kudryavtsev <ku...@bw-sw.com>]

Following up with previous question. I managed to make it work by removing
all and heading to ubuntu 14.04 hypervisor host.

Also, what I found more:

1. when setup databases (management server) if custom port is specified,
databases themself is not created. If create manually, import scripts work
fine.
2. UI: unable to download ISO to __all__ zones. Have to specify certain
zone, else UI gives an error.
3. Ubuntu doesn't load module *br_netfilter* but

/usr/share/cloudstack-common/scripts/vm/network/security_group.py

uses it and nothing good as a result:

2017-11-11 15:38:29,241 - sysctl -w net.bridge.bridge-nf-call-arptables=1
2017-11-11 15:38:29,244 - sysctl -w net.bridge.bridge-nf-call-iptables=1
2017-11-11 15:38:29,247 - sysctl -w net.bridge.bridge-nf-call-ip6tables=1

adding br_netfilter to /etc/modules fixes it. I suppose it's necessary to
add

in script something like: modprobe br_netfilter (or smarter thing).

But It doesn't work completely, actually, security groups are unfunctional:

==> /var/log/cloudstack/agent/agent.log <==
2017-11-11 15:40:41,326 WARN  [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-2:null) (logid:eab9a328) Exception:
/usr/share/cloudstack-common/scripts/vm/network/security_group.py
add_network_rules --vmname i-2-7-VM --vmid 7 --vmip 176.120.28.4 --vmip6
null --sig d60255deb618b7be9f477eed10d65234 --seq 4 --vmmac
1e:00:6f:00:01:01 --vif vnet8 --brname cloudbr0 --nicsecips 0: --rules
I:icmp:-1:-1:
0.0.0.0/0,NEXT;I:tcp:1:65535:0.0.0.0/0,NEXT;I:udp:1:65535:0.0.0.0/0,NEXT;E:icmp:-1:-1:0.0.0.0/0,NEXT;E:tcp:1:65535:0.0.0.0/0,NEXT;E:udp:1:65535:0.0.0.0/0,NEXT
;
java.lang.NullPointerException
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1012)
at com.cloud.utils.script.Script.execute(Script.java:214)
at com.cloud.utils.script.Script.execute(Script.java:182)
at
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.addNetworkRules(LibvirtComputingResource.java:3429)
at
com.cloud.hypervisor.kvm.resource.wrapper.LibvirtSecurityGroupRulesCommandWrapper.execute(LibvirtSecurityGroupRulesCommandWrapper.java:57)
at
com.cloud.hypervisor.kvm.resource.wrapper.LibvirtSecurityGroupRulesCommandWrapper.execute(LibvirtSecurityGroupRulesCommandWrapper.java:36)
at
com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:75)
at
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1369)
at com.cloud.agent.Agent.processRequest(Agent.java:525)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:833)
at com.cloud.utils.nio.Task.call(Task.java:83)
at com.cloud.utils.nio.Task.call(Task.java:29)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2017-11-11 15:40:41,327 WARN
 [resource.wrapper.LibvirtSecurityGroupRulesCommandWrapper]
(agentRequest-Handler-2:null) (logid:eab9a328) Failed to program network
rules for vm i-2-7-VM

So, no rules are actually created. Script doesn't call... I suppose may be
quotes are required because shell interprets ';' as command separator. I
suppose that optimization introduced in 4.10, because in 4.9 SGs work like
a charm...


2017-11-11 3:15 GMT+07:00 Paul Angus <pa...@shapeblue.com>:

> Ivan,
>
> Can you paste a larger section of unfiltered logs.  There would always be
> a message explaining why the mgmt. server thought that a VR should be shut
> down
>
>
>
> Kind regards,
>
> Paul Angus
>
> paul.angus@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> @shapeblue
>
>
>
>
> -----Original Message-----
> From: Simon Weller [mailto:sweller@ena.com.INVALID]
> Sent: 10 November 2017 18:39
> To: dev@cloudstack.apache.org
> Subject: Re: Apache CloudStack 4.10 VR/BasicZone/KVM Problem
>
> What VR template image are you using?
>
>
> ________________________________
> From: Ivan Kudryavtsev <ku...@bw-sw.com>
> Sent: Friday, November 10, 2017 11:59 AM
> To: dev@cloudstack.apache.org
> Subject: Re: Apache CloudStack 4.10 VR/BasicZone/KVM Problem
>
> Hi. No, regular NFS. VR starts great, but stopped by ms, other system vms
> are working. I even added to communication script on compute node "sleep
> 3600" before ssh, so response to management is delayed, I logged so to VR,
> all interfaces are up, iptables rules are OK.
>
> So agent rolls vr good, but stops it by management order with no obvious
> reason.
>
> 11 нояб. 2017 г. 0:54 пользователь "Simon Weller" <sweller@ena.com.invalid
> >
> написал:
>
> > Is the storage ceph?
> >
> >
> > ________________________________
> > From: Ivan Kudryavtsev <ku...@bw-sw.com>
> > Sent: Friday, November 10, 2017 11:52 AM
> > To: dev@cloudstack.apache.org
> > Subject: Re: Apache CloudStack 4.10 VR/BasicZone/KVM Problem
> >
> > Hi, I did, and it does the things right, I even added "tee" to ssh
> > 3922 communication script to out vr response to additional log and it
> > only receives VR version line and sends all info (the same from
> > pastebin) to ACS and receives "stop" order.
> >
> > I'll try to provide additional info, but ad you can see, management
> > receives proper response and sends stop next op. It looks very freaky
> > without any notification...
> >
> > 11 нояб. 2017 г. 0:37 пользователь "Simon Weller"
> > <sweller@ena.com.invalid
> > >
> > написал:
> >
> > > Ivan,
> > >
> > >
> > > Can you put the host agents into debug mode? Hopefully that will
> > > provide more information.
> > >
> > >
> > > https://cwiki.apache.org/confluence/display/CLOUDSTACK/KVM+agent+deb
> > > ug
> KVM agent debug - Apache Cloudstack - Apache Software ...<
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/KVM+agent+debug>
> cwiki.apache.org
> Steps to debug the KVM agent from eclipse: In KVM agent edit
> '/usr/libexec/agent-runner ', add "-Xrunjdwp:transport=dt_socket,address=8787
> ...
>
>
>
> > >
> > >
> > > - Si
> > >
> > > ________________________________
> > > From: Ivan Kudryavtsev <ku...@bw-sw.com>
> > > Sent: Friday, November 10, 2017 11:34 AM
> > > To: dev@cloudstack.apache.org
> > > Subject: Apache CloudStack 4.10 VR/BasicZone/KVM Problem
> > >
> > > Hello, Devs.
> > >
> > > I experience VR Start Problem in the fresh ACS 4.10 deployment
> > >
> > > Intersting place of logs is here: https://pastebin.com/iBXRBA5N
> [https://pastebin.com/i/facebook.png]<https://pastebin.com/iBXRBA5N>
>
> 2017-11-10 23:05:35,853 DEBUG [c.c.a.t.Request] (Work-Job-Executor-15:ctx-6fdf61
> - Pastebin.com<https://pastebin.com/iBXRBA5N>
> pastebin.com
>
>
>
> > >
> > > Basically, the situation looks like:
> > >
> > > 1. Management Server tries to launch VR 2. It gets from Agent proper
> > > VR response with VR details 3. It sends StopCommand without
> > > explanation.
> > >
> > > I'm trying to figure out what happens inside, but the codebase is
> > > huge
> > and
> > > still no positive results. Please, let me know if you have any ideas
> > which
> > > could help me finding the reason. Thanks a lot.
> > >
> > > --
> > > With best regards, Ivan Kudryavtsev
> > > Bitworks Software, Ltd.
> > > Cell: +7-923-414-1515
> > > WWW: http://bitworks.software/ <http://bw-sw.com/>
> > >
> >
>
>


-- 
With best regards, Ivan Kudryavtsev
Bitworks Software, Ltd.
Cell: +7-923-414-1515
WWW: http://bitworks.software/ <http://bw-sw.com/>

RE: Apache CloudStack 4.10 VR/BasicZone/KVM Problem

[Paul Angus <pa...@shapeblue.com>]

Ivan,

Can you paste a larger section of unfiltered logs.  There would always be a message explaining why the mgmt. server thought that a VR should be shut down 



Kind regards,

Paul Angus

paul.angus@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 


-----Original Message-----
From: Simon Weller [mailto:sweller@ena.com.INVALID] 
Sent: 10 November 2017 18:39
To: dev@cloudstack.apache.org
Subject: Re: Apache CloudStack 4.10 VR/BasicZone/KVM Problem

What VR template image are you using?


________________________________
From: Ivan Kudryavtsev <ku...@bw-sw.com>
Sent: Friday, November 10, 2017 11:59 AM
To: dev@cloudstack.apache.org
Subject: Re: Apache CloudStack 4.10 VR/BasicZone/KVM Problem

Hi. No, regular NFS. VR starts great, but stopped by ms, other system vms are working. I even added to communication script on compute node "sleep 3600" before ssh, so response to management is delayed, I logged so to VR, all interfaces are up, iptables rules are OK.

So agent rolls vr good, but stops it by management order with no obvious reason.

11 нояб. 2017 г. 0:54 пользователь "Simon Weller" <sw...@ena.com.invalid>
написал:

> Is the storage ceph?
>
>
> ________________________________
> From: Ivan Kudryavtsev <ku...@bw-sw.com>
> Sent: Friday, November 10, 2017 11:52 AM
> To: dev@cloudstack.apache.org
> Subject: Re: Apache CloudStack 4.10 VR/BasicZone/KVM Problem
>
> Hi, I did, and it does the things right, I even added "tee" to ssh 
> 3922 communication script to out vr response to additional log and it 
> only receives VR version line and sends all info (the same from 
> pastebin) to ACS and receives "stop" order.
>
> I'll try to provide additional info, but ad you can see, management 
> receives proper response and sends stop next op. It looks very freaky 
> without any notification...
>
> 11 нояб. 2017 г. 0:37 пользователь "Simon Weller" 
> <sweller@ena.com.invalid
> >
> написал:
>
> > Ivan,
> >
> >
> > Can you put the host agents into debug mode? Hopefully that will 
> > provide more information.
> >
> >
> > https://cwiki.apache.org/confluence/display/CLOUDSTACK/KVM+agent+deb
> > ug
KVM agent debug - Apache Cloudstack - Apache Software ...<https://cwiki.apache.org/confluence/display/CLOUDSTACK/KVM+agent+debug>
cwiki.apache.org
Steps to debug the KVM agent from eclipse: In KVM agent edit '/usr/libexec/agent-runner ', add "-Xrunjdwp:transport=dt_socket,address=8787 ...



> >
> >
> > - Si
> >
> > ________________________________
> > From: Ivan Kudryavtsev <ku...@bw-sw.com>
> > Sent: Friday, November 10, 2017 11:34 AM
> > To: dev@cloudstack.apache.org
> > Subject: Apache CloudStack 4.10 VR/BasicZone/KVM Problem
> >
> > Hello, Devs.
> >
> > I experience VR Start Problem in the fresh ACS 4.10 deployment
> >
> > Intersting place of logs is here: https://pastebin.com/iBXRBA5N
[https://pastebin.com/i/facebook.png]<https://pastebin.com/iBXRBA5N>

2017-11-10 23:05:35,853 DEBUG [c.c.a.t.Request] (Work-Job-Executor-15:ctx-6fdf61 - Pastebin.com<https://pastebin.com/iBXRBA5N>
pastebin.com



> >
> > Basically, the situation looks like:
> >
> > 1. Management Server tries to launch VR 2. It gets from Agent proper 
> > VR response with VR details 3. It sends StopCommand without 
> > explanation.
> >
> > I'm trying to figure out what happens inside, but the codebase is 
> > huge
> and
> > still no positive results. Please, let me know if you have any ideas
> which
> > could help me finding the reason. Thanks a lot.
> >
> > --
> > With best regards, Ivan Kudryavtsev
> > Bitworks Software, Ltd.
> > Cell: +7-923-414-1515
> > WWW: http://bitworks.software/ <http://bw-sw.com/>
> >
>

Re: Apache CloudStack 4.10 VR/BasicZone/KVM Problem

[Simon Weller <sw...@ena.com.INVALID>]

What VR template image are you using?


________________________________
From: Ivan Kudryavtsev <ku...@bw-sw.com>
Sent: Friday, November 10, 2017 11:59 AM
To: dev@cloudstack.apache.org
Subject: Re: Apache CloudStack 4.10 VR/BasicZone/KVM Problem

Hi. No, regular NFS. VR starts great, but stopped by ms, other system vms
are working. I even added to communication script on compute node "sleep
3600" before ssh, so response to management is delayed, I logged so to VR,
all interfaces are up, iptables rules are OK.

So agent rolls vr good, but stops it by management order with no obvious
reason.

11 нояб. 2017 г. 0:54 пользователь "Simon Weller" <sw...@ena.com.invalid>
написал:

> Is the storage ceph?
>
>
> ________________________________
> From: Ivan Kudryavtsev <ku...@bw-sw.com>
> Sent: Friday, November 10, 2017 11:52 AM
> To: dev@cloudstack.apache.org
> Subject: Re: Apache CloudStack 4.10 VR/BasicZone/KVM Problem
>
> Hi, I did, and it does the things right, I even added "tee" to ssh 3922
> communication script to out vr response to additional log and it only
> receives VR version line and sends all info (the same from pastebin) to ACS
> and receives "stop" order.
>
> I'll try to provide additional info, but ad you can see, management
> receives proper response and sends stop next op. It looks very freaky
> without any notification...
>
> 11 нояб. 2017 г. 0:37 пользователь "Simon Weller" <sweller@ena.com.invalid
> >
> написал:
>
> > Ivan,
> >
> >
> > Can you put the host agents into debug mode? Hopefully that will provide
> > more information.
> >
> >
> > https://cwiki.apache.org/confluence/display/CLOUDSTACK/KVM+agent+debug
KVM agent debug - Apache Cloudstack - Apache Software ...<https://cwiki.apache.org/confluence/display/CLOUDSTACK/KVM+agent+debug>
cwiki.apache.org
Steps to debug the KVM agent from eclipse: In KVM agent edit ‘/usr/libexec/agent-runner ‘, add "-Xrunjdwp:transport=dt_socket,address=8787 ...



> >
> >
> > - Si
> >
> > ________________________________
> > From: Ivan Kudryavtsev <ku...@bw-sw.com>
> > Sent: Friday, November 10, 2017 11:34 AM
> > To: dev@cloudstack.apache.org
> > Subject: Apache CloudStack 4.10 VR/BasicZone/KVM Problem
> >
> > Hello, Devs.
> >
> > I experience VR Start Problem in the fresh ACS 4.10 deployment
> >
> > Intersting place of logs is here: https://pastebin.com/iBXRBA5N
[https://pastebin.com/i/facebook.png]<https://pastebin.com/iBXRBA5N>

2017-11-10 23:05:35,853 DEBUG [c.c.a.t.Request] (Work-Job-Executor-15:ctx-6fdf61 - Pastebin.com<https://pastebin.com/iBXRBA5N>
pastebin.com



> >
> > Basically, the situation looks like:
> >
> > 1. Management Server tries to launch VR
> > 2. It gets from Agent proper VR response with VR details
> > 3. It sends StopCommand without explanation.
> >
> > I'm trying to figure out what happens inside, but the codebase is huge
> and
> > still no positive results. Please, let me know if you have any ideas
> which
> > could help me finding the reason. Thanks a lot.
> >
> > --
> > With best regards, Ivan Kudryavtsev
> > Bitworks Software, Ltd.
> > Cell: +7-923-414-1515
> > WWW: http://bitworks.software/ <http://bw-sw.com/>
> >
>

Re: Apache CloudStack 4.10 VR/BasicZone/KVM Problem

[Ivan Kudryavtsev <ku...@bw-sw.com>]

Hi. No, regular NFS. VR starts great, but stopped by ms, other system vms
are working. I even added to communication script on compute node "sleep
3600" before ssh, so response to management is delayed, I logged so to VR,
all interfaces are up, iptables rules are OK.

So agent rolls vr good, but stops it by management order with no obvious
reason.

11 нояб. 2017 г. 0:54 пользователь "Simon Weller" <sw...@ena.com.invalid>
написал:

> Is the storage ceph?
>
>
> ________________________________
> From: Ivan Kudryavtsev <ku...@bw-sw.com>
> Sent: Friday, November 10, 2017 11:52 AM
> To: dev@cloudstack.apache.org
> Subject: Re: Apache CloudStack 4.10 VR/BasicZone/KVM Problem
>
> Hi, I did, and it does the things right, I even added "tee" to ssh 3922
> communication script to out vr response to additional log and it only
> receives VR version line and sends all info (the same from pastebin) to ACS
> and receives "stop" order.
>
> I'll try to provide additional info, but ad you can see, management
> receives proper response and sends stop next op. It looks very freaky
> without any notification...
>
> 11 нояб. 2017 г. 0:37 пользователь "Simon Weller" <sweller@ena.com.invalid
> >
> написал:
>
> > Ivan,
> >
> >
> > Can you put the host agents into debug mode? Hopefully that will provide
> > more information.
> >
> >
> > https://cwiki.apache.org/confluence/display/CLOUDSTACK/KVM+agent+debug
> >
> >
> > - Si
> >
> > ________________________________
> > From: Ivan Kudryavtsev <ku...@bw-sw.com>
> > Sent: Friday, November 10, 2017 11:34 AM
> > To: dev@cloudstack.apache.org
> > Subject: Apache CloudStack 4.10 VR/BasicZone/KVM Problem
> >
> > Hello, Devs.
> >
> > I experience VR Start Problem in the fresh ACS 4.10 deployment
> >
> > Intersting place of logs is here: https://pastebin.com/iBXRBA5N
> >
> > Basically, the situation looks like:
> >
> > 1. Management Server tries to launch VR
> > 2. It gets from Agent proper VR response with VR details
> > 3. It sends StopCommand without explanation.
> >
> > I'm trying to figure out what happens inside, but the codebase is huge
> and
> > still no positive results. Please, let me know if you have any ideas
> which
> > could help me finding the reason. Thanks a lot.
> >
> > --
> > With best regards, Ivan Kudryavtsev
> > Bitworks Software, Ltd.
> > Cell: +7-923-414-1515
> > WWW: http://bitworks.software/ <http://bw-sw.com/>
> >
>

Re: Apache CloudStack 4.10 VR/BasicZone/KVM Problem

[Simon Weller <sw...@ena.com.INVALID>]

Is the storage ceph?


________________________________
From: Ivan Kudryavtsev <ku...@bw-sw.com>
Sent: Friday, November 10, 2017 11:52 AM
To: dev@cloudstack.apache.org
Subject: Re: Apache CloudStack 4.10 VR/BasicZone/KVM Problem

Hi, I did, and it does the things right, I even added "tee" to ssh 3922
communication script to out vr response to additional log and it only
receives VR version line and sends all info (the same from pastebin) to ACS
and receives "stop" order.

I'll try to provide additional info, but ad you can see, management
receives proper response and sends stop next op. It looks very freaky
without any notification...

11 нояб. 2017 г. 0:37 пользователь "Simon Weller" <sw...@ena.com.invalid>
написал:

> Ivan,
>
>
> Can you put the host agents into debug mode? Hopefully that will provide
> more information.
>
>
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/KVM+agent+debug
>
>
> - Si
>
> ________________________________
> From: Ivan Kudryavtsev <ku...@bw-sw.com>
> Sent: Friday, November 10, 2017 11:34 AM
> To: dev@cloudstack.apache.org
> Subject: Apache CloudStack 4.10 VR/BasicZone/KVM Problem
>
> Hello, Devs.
>
> I experience VR Start Problem in the fresh ACS 4.10 deployment
>
> Intersting place of logs is here: https://pastebin.com/iBXRBA5N
>
> Basically, the situation looks like:
>
> 1. Management Server tries to launch VR
> 2. It gets from Agent proper VR response with VR details
> 3. It sends StopCommand without explanation.
>
> I'm trying to figure out what happens inside, but the codebase is huge and
> still no positive results. Please, let me know if you have any ideas which
> could help me finding the reason. Thanks a lot.
>
> --
> With best regards, Ivan Kudryavtsev
> Bitworks Software, Ltd.
> Cell: +7-923-414-1515
> WWW: http://bitworks.software/ <http://bw-sw.com/>
>

Re: Apache CloudStack 4.10 VR/BasicZone/KVM Problem

[Ivan Kudryavtsev <ku...@bw-sw.com>]

Hi, I did, and it does the things right, I even added "tee" to ssh 3922
communication script to out vr response to additional log and it only
receives VR version line and sends all info (the same from pastebin) to ACS
and receives "stop" order.

I'll try to provide additional info, but ad you can see, management
receives proper response and sends stop next op. It looks very freaky
without any notification...

11 нояб. 2017 г. 0:37 пользователь "Simon Weller" <sw...@ena.com.invalid>
написал:

> Ivan,
>
>
> Can you put the host agents into debug mode? Hopefully that will provide
> more information.
>
>
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/KVM+agent+debug
>
>
> - Si
>
> ________________________________
> From: Ivan Kudryavtsev <ku...@bw-sw.com>
> Sent: Friday, November 10, 2017 11:34 AM
> To: dev@cloudstack.apache.org
> Subject: Apache CloudStack 4.10 VR/BasicZone/KVM Problem
>
> Hello, Devs.
>
> I experience VR Start Problem in the fresh ACS 4.10 deployment
>
> Intersting place of logs is here: https://pastebin.com/iBXRBA5N
>
> Basically, the situation looks like:
>
> 1. Management Server tries to launch VR
> 2. It gets from Agent proper VR response with VR details
> 3. It sends StopCommand without explanation.
>
> I'm trying to figure out what happens inside, but the codebase is huge and
> still no positive results. Please, let me know if you have any ideas which
> could help me finding the reason. Thanks a lot.
>
> --
> With best regards, Ivan Kudryavtsev
> Bitworks Software, Ltd.
> Cell: +7-923-414-1515
> WWW: http://bitworks.software/ <http://bw-sw.com/>
>

Re: Apache CloudStack 4.10 VR/BasicZone/KVM Problem

[Simon Weller <sw...@ena.com.INVALID>]

Ivan,


Can you put the host agents into debug mode? Hopefully that will provide more information.


https://cwiki.apache.org/confluence/display/CLOUDSTACK/KVM+agent+debug


- Si

________________________________
From: Ivan Kudryavtsev <ku...@bw-sw.com>
Sent: Friday, November 10, 2017 11:34 AM
To: dev@cloudstack.apache.org
Subject: Apache CloudStack 4.10 VR/BasicZone/KVM Problem

Hello, Devs.

I experience VR Start Problem in the fresh ACS 4.10 deployment

Intersting place of logs is here: https://pastebin.com/iBXRBA5N

Basically, the situation looks like:

1. Management Server tries to launch VR
2. It gets from Agent proper VR response with VR details
3. It sends StopCommand without explanation.

I'm trying to figure out what happens inside, but the codebase is huge and
still no positive results. Please, let me know if you have any ideas which
could help me finding the reason. Thanks a lot.

--
With best regards, Ivan Kudryavtsev
Bitworks Software, Ltd.
Cell: +7-923-414-1515
WWW: http://bitworks.software/ <http://bw-sw.com/>