You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Hean Seng <he...@gmail.com> on 2020/09/27 09:18:02 UTC
Cloudstack Advance with Security Group
Hi
I created advance zone with security group, all working fine.
But VMcreated ， seems the default security group that assigned to the VM.
all accept policy , i understand is Default Deny, and once add in the port
in Security Group Ingress and Egress, only is allowed
Also, is this rules created at VirtualRouter of the SharedNetwork, or at
the Hypervisor?
--
Regards,
Hean Seng
Re: Cloudstack Advance with Security Group
Posted by Hean Seng <he...@gmail.com>.
i try to edit the security rules, and following capture in log , i guess
is initall iptables rules not created, and thus cause to this :
2020-09-28 02:23:30,276 ERROR [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-4:null) (logid:6ae48108) Unable to apply default
network rule for nic cloudbr0 for VM i-2-81-VM
2020-09-28 02:23:30,276 WARN
[resource.wrapper.LibvirtSecurityGroupRulesCommandWrapper]
(agentRequest-Handler-4:null) (logid:6ae48108) Failed to program default
network rules for vm i-2-81-VM
2020-09-28 02:23:30,323 ERROR [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-3:null) (logid:bbe20113) Unable to apply default
network rule for nic cloudbr0 for VM i-2-78-VM
2020-09-28 02:23:30,323 WARN
[resource.wrapper.LibvirtSecurityGroupRulesCommandWrapper]
(agentRequest-Handler-3:null) (logid:bbe20113) Failed to program default
network rules for vm i-2-78-VM
2020-09-28 02:23:30,370 ERROR [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-2:null) (logid:ffe01b13) Unable to apply default
network rule for nic cloudbr0 for VM i-2-76-VM
2020-09-28 02:23:30,371 WARN
[resource.wrapper.LibvirtSecurityGroupRulesCommandWrapper]
(agentRequest-Handler-2:null) (logid:ffe01b13) Failed to program default
network rules for vm i-2-76-VM
On Mon, Sep 28, 2020 at 2:10 PM Hean Seng <he...@gmail.com> wrote:
> I checked the hypervisor , it seems iptables is nothing inside , this is
> centos7 , initially i turnoff firewalld , but even i turn on it now and
> try to update the security group rules, it seems empty iptable rules :
>
> [root@kvm03 ~]# iptables -L -v -n
>
> Chain INPUT (policy ACCEPT 82903 packets, 1170M bytes)
>
> pkts bytes target prot opt in out source
> destination
>
>
> Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
>
> pkts bytes target prot opt in out source
> destination
>
>
> Chain OUTPUT (policy ACCEPT 80505 packets, 25M bytes)
>
> pkts bytes target prot opt in out source
> destination
>
>
>
>
>
>
>
> On Mon, Sep 28, 2020 at 12:05 PM Pearl d'Silva <pe...@shapeblue.com>
> wrote:
>
>> Hi Hean,
>>
>> In an Advanced Zone with Security Groups enabled, by default, egress
>> traffic from the VM is allowed, while Ingress traffic is denied. Hence, as
>> you rightly mentioned, security group rules are added accordingly. These
>> rules get added on the hypervisor host, and you can verify them, by going
>> into the host and searching for iptables rules corresponding to the VM
>> (internal name - i-x-y-VM).
>> This blog maybe helpful in providing further details:
>>
>> https://shankerbalan.net/blog/cloudstack-advanced-zone-with-security-groups/
>>
>> Thanks,
>> Pearl
>> ________________________________
>> From: Hean Seng <he...@gmail.com>
>> Sent: Sunday, September 27, 2020 2:48 PM
>> To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
>> Subject: Cloudstack Advance with Security Group
>>
>> Hi
>>
>> I created advance zone with security group, all working fine.
>>
>> But VMcreated ， seems the default security group that assigned to the VM.
>> all accept policy , i understand is Default Deny, and once add in the
>> port
>> in Security Group Ingress and Egress, only is allowed
>>
>> Also, is this rules created at VirtualRouter of the SharedNetwork, or at
>> the Hypervisor?
>>
>>
>>
>> --
>> Regards,
>> Hean Seng
>>
>> pearl.dsilva@shapeblue.com
>> www.shapeblue.com
>> 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK
>> @shapeblue
>>
>>
>>
>>
>
> --
> Regards,
> Hean Seng
>
--
Regards,
Hean Seng
Re: Cloudstack Advance with Security Group
Posted by Hean Seng <he...@gmail.com>.
Follow on this release, for those wish to use CentOS 7, i have found the
solution ,
Need manual install python36-libvirt , then security group solved in
centos7 .
Ubuntu default installation has no issue.
On Tue, Sep 29, 2020 at 10:10 PM Hean Seng <he...@gmail.com> wrote:
> Hi
>
> I manage install another Unbuntu 18 HyperVisor on KVM and add to a new
> cluster ( ** apparently one customer only can accept all hypervisor with
> same OS, thus I have to create new customer for Ubuntu KVM hypervisor)
>
> And the Security Group work fine.
>
> But in CentOS7 , it not working due to error mentioned just now
>
> Will this consider BUG ?
>
>
>
>
>
>
>
>
>
>
> On Tue, Sep 29, 2020 at 6:06 PM Hean Seng <he...@gmail.com> wrote:
>
>> Just for your info, i am installing another host in Ubuntu 18, and just
>> defant installation, the script you mention seems normal , as compared to
>> CentOS 7.8 .
>>
>> Now I suspect could be the issue of Cloudstack and CentOS7.8 , or
>> many the version of python etc .
>>
>> I will try to complete the host installation in Ubuntu and adding the
>> hypervisor in Mgmt Node ,and try it see if work
>>
>>
>> root@kmv04:~#
>> /usr/share/cloudstack-common/scripts/vm/network/security_group.py
>>
>> usage: security_group.py [-h] [--vmname VMNAME] [--vmip VMIP] [--vmip6
>> VMIP6]
>>
>> [--vmid VMID] [--vmmac VMMAC] [--vif VIF]
>> [--sig SIG]
>>
>> [--seq SEQ] [--rules RULES] [--brname BRNAME]
>>
>> [--localbrname LOCALBRNAME] [--dhcpSvr DHCPSVR]
>>
>> [--hostIp HOSTIP] [--hostMacAddr HOSTMACADDR]
>>
>> [--nicsecips NICSECIPS] [--action ACTION]
>>
>> [--privnic PRIVNIC] [--isFirstNic] [--check]
>>
>> command
>>
>> security_group.py: error: the following arguments are required: command
>>
>>
>>
>>
>> On Tue, Sep 29, 2020 at 4:12 PM Hean Seng <he...@gmail.com> wrote:
>>
>>> Hi
>>>
>>> It seem libvirt-phython already install previously
>>>
>>>
>>> [root@kvm03 agent]# yum install libvirt-python -y
>>>
>>> Failed to set locale, defaulting to C
>>>
>>> Loaded plugins: fastestmirror
>>>
>>> Loading mirror speeds from cached hostfile
>>>
>>> * base: hk.mirrors.thegigabit.com
>>>
>>> * epel: hk.mirrors.thegigabit.com
>>>
>>> * extras: mirror-hk.koddos.net
>>>
>>> * updates: hk.mirrors.thegigabit.com
>>>
>>> Package libvirt-python-4.5.0-1.el7.x86_64 already installed and latest
>>> version
>>>
>>> Nothing to do
>>>
>>> This all installed
>>>
>>> libvirt-libs-4.5.0-33.el7_8.1.x86_64
>>>
>>> libvirt-daemon-4.5.0-33.el7_8.1.x86_64
>>>
>>> libvirt-daemon-config-nwfilter-4.5.0-33.el7_8.1.x86_64
>>>
>>> libvirt-daemon-driver-storage-rbd-4.5.0-33.el7_8.1.x86_64
>>>
>>> libvirt-daemon-driver-storage-gluster-4.5.0-33.el7_8.1.x86_64
>>>
>>> libvirt-daemon-driver-nodedev-4.5.0-33.el7_8.1.x86_64
>>>
>>> libvirt-client-4.5.0-33.el7_8.1.x86_64
>>>
>>> libvirt-daemon-driver-storage-core-4.5.0-33.el7_8.1.x86_64
>>>
>>> libvirt-daemon-driver-nwfilter-4.5.0-33.el7_8.1.x86_64
>>>
>>> libvirt-daemon-driver-qemu-4.5.0-33.el7_8.1.x86_64
>>>
>>> libvirt-daemon-driver-lxc-4.5.0-33.el7_8.1.x86_64
>>>
>>> libvirt-daemon-driver-storage-logical-4.5.0-33.el7_8.1.x86_64
>>>
>>> libvirt-daemon-driver-storage-disk-4.5.0-33.el7_8.1.x86_64
>>>
>>> libvirt-daemon-driver-storage-iscsi-4.5.0-33.el7_8.1.x86_64
>>>
>>> libvirt-daemon-driver-storage-4.5.0-33.el7_8.1.x86_64
>>>
>>> libvirt-daemon-driver-secret-4.5.0-33.el7_8.1.x86_64
>>>
>>> libvirt-4.5.0-33.el7_8.1.x86_64
>>>
>>> libvirt-bash-completion-4.5.0-33.el7_8.1.x86_64
>>>
>>> libvirt-python-4.5.0-1.el7.x86_64
>>>
>>> libvirt-daemon-driver-network-4.5.0-33.el7_8.1.x86_64
>>>
>>> libvirt-daemon-config-network-4.5.0-33.el7_8.1.x86_64
>>>
>>> libvirt-daemon-driver-storage-scsi-4.5.0-33.el7_8.1.x86_64
>>>
>>> libvirt-daemon-driver-storage-mpath-4.5.0-33.el7_8.1.x86_64
>>>
>>> libvirt-daemon-driver-interface-4.5.0-33.el7_8.1.x86_64
>>>
>>>
>>>
>>>
>>> On Tue, Sep 29, 2020 at 3:46 PM Pearl d'Silva <
>>> pearl.dsilva@shapeblue.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> Yes, the script wouldn't do anything, if it's just run without args,
>>>> however, it should prompt you with the args to be provided. Based on the
>>>> error, it seems that you are missing a python library, hence the script
>>>> terminates and that's why the log file isn't generated in the first place.
>>>> You can install the package using:
>>>> yum install libvirt-python
>>>>
>>>> Thanks,
>>>> Pearl
>>>> ________________________________
>>>> From: Hean Seng <he...@gmail.com>
>>>> Sent: Tuesday, September 29, 2020 1:10 PM
>>>> To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
>>>> Subject: Re: Cloudstack Advance with Security Group
>>>>
>>>> The error when running manually:
>>>>
>>>> ]# /usr/share/cloudstack-common/scripts/vm/network/security_group.py
>>>>
>>>> Traceback (most recent call last):
>>>>
>>>> File
>>>> "/usr/share/cloudstack-common/scripts/vm/network/security_group.py",
>>>> line 26, in <module>
>>>>
>>>> import libvirt
>>>>
>>>> Package libvirt-4.5.0-33.el7_8.1.x86_64 already installed and latest
>>>> version
>>>>
>>>>
>>>> so I guess, cannot runjust like that ?
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> pearl.dsilva@shapeblue.com
>>>> www.shapeblue.com
>>>> 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK
>>>> @shapeblue
>>>>
>>>>
>>>>
>>>> On Tue, Sep 29, 2020 at 3:08 PM Hean Seng <he...@gmail.com> wrote:
>>>>
>>>> > Hi
>>>> >
>>>> > I am running on CentOS 7.8.2003
>>>> >
>>>> > Cloudstack Agent , and KVM hypervisor:
>>>> >
>>>> > cloudstack-agent-4.14.0.0-1.el7.x86_64
>>>> >
>>>> > iptables and ebtables aldy install, iptables -L and ebtables -L show
>>>> > default rules .
>>>> >
>>>> > This is dumpxml, seem the bridge is there, the VLAN for Guest is
>>>> VLAN401,
>>>> > which the setup seem right :
>>>> >
>>>> > <interface type='bridge'>
>>>> >
>>>> > <mac address='02:00:21:d9:00:01'/>
>>>> >
>>>> > <source bridge='brem1-401'/>
>>>> >
>>>> > <bandwidth>
>>>> >
>>>> > <inbound average='1280' peak='1280'/>
>>>> >
>>>> > <outbound average='1280' peak='1280'/>
>>>> >
>>>> > </bandwidth>
>>>> >
>>>> > <target dev='vnet9'/>
>>>> >
>>>> > <model type='virtio'/>
>>>> >
>>>> > <link state='up'/>
>>>> >
>>>> > <alias name='net0'/>
>>>> >
>>>> > <address type='pci' domain='0x0000' bus='0x00' slot='0x03'
>>>> > function='0x0'/>
>>>> >
>>>> >
>>>> >
>>>> > # ifconfig brem1-401
>>>> >
>>>> > brem1-401: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
>>>> >
>>>> > inet6 fe80::3824:b2ff:fe09:873f prefixlen 64 scopeid
>>>> 0x20<link>
>>>> >
>>>> > ether 04:7d:7b:60:09:66 txqueuelen 1000 (Ethernet)
>>>> >
>>>> > RX packets 7191638 bytes 333185781 (317.7 MiB)
>>>> >
>>>> > RX errors 0 dropped 0 overruns 0 frame 0
>>>> >
>>>> > TX packets 8 bytes 656 (656.0 B)
>>>> >
>>>> > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
>>>> >
>>>> > brctl show
>>>> >
>>>> > bridge name bridge id STP enabled interfaces
>>>> >
>>>> > brem1-401 8000.047d7b600966 no em1.401
>>>> >
>>>> > vnet10
>>>> >
>>>> > vnet11
>>>> >
>>>> > vnet12
>>>> >
>>>> > vnet14
>>>> >
>>>> > vnet4
>>>> >
>>>> > vnet8
>>>> >
>>>> > vnet9
>>>> >
>>>> > cloud0 8000.fe00a9fe1cea no vnet13
>>>> >
>>>> > vnet15
>>>> >
>>>> > vnet2
>>>> >
>>>> > vnet6
>>>> >
>>>> > cloudbr0 8000.047d7b600966 yes em1
>>>> >
>>>> > vnet3
>>>> >
>>>> > vnet5
>>>> >
>>>> > vnet7
>>>> >
>>>> > cloudbr1 8000.000000000000 yes
>>>> >
>>>> >
>>>> >
>>>> > I always have /var/log/cloudstack/agent/ this path , just inside
>>>> >
>>>> > ls /var/log/cloudstack/agent/
>>>> >
>>>> > agent.log agent.log.2020-09-26.gz agent.log.2020-09-27.gz
>>>> > agent.log.2020-09-28.gz resizevolume.log setup.log
>>>> >
>>>> > it doesnt have securitygroup folder/ log, all the log ip pasted is at
>>>> > agent.log
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > On Tue, Sep 29, 2020 at 2:44 PM Pearl d'Silva <
>>>> pearl.dsilva@shapeblue.com>
>>>> > wrote:
>>>> >
>>>> >> Hi Hean,
>>>> >>
>>>> >> I was able to deploy a VM in a shared network in an advanced zone
>>>> with
>>>> >> security groups enabled and could see the iptables rules getting
>>>> added for
>>>> >> the VM on the host, so it is probably not a bug that we are seeing.
>>>> Seems
>>>> >> like a setup issue.
>>>> >> Applying the rules can fail because:
>>>> >>
>>>> >> * The hypervisor host doesn't have iptables / ebtables command -
>>>> >> which is probably unlikely
>>>> >> * The VM deployed doesn't have an interface - can be verified by
>>>> >> dumping its configuration - virsh dumpxml <domain_name> | grep
>>>> interface
>>>> >> * or application of the default network rules for the VM itself
>>>> >> failed -If you are up for some debugging, can you try running the
>>>> script
>>>> >> manually in the hypervisor host:
>>>> >>
>>>> >> /usr/share/cloudstack-common/scripts/vm/network/security_group.py
>>>> >> default_network_rules --vmname <domain_name> --vmid <vm_id> --vmip
>>>> <vm_ip>
>>>> >> --vmmac <vm_mac> --vif <device/networkname> --brname <bridge_name>
>>>> >> --nicsecips 0; --isFirstNic --check
>>>> >>
>>>> >> when you do a virsh dumpxml <vm_domain_name>, look out for an
>>>> interface
>>>> >> section that looks something like:
>>>> >>
>>>> >> <interface type='bridge'>
>>>> >> <mac address='1e:00:86:00:00:ba'/>
>>>> >> <source bridge='breth1-11'/>
>>>> >> <bandwidth>
>>>> >> <inbound average='25600' peak='25600'/>
>>>> >> <outbound average='25600' peak='25600'/>
>>>> >> </bandwidth>
>>>> >> <target dev='vnet0'/>
>>>> >> <model type='virtio'/>
>>>> >> <link state='up'/>
>>>> >> <alias name='net0'/>
>>>> >> <address type='pci' domain='0x0000' bus='0x00' slot='0x03'
>>>> >> function='0x0'/>
>>>> >> </interface>
>>>> >>
>>>> >>
>>>> >> the inputs to the script can be found as follows:
>>>> >> domain_name - internal name of your vm (i-x-y-VM); can also be
>>>> obtained
>>>> >> by "virsh list"
>>>> >> vm_id - the 'y' part of the internal name : i-x-y-VM
>>>> >> vm_ip - Now that debug logs have been enabled, you can search for
>>>> >> "StartCommand" in your logs get the ip from the 'nics' section or
>>>> you could
>>>> >> choose any ip from the shared network range
>>>> >> vm_mac - from the above bridge section, you will be able to get the
>>>> MAC
>>>> >> address
>>>> >> vif / device name - taget dev name - in the above case - vnet0
>>>> >> bridge_name - again can be obtained from the above snippet (eg,
>>>> breth1-11)
>>>> >> if there are no secondary ips, then pass 0 for nicsecips otherwise
>>>> >> specify the seconday ips separated by a semicolon (;)
>>>> >>
>>>> >> Basically, at the end of it, it should look something like:
>>>> >> /usr/share/cloudstack-common/scripts/vm/network/security_group.py
>>>> >> default_network_rules --vmname i-2-3-VM --vmid 3 --vmip <ip> --vmmac
>>>> >> 1e:00:86:00:00:ba --vif vnet0 --brname breth1-11 --nicsecips 0;
>>>> --isFirstNic
>>>> >>
>>>> >> Hopefully, now you should be able to see a security_group.log file
>>>> in the
>>>> >> /var/log/cloudstack/agent/ path.
>>>> >>
>>>> >> Thanks,
>>>> >> Pearl
>>>> >>
>>>> >>
>>>> >>
>>>> >>
>>>> >>
>>>> >> ________________________________
>>>> >> From: Hean Seng <he...@gmail.com>
>>>> >> Sent: Tuesday, September 29, 2020 10:45 AM
>>>> >> To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
>>>> >> Subject: Re: Cloudstack Advance with Security Group
>>>> >>
>>>> >> Hi Pearl,
>>>> >>
>>>> >> I had change to debug, following is the error captured:
>>>> >>
>>>> >>
>>>> >> 2020-09-29 01:13:30,090 DEBUG [cloud.agent.Agent]
>>>> >> (agentRequest-Handler-2:null) (logid:388b92e0) Processing command:
>>>> >> com.cloud.agent.api.SecurityGroupRulesCmd
>>>> >>
>>>> >> 2020-09-29 01:13:30,090 DEBUG [kvm.resource.LibvirtConnection]
>>>> >> (agentRequest-Handler-2:null) (logid:388b92e0) Looking for libvirtd
>>>> >> connection at: qemu:///system
>>>> >>
>>>> >> 2020-09-29 01:13:30,094 DEBUG [kvm.resource.LibvirtComputingResource]
>>>> >> (agentRequest-Handler-2:null) (logid:388b92e0) Checking default
>>>> network
>>>> >> rules for vm i-23-77-VM
>>>> >>
>>>> >> 2020-09-29 01:13:30,094 ERROR [kvm.resource.LibvirtComputingResource]
>>>> >> (agentRequest-Handler-2:null) (logid:388b92e0) Unable to apply
>>>> default
>>>> >> network rule for nic cloudbr0 for VM i-23-77-VM
>>>> >>
>>>> >> 2020-09-29 01:13:30,094 WARN
>>>> >> [resource.wrapper.LibvirtSecurityGroupRulesCommandWrapper]
>>>> >> (agentRequest-Handler-2:null) (logid:388b92e0) Failed to program
>>>> default
>>>> >> network rules for vm i-23-77-VM
>>>> >>
>>>> >> 2020-09-29 01:13:30,095 DEBUG [cloud.agent.Agent]
>>>> >> (agentRequest-Handler-2:null) (logid:388b92e0) Seq
>>>> >> 11-6057904448766738456: {
>>>> >> Ans: , MgmtId: 72752492851638, via: 11, Ver: v1, Flags: 110,
>>>> >>
>>>> >>
>>>> [{"com.cloud.agent.api.SecurityGroupRuleAnswer":{"logSequenceNumber":15,"vmId":77,"reason":"PROGRAMMING_FAILED","result":false,"details":"programming
>>>> >> default network rules failed","wait":0}}] }
>>>> >>
>>>> >>
>>>> >>
>>>> >>
>>>> >>
>>>> >> On Tue, Sep 29, 2020 at 12:38 PM Pearl d'Silva <
>>>> >> pearl.dsilva@shapeblue.com>
>>>> >> wrote:
>>>> >>
>>>> >> > Hi Hean,
>>>> >> >
>>>> >> >
>>>> >> > Could you try enabling debug logging on your hypervisor hosts, by
>>>> >> running
>>>> >> > the following:
>>>> >> > *sed -i 's/INFO/DEBUG/g' /etc/cloudstack/agent/log4j-cloud.xml*
>>>> >> >
>>>> >> > And then restart the cloudstack-agent service on the hosts. Post
>>>> this,
>>>> >> try
>>>> >> > deploying a VM in the shared network. This may give a better
>>>> insight
>>>> >> into
>>>> >> > where exactly the issue lies.
>>>> >> >
>>>> >> > Thanks,
>>>> >> > Pearl
>>>> >> >
>>>> >> >
>>>> >> > Software Engineer
>>>> >> > pearl.dsilva@shapeblue.com
>>>> >> > www.shapeblue.com<http://www.shapeblue.com>
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> > ------------------------------
>>>> >> > *From:* Ivan Kudryavtsev <iv...@bw-sw.com>
>>>> >> > *Sent:* Monday, September 28, 2020 2:17 PM
>>>> >> > *To:* users <us...@cloudstack.apache.org>
>>>> >> > *Subject:* Re: Cloudstack Advance with Security Group
>>>> >> >
>>>> >> > This is it.
>>>> >> >
>>>> >>
>>>> >> pearl.dsilva@shapeblue.com
>>>> >> www.shapeblue.com<http://www.shapeblue.com>
>>>> >> 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK
>>>> >> @shapeblue
>>>> >>
>>>> >>
>>>> >>
>>>> >> > On Mon, Sep 28, 2020 at 3:46 PM Hean Seng <he...@gmail.com>
>>>> wrote:
>>>> >> >
>>>> >> > > In the log, I cannot see anything much, except a few lines
>>>> showing
>>>> >> above
>>>> >> > > . Not sure if this is a bug on 4.14.
>>>> >> > >
>>>> >> > >
>>>> >> > >
>>>> >> > > 2020-09-28 03:53:36,585 ERROR
>>>> [kvm.resource.LibvirtComputingResource]
>>>> >> > > (agentRequest-Handler-4:null) (logid:b6a4c077) Unable to apply
>>>> default
>>>> >> > > network rule for nic cloudbr0 for VM i-2-81-VM
>>>> >> > >
>>>> >> > > 2020-09-28 03:53:36,858 ERROR
>>>> [kvm.resource.LibvirtComputingResource]
>>>> >> > > (agentRequest-Handler-3:null) (logid:74058678) Unable to apply
>>>> default
>>>> >> > > network rule for nic cloudbr0 for VM i-2-81-VM
>>>> >> > >
>>>> >> > > 2020-09-28 03:53:36,858 WARN
>>>> >> > > [resource.wrapper.LibvirtSecurityGroupRulesCommandWrapper]
>>>> >> > > (agentRequest-Handler-3:null) (logid:74058678) Failed to program
>>>> >> default
>>>> >> > > network rules for vm i-2-81-VM
>>>> >> > >
>>>> >> > >
>>>> >> > >
>>>> >> > >
>>>> >> > > On Mon, Sep 28, 2020 at 4:34 PM Ivan Kudryavtsev <ivan@bw-sw.com
>>>> >
>>>> >> wrote:
>>>> >> > >
>>>> >> > > > Hi,
>>>> >> > > > no I'm on 4.11, so can not help with exact 4.14, and I'm on
>>>> Ubuntu,
>>>> >> > > though,
>>>> >> > > > but for any KVM hypervisor Linux distribution, the logic is the
>>>> >> same.
>>>> >> > > >
>>>> >> > > > On Mon, Sep 28, 2020 at 3:31 PM Hean Seng <he...@gmail.com>
>>>> >> wrote:
>>>> >> > > >
>>>> >> > > > > Hi
>>>> >> > > > >
>>>> >> > > > > Are you running on CentOS7 ?
>>>> >> > > > >
>>>> >> > > > > I am running on CentOS7 , ACS 4.14 , and seem there is no
>>>> log
>>>> >> at
>>>> >> > of
>>>> >> > > > > security_group.log
>>>> >> > > > >
>>>> >> > > > > # ls /var/log/cloudstack/agent/
>>>> >> > > > >
>>>> >> > > > > agent.log resizevolume.log setup.log
>>>> >> > > > >
>>>> >> > > > >
>>>> >> > > > > I recheck back the Intall guide, seems no missing anything.
>>>> >> > > > >
>>>> >> > > > >
>>>> >> > > > > Older intallation guide, 4.11 mentioned need , allow
>>>> >> > > > > /usr/lib/sysctl.d/00-system.conf
>>>> >> > > > >
>>>> >> > > > > # Enable netfilter on bridges.
>>>> >> net.bridge.bridge-nf-call-ip6tables =
>>>> >> > 1
>>>> >> > > > > net.bridge.bridge-nf-call-iptables = 1
>>>> >> > > > net.bridge.bridge-nf-call-arptables
>>>> >> > > > > = 1
>>>> >> > > > >
>>>> >> > > > > And it has been done too.
>>>> >> > > > >
>>>> >> > > > >
>>>> >> > > > >
>>>> >> > > > > On Mon, Sep 28, 2020 at 4:05 PM Ivan Kudryavtsev <
>>>> ivan@bw-sw.com>
>>>> >> > > wrote:
>>>> >> > > > >
>>>> >> > > > > > Hi,
>>>> >> > > > > >
>>>> >> > > > > > No, this is not the issue.
>>>> >> > > > > > It's a normal state of the system, as KVM hooks are a new
>>>> and
>>>> >> > > optional
>>>> >> > > > > > feature of 4.14.
>>>> >> > > > > >
>>>> >> > > > > > You should find some sort of messages regarding
>>>> security_groups
>>>> >> at
>>>> >> > > > > > /var/log/cloudstack/agent/security_group.log
>>>> >> > > > > >
>>>> >> > > > > >
>>>> >> > > > > > On Mon, Sep 28, 2020 at 2:10 PM Hean Seng <
>>>> heanseng@gmail.com>
>>>> >> > > wrote:
>>>> >> > > > > >
>>>> >> > > > > > > I not sure where goes wrong, are you running on CentOS
>>>> 7 ? I
>>>> >> > have
>>>> >> > > > this
>>>> >> > > > > > > error too, do you think is this contribute to the error
>>>> as
>>>> >> well:
>>>> >> > > > > > >
>>>> >> > > > > > > 2020-09-28 03:04:52,762 WARN
>>>> >> [kvm.resource.LibvirtKvmAgentHook]
>>>> >> > > > > > > (agentRequest-Handler-5:null) (logid:4f23845b) Groovy
>>>> script
>>>> >> > > > > > >
>>>> >> '/etc/cloudstack/agent/hooks/libvirt-vm-xml-transformer.groovy'
>>>> >> > is
>>>> >> > > > not
>>>> >> > > > > > > available. Transformations will not be applied.
>>>> >> > > > > > >
>>>> >> > > > > > > 2020-09-28 03:04:52,762 WARN
>>>> >> [kvm.resource.LibvirtKvmAgentHook]
>>>> >> > > > > > > (agentRequest-Handler-5:null) (logid:4f23845b) Groovy
>>>> >> scripting
>>>> >> > > > engine
>>>> >> > > > > is
>>>> >> > > > > > > not initialized. Data transformation skipped.
>>>> >> > > > > > >
>>>> >> > > > > > > 2020-09-28 03:04:53,083 WARN
>>>> >> [kvm.resource.LibvirtKvmAgentHook]
>>>> >> > > > > > > (agentRequest-Handler-5:null) (logid:4f23845b) Groovy
>>>> script
>>>> >> > > > > > >
>>>> '/etc/cloudstack/agent/hooks/libvirt-vm-state-change.groovy'
>>>> >> is
>>>> >> > not
>>>> >> > > > > > > available. Transformations will not be applied.
>>>> >> > > > > > >
>>>> >> > > > > > > On Mon, Sep 28, 2020 at 2:27 PM Ivan Kudryavtsev <
>>>> >> ivan@bw-sw.com
>>>> >> > >
>>>> >> > > > > wrote:
>>>> >> > > > > > >
>>>> >> > > > > > > > This just means you installed it in the wrong way.
>>>> Ebtables
>>>> >> and
>>>> >> > > > > > Iptables
>>>> >> > > > > > > > must be filled with rules like
>>>> >> > > > > > > >
>>>> >> > > > > > > > -A i-6242-10304-def -m state --state
>>>> RELATED,ESTABLISHED -j
>>>> >> > > ACCEPT
>>>> >> > > > > > > > -A i-6242-10304-def -p udp -m physdev --physdev-in
>>>> vnet18
>>>> >> > > > > > > > --physdev-is-bridged -m udp --sport 68 --dport 67 -j
>>>> ACCEPT
>>>> >> > > > > > > > -A i-6242-10304-def -p udp -m physdev --physdev-out
>>>> vnet18
>>>> >> > > > > > > > --physdev-is-bridged -m udp --sport 67 --dport 68 -j
>>>> ACCEPT
>>>> >> > > > > > > > -A i-6242-10304-def -m physdev --physdev-in vnet18
>>>> >> > > > > --physdev-is-bridged
>>>> >> > > > > > > -m
>>>> >> > > > > > > > set ! --match-set i-6242-10304-vm src -j DROP
>>>> >> > > > > > > > -A i-6242-10304-def -p udp -m physdev --physdev-in
>>>> vnet18
>>>> >> > > > > > > > --physdev-is-bridged -m set --match-set
>>>> i-6242-10304-vm src
>>>> >> -m
>>>> >> > > udp
>>>> >> > > > > > > --dport
>>>> >> > > > > > > > 53 -j RETURN
>>>> >> > > > > > > > -A i-6242-10304-def -p tcp -m physdev --physdev-in
>>>> vnet18
>>>> >> > > > > > > > --physdev-is-bridged -m set --match-set
>>>> i-6242-10304-vm src
>>>> >> -m
>>>> >> > > tcp
>>>> >> > > > > > > --dport
>>>> >> > > > > > > > 53 -j RETURN
>>>> >> > > > > > > > -A i-6242-10304-def -m physdev --physdev-in vnet18
>>>> >> > > > > --physdev-is-bridged
>>>> >> > > > > > > -m
>>>> >> > > > > > > > set --match-set i-6242-10304-vm src -j
>>>> i-6242-10304-vm-eg
>>>> >> > > > > > > > -A i-6242-10304-def -m physdev --physdev-out vnet18
>>>> >> > > > > > --physdev-is-bridged
>>>> >> > > > > > > -j
>>>> >> > > > > > > > i-6242-10304-vm
>>>> >> > > > > > > > -A i-6242-10304-vm -p udp -m udp --dport 1:65535 -m
>>>> state
>>>> >> > --state
>>>> >> > > > NEW
>>>> >> > > > > > -j
>>>> >> > > > > > > > ACCEPT
>>>> >> > > > > > > > -A i-6242-10304-vm -p tcp -m tcp --dport 1:65535 -m
>>>> state
>>>> >> > --state
>>>> >> > > > NEW
>>>> >> > > > > > -j
>>>> >> > > > > > > > ACCEPT
>>>> >> > > > > > > > -A i-6242-10304-vm -p icmp -m icmp --icmp-type any -j
>>>> ACCEPT
>>>> >> > > > > > > > -A i-6242-10304-vm -j DROP
>>>> >> > > > > > > >
>>>> >> > > > > > > >
>>>> >> > > > > > > > Bridge chain: i-4435-8929-vm-in, entries: 7, policy:
>>>> ACCEPT
>>>> >> > > > > > > > -s ! 1e:0:32:0:2:2 -j DROP
>>>> >> > > > > > > > -p ARP -s ! 1e:0:32:0:2:2 -j DROP
>>>> >> > > > > > > > -p ARP --arp-mac-src ! 1e:0:32:0:2:2 -j DROP
>>>> >> > > > > > > > -p ARP -j i-4435-8929-vm-in-ips
>>>> >> > > > > > > > -p ARP --arp-op Request -j ACCEPT
>>>> >> > > > > > > > -p ARP --arp-op Reply -j ACCEPT
>>>> >> > > > > > > > -p ARP -j DROP
>>>> >> > > > > > > >
>>>> >> > > > > > > >
>>>> >> > > > > > > >
>>>> >> > > > > > > > On Mon, Sep 28, 2020 at 1:10 PM Hean Seng <
>>>> >> heanseng@gmail.com>
>>>> >> > > > > wrote:
>>>> >> > > > > > > >
>>>> >> > > > > > > > > I checked the hypervisor , it seems iptables is
>>>> nothing
>>>> >> > inside
>>>> >> > > ,
>>>> >> > > > > > this
>>>> >> > > > > > > is
>>>> >> > > > > > > > > centos7 , initially i turnoff firewalld , but even
>>>> i
>>>> >> turn
>>>> >> > on
>>>> >> > > it
>>>> >> > > > > now
>>>> >> > > > > > > and
>>>> >> > > > > > > > > try to update the security group rules, it seems
>>>> empty
>>>> >> > iptable
>>>> >> > > > > rules
>>>> >> > > > > > :
>>>> >> > > > > > > > >
>>>> >> > > > > > > > > [root@kvm03 ~]# iptables -L -v -n
>>>> >> > > > > > > > >
>>>> >> > > > > > > > > Chain INPUT (policy ACCEPT 82903 packets, 1170M
>>>> bytes)
>>>> >> > > > > > > > >
>>>> >> > > > > > > > > pkts bytes target prot opt in out source
>>>> >> > > > > > > > > destination
>>>> >> > > > > > > > >
>>>> >> > > > > > > > >
>>>> >> > > > > > > > > Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
>>>> >> > > > > > > > >
>>>> >> > > > > > > > > pkts bytes target prot opt in out source
>>>> >> > > > > > > > > destination
>>>> >> > > > > > > > >
>>>> >> > > > > > > > >
>>>> >> > > > > > > > > Chain OUTPUT (policy ACCEPT 80505 packets, 25M bytes)
>>>> >> > > > > > > > >
>>>> >> > > > > > > > > pkts bytes target prot opt in out source
>>>> >> > > > > > > > > destination
>>>> >> > > > > > > > >
>>>> >> > > > > > > > >
>>>> >> > > > > > > > >
>>>> >> > > > > > > > >
>>>> >> > > > > > > > >
>>>> >> > > > > > > > >
>>>> >> > > > > > > > >
>>>> >> > > > > > > > > On Mon, Sep 28, 2020 at 12:05 PM Pearl d'Silva <
>>>> >> > > > > > > > pearl.dsilva@shapeblue.com
>>>> >> > > > > > > > > >
>>>> >> > > > > > > > > wrote:
>>>> >> > > > > > > > >
>>>> >> > > > > > > > > > Hi Hean,
>>>> >> > > > > > > > > >
>>>> >> > > > > > > > > > In an Advanced Zone with Security Groups enabled,
>>>> by
>>>> >> > default,
>>>> >> > > > > > egress
>>>> >> > > > > > > > > > traffic from the VM is allowed, while Ingress
>>>> traffic is
>>>> >> > > > denied.
>>>> >> > > > > > > Hence,
>>>> >> > > > > > > > > as
>>>> >> > > > > > > > > > you rightly mentioned, security group rules are
>>>> added
>>>> >> > > > > accordingly.
>>>> >> > > > > > > > These
>>>> >> > > > > > > > > > rules get added on the hypervisor host, and you can
>>>> >> verify
>>>> >> > > > them,
>>>> >> > > > > by
>>>> >> > > > > > > > going
>>>> >> > > > > > > > > > into the host and searching for iptables rules
>>>> >> > corresponding
>>>> >> > > to
>>>> >> > > > > the
>>>> >> > > > > > > VM
>>>> >> > > > > > > > > > (internal name - i-x-y-VM).
>>>> >> > > > > > > > > > This blog maybe helpful in providing further
>>>> details:
>>>> >> > > > > > > > > >
>>>> >> > > > > > > > > >
>>>> >> > > > > > > > >
>>>> >> > > > > > > >
>>>> >> > > > > > >
>>>> >> > > > > >
>>>> >> > > > >
>>>> >> > > >
>>>> >> > >
>>>> >> >
>>>> >>
>>>> https://shankerbalan.net/blog/cloudstack-advanced-zone-with-security-groups/
>>>> >> > > > > > > > > >
>>>> >> > > > > > > > > > Thanks,
>>>> >> > > > > > > > > > Pearl
>>>> >> > > > > > > > > > ________________________________
>>>> >> > > > > > > > > > From: Hean Seng <he...@gmail.com>
>>>> >> > > > > > > > > > Sent: Sunday, September 27, 2020 2:48 PM
>>>> >> > > > > > > > > > To: users@cloudstack.apache.org <
>>>> >> > users@cloudstack.apache.org
>>>> >> > > >
>>>> >> > > > > > > > > > Subject: Cloudstack Advance with Security Group
>>>> >> > > > > > > > > >
>>>> >> > > > > > > > > > Hi
>>>> >> > > > > > > > > >
>>>> >> > > > > > > > > > I created advance zone with security group, all
>>>> working
>>>> >> > fine.
>>>> >> > > > > > > > > >
>>>> >> > > > > > > > > > But VMcreated ， seems the default security group
>>>> that
>>>> >> > > assigned
>>>> >> > > > to
>>>> >> > > > > > the
>>>> >> > > > > > > > VM.
>>>> >> > > > > > > > > > all accept policy , i understand is Default Deny,
>>>> and
>>>> >> once
>>>> >> > > add
>>>> >> > > > > in
>>>> >> > > > > > > the
>>>> >> > > > > > > > > port
>>>> >> > > > > > > > > > in Security Group Ingress and Egress, only is
>>>> allowed
>>>> >> > > > > > > > > >
>>>> >> > > > > > > > > > Also, is this rules created at VirtualRouter of the
>>>> >> > > > > SharedNetwork,
>>>> >> > > > > > or
>>>> >> > > > > > > > at
>>>> >> > > > > > > > > > the Hypervisor?
>>>> >> > > > > > > > > >
>>>> >> > > > > > > > > >
>>>> >> > > > > > > > > >
>>>> >> > > > > > > > > > --
>>>> >> > > > > > > > > > Regards,
>>>> >> > > > > > > > > > Hean Seng
>>>> >> > > > > > > > > >
>>>> >> > > > > > > > > > pearl.dsilva@shapeblue.com
>>>> >> > > > > > > > > > www.shapeblue.com<http://www.shapeblue.com>
>>>> >> > > > > > > > > > 3 London Bridge Street, 3rd floor, News Building,
>>>> >> London
>>>> >> > > SE1
>>>> >> > > > > > 9SGUK
>>>> >> > > > > > > > > > @shapeblue
>>>> >> > > > > > > > > >
>>>> >> > > > > > > > > >
>>>> >> > > > > > > > > >
>>>> >> > > > > > > > > >
>>>> >> > > > > > > > >
>>>> >> > > > > > > > > --
>>>> >> > > > > > > > > Regards,
>>>> >> > > > > > > > > Hean Seng
>>>> >> > > > > > > > >
>>>> >> > > > > > > >
>>>> >> > > > > > >
>>>> >> > > > > > >
>>>> >> > > > > > > --
>>>> >> > > > > > > Regards,
>>>> >> > > > > > > Hean Seng
>>>> >> > > > > > >
>>>> >> > > > > >
>>>> >> > > > >
>>>> >> > > > >
>>>> >> > > > > --
>>>> >> > > > > Regards,
>>>> >> > > > > Hean Seng
>>>> >> > > > >
>>>> >> > > >
>>>> >> > >
>>>> >> > >
>>>> >> > > --
>>>> >> > > Regards,
>>>> >> > > Hean Seng
>>>> >> > >
>>>> >> >
>>>> >>
>>>> >>
>>>> >> --
>>>> >> Regards,
>>>> >> Hean Seng
>>>> >>
>>>> >
>>>> >
>>>> > --
>>>> > Regards,
>>>> > Hean Seng
>>>> >
>>>>
>>>>
>>>> --
>>>> Regards,
>>>> Hean Seng
>>>>
>>>
>>>
>>> --
>>> Regards,
>>> Hean Seng
>>>
>>
>>
>> --
>> Regards,
>> Hean Seng
>>
>
>
> --
> Regards,
> Hean Seng
>
--
Regards,
Hean Seng
Re: Cloudstack Advance with Security Group
Posted by Hean Seng <he...@gmail.com>.
Hi
I manage install another Unbuntu 18 HyperVisor on KVM and add to a new
cluster ( ** apparently one customer only can accept all hypervisor with
same OS, thus I have to create new customer for Ubuntu KVM hypervisor)
And the Security Group work fine.
But in CentOS7 , it not working due to error mentioned just now
Will this consider BUG ?
On Tue, Sep 29, 2020 at 6:06 PM Hean Seng <he...@gmail.com> wrote:
> Just for your info, i am installing another host in Ubuntu 18, and just
> defant installation, the script you mention seems normal , as compared to
> CentOS 7.8 .
>
> Now I suspect could be the issue of Cloudstack and CentOS7.8 , or
> many the version of python etc .
>
> I will try to complete the host installation in Ubuntu and adding the
> hypervisor in Mgmt Node ,and try it see if work
>
>
> root@kmv04:~#
> /usr/share/cloudstack-common/scripts/vm/network/security_group.py
>
> usage: security_group.py [-h] [--vmname VMNAME] [--vmip VMIP] [--vmip6
> VMIP6]
>
> [--vmid VMID] [--vmmac VMMAC] [--vif VIF] [--sig
> SIG]
>
> [--seq SEQ] [--rules RULES] [--brname BRNAME]
>
> [--localbrname LOCALBRNAME] [--dhcpSvr DHCPSVR]
>
> [--hostIp HOSTIP] [--hostMacAddr HOSTMACADDR]
>
> [--nicsecips NICSECIPS] [--action ACTION]
>
> [--privnic PRIVNIC] [--isFirstNic] [--check]
>
> command
>
> security_group.py: error: the following arguments are required: command
>
>
>
>
> On Tue, Sep 29, 2020 at 4:12 PM Hean Seng <he...@gmail.com> wrote:
>
>> Hi
>>
>> It seem libvirt-phython already install previously
>>
>>
>> [root@kvm03 agent]# yum install libvirt-python -y
>>
>> Failed to set locale, defaulting to C
>>
>> Loaded plugins: fastestmirror
>>
>> Loading mirror speeds from cached hostfile
>>
>> * base: hk.mirrors.thegigabit.com
>>
>> * epel: hk.mirrors.thegigabit.com
>>
>> * extras: mirror-hk.koddos.net
>>
>> * updates: hk.mirrors.thegigabit.com
>>
>> Package libvirt-python-4.5.0-1.el7.x86_64 already installed and latest
>> version
>>
>> Nothing to do
>>
>> This all installed
>>
>> libvirt-libs-4.5.0-33.el7_8.1.x86_64
>>
>> libvirt-daemon-4.5.0-33.el7_8.1.x86_64
>>
>> libvirt-daemon-config-nwfilter-4.5.0-33.el7_8.1.x86_64
>>
>> libvirt-daemon-driver-storage-rbd-4.5.0-33.el7_8.1.x86_64
>>
>> libvirt-daemon-driver-storage-gluster-4.5.0-33.el7_8.1.x86_64
>>
>> libvirt-daemon-driver-nodedev-4.5.0-33.el7_8.1.x86_64
>>
>> libvirt-client-4.5.0-33.el7_8.1.x86_64
>>
>> libvirt-daemon-driver-storage-core-4.5.0-33.el7_8.1.x86_64
>>
>> libvirt-daemon-driver-nwfilter-4.5.0-33.el7_8.1.x86_64
>>
>> libvirt-daemon-driver-qemu-4.5.0-33.el7_8.1.x86_64
>>
>> libvirt-daemon-driver-lxc-4.5.0-33.el7_8.1.x86_64
>>
>> libvirt-daemon-driver-storage-logical-4.5.0-33.el7_8.1.x86_64
>>
>> libvirt-daemon-driver-storage-disk-4.5.0-33.el7_8.1.x86_64
>>
>> libvirt-daemon-driver-storage-iscsi-4.5.0-33.el7_8.1.x86_64
>>
>> libvirt-daemon-driver-storage-4.5.0-33.el7_8.1.x86_64
>>
>> libvirt-daemon-driver-secret-4.5.0-33.el7_8.1.x86_64
>>
>> libvirt-4.5.0-33.el7_8.1.x86_64
>>
>> libvirt-bash-completion-4.5.0-33.el7_8.1.x86_64
>>
>> libvirt-python-4.5.0-1.el7.x86_64
>>
>> libvirt-daemon-driver-network-4.5.0-33.el7_8.1.x86_64
>>
>> libvirt-daemon-config-network-4.5.0-33.el7_8.1.x86_64
>>
>> libvirt-daemon-driver-storage-scsi-4.5.0-33.el7_8.1.x86_64
>>
>> libvirt-daemon-driver-storage-mpath-4.5.0-33.el7_8.1.x86_64
>>
>> libvirt-daemon-driver-interface-4.5.0-33.el7_8.1.x86_64
>>
>>
>>
>>
>> On Tue, Sep 29, 2020 at 3:46 PM Pearl d'Silva <pe...@shapeblue.com>
>> wrote:
>>
>>> Hi,
>>>
>>> Yes, the script wouldn't do anything, if it's just run without args,
>>> however, it should prompt you with the args to be provided. Based on the
>>> error, it seems that you are missing a python library, hence the script
>>> terminates and that's why the log file isn't generated in the first place.
>>> You can install the package using:
>>> yum install libvirt-python
>>>
>>> Thanks,
>>> Pearl
>>> ________________________________
>>> From: Hean Seng <he...@gmail.com>
>>> Sent: Tuesday, September 29, 2020 1:10 PM
>>> To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
>>> Subject: Re: Cloudstack Advance with Security Group
>>>
>>> The error when running manually:
>>>
>>> ]# /usr/share/cloudstack-common/scripts/vm/network/security_group.py
>>>
>>> Traceback (most recent call last):
>>>
>>> File
>>> "/usr/share/cloudstack-common/scripts/vm/network/security_group.py",
>>> line 26, in <module>
>>>
>>> import libvirt
>>>
>>> Package libvirt-4.5.0-33.el7_8.1.x86_64 already installed and latest
>>> version
>>>
>>>
>>> so I guess, cannot runjust like that ?
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> pearl.dsilva@shapeblue.com
>>> www.shapeblue.com
>>> 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK
>>> @shapeblue
>>>
>>>
>>>
>>> On Tue, Sep 29, 2020 at 3:08 PM Hean Seng <he...@gmail.com> wrote:
>>>
>>> > Hi
>>> >
>>> > I am running on CentOS 7.8.2003
>>> >
>>> > Cloudstack Agent , and KVM hypervisor:
>>> >
>>> > cloudstack-agent-4.14.0.0-1.el7.x86_64
>>> >
>>> > iptables and ebtables aldy install, iptables -L and ebtables -L show
>>> > default rules .
>>> >
>>> > This is dumpxml, seem the bridge is there, the VLAN for Guest is
>>> VLAN401,
>>> > which the setup seem right :
>>> >
>>> > <interface type='bridge'>
>>> >
>>> > <mac address='02:00:21:d9:00:01'/>
>>> >
>>> > <source bridge='brem1-401'/>
>>> >
>>> > <bandwidth>
>>> >
>>> > <inbound average='1280' peak='1280'/>
>>> >
>>> > <outbound average='1280' peak='1280'/>
>>> >
>>> > </bandwidth>
>>> >
>>> > <target dev='vnet9'/>
>>> >
>>> > <model type='virtio'/>
>>> >
>>> > <link state='up'/>
>>> >
>>> > <alias name='net0'/>
>>> >
>>> > <address type='pci' domain='0x0000' bus='0x00' slot='0x03'
>>> > function='0x0'/>
>>> >
>>> >
>>> >
>>> > # ifconfig brem1-401
>>> >
>>> > brem1-401: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
>>> >
>>> > inet6 fe80::3824:b2ff:fe09:873f prefixlen 64 scopeid
>>> 0x20<link>
>>> >
>>> > ether 04:7d:7b:60:09:66 txqueuelen 1000 (Ethernet)
>>> >
>>> > RX packets 7191638 bytes 333185781 (317.7 MiB)
>>> >
>>> > RX errors 0 dropped 0 overruns 0 frame 0
>>> >
>>> > TX packets 8 bytes 656 (656.0 B)
>>> >
>>> > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
>>> >
>>> > brctl show
>>> >
>>> > bridge name bridge id STP enabled interfaces
>>> >
>>> > brem1-401 8000.047d7b600966 no em1.401
>>> >
>>> > vnet10
>>> >
>>> > vnet11
>>> >
>>> > vnet12
>>> >
>>> > vnet14
>>> >
>>> > vnet4
>>> >
>>> > vnet8
>>> >
>>> > vnet9
>>> >
>>> > cloud0 8000.fe00a9fe1cea no vnet13
>>> >
>>> > vnet15
>>> >
>>> > vnet2
>>> >
>>> > vnet6
>>> >
>>> > cloudbr0 8000.047d7b600966 yes em1
>>> >
>>> > vnet3
>>> >
>>> > vnet5
>>> >
>>> > vnet7
>>> >
>>> > cloudbr1 8000.000000000000 yes
>>> >
>>> >
>>> >
>>> > I always have /var/log/cloudstack/agent/ this path , just inside
>>> >
>>> > ls /var/log/cloudstack/agent/
>>> >
>>> > agent.log agent.log.2020-09-26.gz agent.log.2020-09-27.gz
>>> > agent.log.2020-09-28.gz resizevolume.log setup.log
>>> >
>>> > it doesnt have securitygroup folder/ log, all the log ip pasted is at
>>> > agent.log
>>> >
>>> >
>>> >
>>> >
>>> > On Tue, Sep 29, 2020 at 2:44 PM Pearl d'Silva <
>>> pearl.dsilva@shapeblue.com>
>>> > wrote:
>>> >
>>> >> Hi Hean,
>>> >>
>>> >> I was able to deploy a VM in a shared network in an advanced zone with
>>> >> security groups enabled and could see the iptables rules getting
>>> added for
>>> >> the VM on the host, so it is probably not a bug that we are seeing.
>>> Seems
>>> >> like a setup issue.
>>> >> Applying the rules can fail because:
>>> >>
>>> >> * The hypervisor host doesn't have iptables / ebtables command -
>>> >> which is probably unlikely
>>> >> * The VM deployed doesn't have an interface - can be verified by
>>> >> dumping its configuration - virsh dumpxml <domain_name> | grep
>>> interface
>>> >> * or application of the default network rules for the VM itself
>>> >> failed -If you are up for some debugging, can you try running the
>>> script
>>> >> manually in the hypervisor host:
>>> >>
>>> >> /usr/share/cloudstack-common/scripts/vm/network/security_group.py
>>> >> default_network_rules --vmname <domain_name> --vmid <vm_id> --vmip
>>> <vm_ip>
>>> >> --vmmac <vm_mac> --vif <device/networkname> --brname <bridge_name>
>>> >> --nicsecips 0; --isFirstNic --check
>>> >>
>>> >> when you do a virsh dumpxml <vm_domain_name>, look out for an
>>> interface
>>> >> section that looks something like:
>>> >>
>>> >> <interface type='bridge'>
>>> >> <mac address='1e:00:86:00:00:ba'/>
>>> >> <source bridge='breth1-11'/>
>>> >> <bandwidth>
>>> >> <inbound average='25600' peak='25600'/>
>>> >> <outbound average='25600' peak='25600'/>
>>> >> </bandwidth>
>>> >> <target dev='vnet0'/>
>>> >> <model type='virtio'/>
>>> >> <link state='up'/>
>>> >> <alias name='net0'/>
>>> >> <address type='pci' domain='0x0000' bus='0x00' slot='0x03'
>>> >> function='0x0'/>
>>> >> </interface>
>>> >>
>>> >>
>>> >> the inputs to the script can be found as follows:
>>> >> domain_name - internal name of your vm (i-x-y-VM); can also be
>>> obtained
>>> >> by "virsh list"
>>> >> vm_id - the 'y' part of the internal name : i-x-y-VM
>>> >> vm_ip - Now that debug logs have been enabled, you can search for
>>> >> "StartCommand" in your logs get the ip from the 'nics' section or you
>>> could
>>> >> choose any ip from the shared network range
>>> >> vm_mac - from the above bridge section, you will be able to get the
>>> MAC
>>> >> address
>>> >> vif / device name - taget dev name - in the above case - vnet0
>>> >> bridge_name - again can be obtained from the above snippet (eg,
>>> breth1-11)
>>> >> if there are no secondary ips, then pass 0 for nicsecips otherwise
>>> >> specify the seconday ips separated by a semicolon (;)
>>> >>
>>> >> Basically, at the end of it, it should look something like:
>>> >> /usr/share/cloudstack-common/scripts/vm/network/security_group.py
>>> >> default_network_rules --vmname i-2-3-VM --vmid 3 --vmip <ip> --vmmac
>>> >> 1e:00:86:00:00:ba --vif vnet0 --brname breth1-11 --nicsecips 0;
>>> --isFirstNic
>>> >>
>>> >> Hopefully, now you should be able to see a security_group.log file in
>>> the
>>> >> /var/log/cloudstack/agent/ path.
>>> >>
>>> >> Thanks,
>>> >> Pearl
>>> >>
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> ________________________________
>>> >> From: Hean Seng <he...@gmail.com>
>>> >> Sent: Tuesday, September 29, 2020 10:45 AM
>>> >> To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
>>> >> Subject: Re: Cloudstack Advance with Security Group
>>> >>
>>> >> Hi Pearl,
>>> >>
>>> >> I had change to debug, following is the error captured:
>>> >>
>>> >>
>>> >> 2020-09-29 01:13:30,090 DEBUG [cloud.agent.Agent]
>>> >> (agentRequest-Handler-2:null) (logid:388b92e0) Processing command:
>>> >> com.cloud.agent.api.SecurityGroupRulesCmd
>>> >>
>>> >> 2020-09-29 01:13:30,090 DEBUG [kvm.resource.LibvirtConnection]
>>> >> (agentRequest-Handler-2:null) (logid:388b92e0) Looking for libvirtd
>>> >> connection at: qemu:///system
>>> >>
>>> >> 2020-09-29 01:13:30,094 DEBUG [kvm.resource.LibvirtComputingResource]
>>> >> (agentRequest-Handler-2:null) (logid:388b92e0) Checking default
>>> network
>>> >> rules for vm i-23-77-VM
>>> >>
>>> >> 2020-09-29 01:13:30,094 ERROR [kvm.resource.LibvirtComputingResource]
>>> >> (agentRequest-Handler-2:null) (logid:388b92e0) Unable to apply default
>>> >> network rule for nic cloudbr0 for VM i-23-77-VM
>>> >>
>>> >> 2020-09-29 01:13:30,094 WARN
>>> >> [resource.wrapper.LibvirtSecurityGroupRulesCommandWrapper]
>>> >> (agentRequest-Handler-2:null) (logid:388b92e0) Failed to program
>>> default
>>> >> network rules for vm i-23-77-VM
>>> >>
>>> >> 2020-09-29 01:13:30,095 DEBUG [cloud.agent.Agent]
>>> >> (agentRequest-Handler-2:null) (logid:388b92e0) Seq
>>> >> 11-6057904448766738456: {
>>> >> Ans: , MgmtId: 72752492851638, via: 11, Ver: v1, Flags: 110,
>>> >>
>>> >>
>>> [{"com.cloud.agent.api.SecurityGroupRuleAnswer":{"logSequenceNumber":15,"vmId":77,"reason":"PROGRAMMING_FAILED","result":false,"details":"programming
>>> >> default network rules failed","wait":0}}] }
>>> >>
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> On Tue, Sep 29, 2020 at 12:38 PM Pearl d'Silva <
>>> >> pearl.dsilva@shapeblue.com>
>>> >> wrote:
>>> >>
>>> >> > Hi Hean,
>>> >> >
>>> >> >
>>> >> > Could you try enabling debug logging on your hypervisor hosts, by
>>> >> running
>>> >> > the following:
>>> >> > *sed -i 's/INFO/DEBUG/g' /etc/cloudstack/agent/log4j-cloud.xml*
>>> >> >
>>> >> > And then restart the cloudstack-agent service on the hosts. Post
>>> this,
>>> >> try
>>> >> > deploying a VM in the shared network. This may give a better insight
>>> >> into
>>> >> > where exactly the issue lies.
>>> >> >
>>> >> > Thanks,
>>> >> > Pearl
>>> >> >
>>> >> >
>>> >> > Software Engineer
>>> >> > pearl.dsilva@shapeblue.com
>>> >> > www.shapeblue.com<http://www.shapeblue.com>
>>> >> >
>>> >> >
>>> >> >
>>> >> >
>>> >> >
>>> >> >
>>> >> > ------------------------------
>>> >> > *From:* Ivan Kudryavtsev <iv...@bw-sw.com>
>>> >> > *Sent:* Monday, September 28, 2020 2:17 PM
>>> >> > *To:* users <us...@cloudstack.apache.org>
>>> >> > *Subject:* Re: Cloudstack Advance with Security Group
>>> >> >
>>> >> > This is it.
>>> >> >
>>> >>
>>> >> pearl.dsilva@shapeblue.com
>>> >> www.shapeblue.com<http://www.shapeblue.com>
>>> >> 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK
>>> >> @shapeblue
>>> >>
>>> >>
>>> >>
>>> >> > On Mon, Sep 28, 2020 at 3:46 PM Hean Seng <he...@gmail.com>
>>> wrote:
>>> >> >
>>> >> > > In the log, I cannot see anything much, except a few lines
>>> showing
>>> >> above
>>> >> > > . Not sure if this is a bug on 4.14.
>>> >> > >
>>> >> > >
>>> >> > >
>>> >> > > 2020-09-28 03:53:36,585 ERROR
>>> [kvm.resource.LibvirtComputingResource]
>>> >> > > (agentRequest-Handler-4:null) (logid:b6a4c077) Unable to apply
>>> default
>>> >> > > network rule for nic cloudbr0 for VM i-2-81-VM
>>> >> > >
>>> >> > > 2020-09-28 03:53:36,858 ERROR
>>> [kvm.resource.LibvirtComputingResource]
>>> >> > > (agentRequest-Handler-3:null) (logid:74058678) Unable to apply
>>> default
>>> >> > > network rule for nic cloudbr0 for VM i-2-81-VM
>>> >> > >
>>> >> > > 2020-09-28 03:53:36,858 WARN
>>> >> > > [resource.wrapper.LibvirtSecurityGroupRulesCommandWrapper]
>>> >> > > (agentRequest-Handler-3:null) (logid:74058678) Failed to program
>>> >> default
>>> >> > > network rules for vm i-2-81-VM
>>> >> > >
>>> >> > >
>>> >> > >
>>> >> > >
>>> >> > > On Mon, Sep 28, 2020 at 4:34 PM Ivan Kudryavtsev <iv...@bw-sw.com>
>>> >> wrote:
>>> >> > >
>>> >> > > > Hi,
>>> >> > > > no I'm on 4.11, so can not help with exact 4.14, and I'm on
>>> Ubuntu,
>>> >> > > though,
>>> >> > > > but for any KVM hypervisor Linux distribution, the logic is the
>>> >> same.
>>> >> > > >
>>> >> > > > On Mon, Sep 28, 2020 at 3:31 PM Hean Seng <he...@gmail.com>
>>> >> wrote:
>>> >> > > >
>>> >> > > > > Hi
>>> >> > > > >
>>> >> > > > > Are you running on CentOS7 ?
>>> >> > > > >
>>> >> > > > > I am running on CentOS7 , ACS 4.14 , and seem there is no
>>> log
>>> >> at
>>> >> > of
>>> >> > > > > security_group.log
>>> >> > > > >
>>> >> > > > > # ls /var/log/cloudstack/agent/
>>> >> > > > >
>>> >> > > > > agent.log resizevolume.log setup.log
>>> >> > > > >
>>> >> > > > >
>>> >> > > > > I recheck back the Intall guide, seems no missing anything.
>>> >> > > > >
>>> >> > > > >
>>> >> > > > > Older intallation guide, 4.11 mentioned need , allow
>>> >> > > > > /usr/lib/sysctl.d/00-system.conf
>>> >> > > > >
>>> >> > > > > # Enable netfilter on bridges.
>>> >> net.bridge.bridge-nf-call-ip6tables =
>>> >> > 1
>>> >> > > > > net.bridge.bridge-nf-call-iptables = 1
>>> >> > > > net.bridge.bridge-nf-call-arptables
>>> >> > > > > = 1
>>> >> > > > >
>>> >> > > > > And it has been done too.
>>> >> > > > >
>>> >> > > > >
>>> >> > > > >
>>> >> > > > > On Mon, Sep 28, 2020 at 4:05 PM Ivan Kudryavtsev <
>>> ivan@bw-sw.com>
>>> >> > > wrote:
>>> >> > > > >
>>> >> > > > > > Hi,
>>> >> > > > > >
>>> >> > > > > > No, this is not the issue.
>>> >> > > > > > It's a normal state of the system, as KVM hooks are a new
>>> and
>>> >> > > optional
>>> >> > > > > > feature of 4.14.
>>> >> > > > > >
>>> >> > > > > > You should find some sort of messages regarding
>>> security_groups
>>> >> at
>>> >> > > > > > /var/log/cloudstack/agent/security_group.log
>>> >> > > > > >
>>> >> > > > > >
>>> >> > > > > > On Mon, Sep 28, 2020 at 2:10 PM Hean Seng <
>>> heanseng@gmail.com>
>>> >> > > wrote:
>>> >> > > > > >
>>> >> > > > > > > I not sure where goes wrong, are you running on CentOS 7
>>> ? I
>>> >> > have
>>> >> > > > this
>>> >> > > > > > > error too, do you think is this contribute to the error as
>>> >> well:
>>> >> > > > > > >
>>> >> > > > > > > 2020-09-28 03:04:52,762 WARN
>>> >> [kvm.resource.LibvirtKvmAgentHook]
>>> >> > > > > > > (agentRequest-Handler-5:null) (logid:4f23845b) Groovy
>>> script
>>> >> > > > > > >
>>> >> '/etc/cloudstack/agent/hooks/libvirt-vm-xml-transformer.groovy'
>>> >> > is
>>> >> > > > not
>>> >> > > > > > > available. Transformations will not be applied.
>>> >> > > > > > >
>>> >> > > > > > > 2020-09-28 03:04:52,762 WARN
>>> >> [kvm.resource.LibvirtKvmAgentHook]
>>> >> > > > > > > (agentRequest-Handler-5:null) (logid:4f23845b) Groovy
>>> >> scripting
>>> >> > > > engine
>>> >> > > > > is
>>> >> > > > > > > not initialized. Data transformation skipped.
>>> >> > > > > > >
>>> >> > > > > > > 2020-09-28 03:04:53,083 WARN
>>> >> [kvm.resource.LibvirtKvmAgentHook]
>>> >> > > > > > > (agentRequest-Handler-5:null) (logid:4f23845b) Groovy
>>> script
>>> >> > > > > > >
>>> '/etc/cloudstack/agent/hooks/libvirt-vm-state-change.groovy'
>>> >> is
>>> >> > not
>>> >> > > > > > > available. Transformations will not be applied.
>>> >> > > > > > >
>>> >> > > > > > > On Mon, Sep 28, 2020 at 2:27 PM Ivan Kudryavtsev <
>>> >> ivan@bw-sw.com
>>> >> > >
>>> >> > > > > wrote:
>>> >> > > > > > >
>>> >> > > > > > > > This just means you installed it in the wrong way.
>>> Ebtables
>>> >> and
>>> >> > > > > > Iptables
>>> >> > > > > > > > must be filled with rules like
>>> >> > > > > > > >
>>> >> > > > > > > > -A i-6242-10304-def -m state --state
>>> RELATED,ESTABLISHED -j
>>> >> > > ACCEPT
>>> >> > > > > > > > -A i-6242-10304-def -p udp -m physdev --physdev-in
>>> vnet18
>>> >> > > > > > > > --physdev-is-bridged -m udp --sport 68 --dport 67 -j
>>> ACCEPT
>>> >> > > > > > > > -A i-6242-10304-def -p udp -m physdev --physdev-out
>>> vnet18
>>> >> > > > > > > > --physdev-is-bridged -m udp --sport 67 --dport 68 -j
>>> ACCEPT
>>> >> > > > > > > > -A i-6242-10304-def -m physdev --physdev-in vnet18
>>> >> > > > > --physdev-is-bridged
>>> >> > > > > > > -m
>>> >> > > > > > > > set ! --match-set i-6242-10304-vm src -j DROP
>>> >> > > > > > > > -A i-6242-10304-def -p udp -m physdev --physdev-in
>>> vnet18
>>> >> > > > > > > > --physdev-is-bridged -m set --match-set i-6242-10304-vm
>>> src
>>> >> -m
>>> >> > > udp
>>> >> > > > > > > --dport
>>> >> > > > > > > > 53 -j RETURN
>>> >> > > > > > > > -A i-6242-10304-def -p tcp -m physdev --physdev-in
>>> vnet18
>>> >> > > > > > > > --physdev-is-bridged -m set --match-set i-6242-10304-vm
>>> src
>>> >> -m
>>> >> > > tcp
>>> >> > > > > > > --dport
>>> >> > > > > > > > 53 -j RETURN
>>> >> > > > > > > > -A i-6242-10304-def -m physdev --physdev-in vnet18
>>> >> > > > > --physdev-is-bridged
>>> >> > > > > > > -m
>>> >> > > > > > > > set --match-set i-6242-10304-vm src -j
>>> i-6242-10304-vm-eg
>>> >> > > > > > > > -A i-6242-10304-def -m physdev --physdev-out vnet18
>>> >> > > > > > --physdev-is-bridged
>>> >> > > > > > > -j
>>> >> > > > > > > > i-6242-10304-vm
>>> >> > > > > > > > -A i-6242-10304-vm -p udp -m udp --dport 1:65535 -m
>>> state
>>> >> > --state
>>> >> > > > NEW
>>> >> > > > > > -j
>>> >> > > > > > > > ACCEPT
>>> >> > > > > > > > -A i-6242-10304-vm -p tcp -m tcp --dport 1:65535 -m
>>> state
>>> >> > --state
>>> >> > > > NEW
>>> >> > > > > > -j
>>> >> > > > > > > > ACCEPT
>>> >> > > > > > > > -A i-6242-10304-vm -p icmp -m icmp --icmp-type any -j
>>> ACCEPT
>>> >> > > > > > > > -A i-6242-10304-vm -j DROP
>>> >> > > > > > > >
>>> >> > > > > > > >
>>> >> > > > > > > > Bridge chain: i-4435-8929-vm-in, entries: 7, policy:
>>> ACCEPT
>>> >> > > > > > > > -s ! 1e:0:32:0:2:2 -j DROP
>>> >> > > > > > > > -p ARP -s ! 1e:0:32:0:2:2 -j DROP
>>> >> > > > > > > > -p ARP --arp-mac-src ! 1e:0:32:0:2:2 -j DROP
>>> >> > > > > > > > -p ARP -j i-4435-8929-vm-in-ips
>>> >> > > > > > > > -p ARP --arp-op Request -j ACCEPT
>>> >> > > > > > > > -p ARP --arp-op Reply -j ACCEPT
>>> >> > > > > > > > -p ARP -j DROP
>>> >> > > > > > > >
>>> >> > > > > > > >
>>> >> > > > > > > >
>>> >> > > > > > > > On Mon, Sep 28, 2020 at 1:10 PM Hean Seng <
>>> >> heanseng@gmail.com>
>>> >> > > > > wrote:
>>> >> > > > > > > >
>>> >> > > > > > > > > I checked the hypervisor , it seems iptables is
>>> nothing
>>> >> > inside
>>> >> > > ,
>>> >> > > > > > this
>>> >> > > > > > > is
>>> >> > > > > > > > > centos7 , initially i turnoff firewalld , but even i
>>> >> turn
>>> >> > on
>>> >> > > it
>>> >> > > > > now
>>> >> > > > > > > and
>>> >> > > > > > > > > try to update the security group rules, it seems empty
>>> >> > iptable
>>> >> > > > > rules
>>> >> > > > > > :
>>> >> > > > > > > > >
>>> >> > > > > > > > > [root@kvm03 ~]# iptables -L -v -n
>>> >> > > > > > > > >
>>> >> > > > > > > > > Chain INPUT (policy ACCEPT 82903 packets, 1170M bytes)
>>> >> > > > > > > > >
>>> >> > > > > > > > > pkts bytes target prot opt in out source
>>> >> > > > > > > > > destination
>>> >> > > > > > > > >
>>> >> > > > > > > > >
>>> >> > > > > > > > > Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
>>> >> > > > > > > > >
>>> >> > > > > > > > > pkts bytes target prot opt in out source
>>> >> > > > > > > > > destination
>>> >> > > > > > > > >
>>> >> > > > > > > > >
>>> >> > > > > > > > > Chain OUTPUT (policy ACCEPT 80505 packets, 25M bytes)
>>> >> > > > > > > > >
>>> >> > > > > > > > > pkts bytes target prot opt in out source
>>> >> > > > > > > > > destination
>>> >> > > > > > > > >
>>> >> > > > > > > > >
>>> >> > > > > > > > >
>>> >> > > > > > > > >
>>> >> > > > > > > > >
>>> >> > > > > > > > >
>>> >> > > > > > > > >
>>> >> > > > > > > > > On Mon, Sep 28, 2020 at 12:05 PM Pearl d'Silva <
>>> >> > > > > > > > pearl.dsilva@shapeblue.com
>>> >> > > > > > > > > >
>>> >> > > > > > > > > wrote:
>>> >> > > > > > > > >
>>> >> > > > > > > > > > Hi Hean,
>>> >> > > > > > > > > >
>>> >> > > > > > > > > > In an Advanced Zone with Security Groups enabled, by
>>> >> > default,
>>> >> > > > > > egress
>>> >> > > > > > > > > > traffic from the VM is allowed, while Ingress
>>> traffic is
>>> >> > > > denied.
>>> >> > > > > > > Hence,
>>> >> > > > > > > > > as
>>> >> > > > > > > > > > you rightly mentioned, security group rules are
>>> added
>>> >> > > > > accordingly.
>>> >> > > > > > > > These
>>> >> > > > > > > > > > rules get added on the hypervisor host, and you can
>>> >> verify
>>> >> > > > them,
>>> >> > > > > by
>>> >> > > > > > > > going
>>> >> > > > > > > > > > into the host and searching for iptables rules
>>> >> > corresponding
>>> >> > > to
>>> >> > > > > the
>>> >> > > > > > > VM
>>> >> > > > > > > > > > (internal name - i-x-y-VM).
>>> >> > > > > > > > > > This blog maybe helpful in providing further
>>> details:
>>> >> > > > > > > > > >
>>> >> > > > > > > > > >
>>> >> > > > > > > > >
>>> >> > > > > > > >
>>> >> > > > > > >
>>> >> > > > > >
>>> >> > > > >
>>> >> > > >
>>> >> > >
>>> >> >
>>> >>
>>> https://shankerbalan.net/blog/cloudstack-advanced-zone-with-security-groups/
>>> >> > > > > > > > > >
>>> >> > > > > > > > > > Thanks,
>>> >> > > > > > > > > > Pearl
>>> >> > > > > > > > > > ________________________________
>>> >> > > > > > > > > > From: Hean Seng <he...@gmail.com>
>>> >> > > > > > > > > > Sent: Sunday, September 27, 2020 2:48 PM
>>> >> > > > > > > > > > To: users@cloudstack.apache.org <
>>> >> > users@cloudstack.apache.org
>>> >> > > >
>>> >> > > > > > > > > > Subject: Cloudstack Advance with Security Group
>>> >> > > > > > > > > >
>>> >> > > > > > > > > > Hi
>>> >> > > > > > > > > >
>>> >> > > > > > > > > > I created advance zone with security group, all
>>> working
>>> >> > fine.
>>> >> > > > > > > > > >
>>> >> > > > > > > > > > But VMcreated ， seems the default security group
>>> that
>>> >> > > assigned
>>> >> > > > to
>>> >> > > > > > the
>>> >> > > > > > > > VM.
>>> >> > > > > > > > > > all accept policy , i understand is Default Deny,
>>> and
>>> >> once
>>> >> > > add
>>> >> > > > > in
>>> >> > > > > > > the
>>> >> > > > > > > > > port
>>> >> > > > > > > > > > in Security Group Ingress and Egress, only is
>>> allowed
>>> >> > > > > > > > > >
>>> >> > > > > > > > > > Also, is this rules created at VirtualRouter of the
>>> >> > > > > SharedNetwork,
>>> >> > > > > > or
>>> >> > > > > > > > at
>>> >> > > > > > > > > > the Hypervisor?
>>> >> > > > > > > > > >
>>> >> > > > > > > > > >
>>> >> > > > > > > > > >
>>> >> > > > > > > > > > --
>>> >> > > > > > > > > > Regards,
>>> >> > > > > > > > > > Hean Seng
>>> >> > > > > > > > > >
>>> >> > > > > > > > > > pearl.dsilva@shapeblue.com
>>> >> > > > > > > > > > www.shapeblue.com<http://www.shapeblue.com>
>>> >> > > > > > > > > > 3 London Bridge Street, 3rd floor, News Building,
>>> >> London
>>> >> > > SE1
>>> >> > > > > > 9SGUK
>>> >> > > > > > > > > > @shapeblue
>>> >> > > > > > > > > >
>>> >> > > > > > > > > >
>>> >> > > > > > > > > >
>>> >> > > > > > > > > >
>>> >> > > > > > > > >
>>> >> > > > > > > > > --
>>> >> > > > > > > > > Regards,
>>> >> > > > > > > > > Hean Seng
>>> >> > > > > > > > >
>>> >> > > > > > > >
>>> >> > > > > > >
>>> >> > > > > > >
>>> >> > > > > > > --
>>> >> > > > > > > Regards,
>>> >> > > > > > > Hean Seng
>>> >> > > > > > >
>>> >> > > > > >
>>> >> > > > >
>>> >> > > > >
>>> >> > > > > --
>>> >> > > > > Regards,
>>> >> > > > > Hean Seng
>>> >> > > > >
>>> >> > > >
>>> >> > >
>>> >> > >
>>> >> > > --
>>> >> > > Regards,
>>> >> > > Hean Seng
>>> >> > >
>>> >> >
>>> >>
>>> >>
>>> >> --
>>> >> Regards,
>>> >> Hean Seng
>>> >>
>>> >
>>> >
>>> > --
>>> > Regards,
>>> > Hean Seng
>>> >
>>>
>>>
>>> --
>>> Regards,
>>> Hean Seng
>>>
>>
>>
>> --
>> Regards,
>> Hean Seng
>>
>
>
> --
> Regards,
> Hean Seng
>
--
Regards,
Hean Seng
Re: Cloudstack Advance with Security Group
Posted by Hean Seng <he...@gmail.com>.
Just for your info, i am installing another host in Ubuntu 18, and just
defant installation, the script you mention seems normal , as compared to
CentOS 7.8 .
Now I suspect could be the issue of Cloudstack and CentOS7.8 , or many the
version of python etc .
I will try to complete the host installation in Ubuntu and adding the
hypervisor in Mgmt Node ,and try it see if work
root@kmv04:~#
/usr/share/cloudstack-common/scripts/vm/network/security_group.py
usage: security_group.py [-h] [--vmname VMNAME] [--vmip VMIP] [--vmip6
VMIP6]
[--vmid VMID] [--vmmac VMMAC] [--vif VIF] [--sig
SIG]
[--seq SEQ] [--rules RULES] [--brname BRNAME]
[--localbrname LOCALBRNAME] [--dhcpSvr DHCPSVR]
[--hostIp HOSTIP] [--hostMacAddr HOSTMACADDR]
[--nicsecips NICSECIPS] [--action ACTION]
[--privnic PRIVNIC] [--isFirstNic] [--check]
command
security_group.py: error: the following arguments are required: command
On Tue, Sep 29, 2020 at 4:12 PM Hean Seng <he...@gmail.com> wrote:
> Hi
>
> It seem libvirt-phython already install previously
>
>
> [root@kvm03 agent]# yum install libvirt-python -y
>
> Failed to set locale, defaulting to C
>
> Loaded plugins: fastestmirror
>
> Loading mirror speeds from cached hostfile
>
> * base: hk.mirrors.thegigabit.com
>
> * epel: hk.mirrors.thegigabit.com
>
> * extras: mirror-hk.koddos.net
>
> * updates: hk.mirrors.thegigabit.com
>
> Package libvirt-python-4.5.0-1.el7.x86_64 already installed and latest
> version
>
> Nothing to do
>
> This all installed
>
> libvirt-libs-4.5.0-33.el7_8.1.x86_64
>
> libvirt-daemon-4.5.0-33.el7_8.1.x86_64
>
> libvirt-daemon-config-nwfilter-4.5.0-33.el7_8.1.x86_64
>
> libvirt-daemon-driver-storage-rbd-4.5.0-33.el7_8.1.x86_64
>
> libvirt-daemon-driver-storage-gluster-4.5.0-33.el7_8.1.x86_64
>
> libvirt-daemon-driver-nodedev-4.5.0-33.el7_8.1.x86_64
>
> libvirt-client-4.5.0-33.el7_8.1.x86_64
>
> libvirt-daemon-driver-storage-core-4.5.0-33.el7_8.1.x86_64
>
> libvirt-daemon-driver-nwfilter-4.5.0-33.el7_8.1.x86_64
>
> libvirt-daemon-driver-qemu-4.5.0-33.el7_8.1.x86_64
>
> libvirt-daemon-driver-lxc-4.5.0-33.el7_8.1.x86_64
>
> libvirt-daemon-driver-storage-logical-4.5.0-33.el7_8.1.x86_64
>
> libvirt-daemon-driver-storage-disk-4.5.0-33.el7_8.1.x86_64
>
> libvirt-daemon-driver-storage-iscsi-4.5.0-33.el7_8.1.x86_64
>
> libvirt-daemon-driver-storage-4.5.0-33.el7_8.1.x86_64
>
> libvirt-daemon-driver-secret-4.5.0-33.el7_8.1.x86_64
>
> libvirt-4.5.0-33.el7_8.1.x86_64
>
> libvirt-bash-completion-4.5.0-33.el7_8.1.x86_64
>
> libvirt-python-4.5.0-1.el7.x86_64
>
> libvirt-daemon-driver-network-4.5.0-33.el7_8.1.x86_64
>
> libvirt-daemon-config-network-4.5.0-33.el7_8.1.x86_64
>
> libvirt-daemon-driver-storage-scsi-4.5.0-33.el7_8.1.x86_64
>
> libvirt-daemon-driver-storage-mpath-4.5.0-33.el7_8.1.x86_64
>
> libvirt-daemon-driver-interface-4.5.0-33.el7_8.1.x86_64
>
>
>
>
> On Tue, Sep 29, 2020 at 3:46 PM Pearl d'Silva <pe...@shapeblue.com>
> wrote:
>
>> Hi,
>>
>> Yes, the script wouldn't do anything, if it's just run without args,
>> however, it should prompt you with the args to be provided. Based on the
>> error, it seems that you are missing a python library, hence the script
>> terminates and that's why the log file isn't generated in the first place.
>> You can install the package using:
>> yum install libvirt-python
>>
>> Thanks,
>> Pearl
>> ________________________________
>> From: Hean Seng <he...@gmail.com>
>> Sent: Tuesday, September 29, 2020 1:10 PM
>> To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
>> Subject: Re: Cloudstack Advance with Security Group
>>
>> The error when running manually:
>>
>> ]# /usr/share/cloudstack-common/scripts/vm/network/security_group.py
>>
>> Traceback (most recent call last):
>>
>> File
>> "/usr/share/cloudstack-common/scripts/vm/network/security_group.py",
>> line 26, in <module>
>>
>> import libvirt
>>
>> Package libvirt-4.5.0-33.el7_8.1.x86_64 already installed and latest
>> version
>>
>>
>> so I guess, cannot runjust like that ?
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> pearl.dsilva@shapeblue.com
>> www.shapeblue.com
>> 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK
>> @shapeblue
>>
>>
>>
>> On Tue, Sep 29, 2020 at 3:08 PM Hean Seng <he...@gmail.com> wrote:
>>
>> > Hi
>> >
>> > I am running on CentOS 7.8.2003
>> >
>> > Cloudstack Agent , and KVM hypervisor:
>> >
>> > cloudstack-agent-4.14.0.0-1.el7.x86_64
>> >
>> > iptables and ebtables aldy install, iptables -L and ebtables -L show
>> > default rules .
>> >
>> > This is dumpxml, seem the bridge is there, the VLAN for Guest is
>> VLAN401,
>> > which the setup seem right :
>> >
>> > <interface type='bridge'>
>> >
>> > <mac address='02:00:21:d9:00:01'/>
>> >
>> > <source bridge='brem1-401'/>
>> >
>> > <bandwidth>
>> >
>> > <inbound average='1280' peak='1280'/>
>> >
>> > <outbound average='1280' peak='1280'/>
>> >
>> > </bandwidth>
>> >
>> > <target dev='vnet9'/>
>> >
>> > <model type='virtio'/>
>> >
>> > <link state='up'/>
>> >
>> > <alias name='net0'/>
>> >
>> > <address type='pci' domain='0x0000' bus='0x00' slot='0x03'
>> > function='0x0'/>
>> >
>> >
>> >
>> > # ifconfig brem1-401
>> >
>> > brem1-401: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
>> >
>> > inet6 fe80::3824:b2ff:fe09:873f prefixlen 64 scopeid
>> 0x20<link>
>> >
>> > ether 04:7d:7b:60:09:66 txqueuelen 1000 (Ethernet)
>> >
>> > RX packets 7191638 bytes 333185781 (317.7 MiB)
>> >
>> > RX errors 0 dropped 0 overruns 0 frame 0
>> >
>> > TX packets 8 bytes 656 (656.0 B)
>> >
>> > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
>> >
>> > brctl show
>> >
>> > bridge name bridge id STP enabled interfaces
>> >
>> > brem1-401 8000.047d7b600966 no em1.401
>> >
>> > vnet10
>> >
>> > vnet11
>> >
>> > vnet12
>> >
>> > vnet14
>> >
>> > vnet4
>> >
>> > vnet8
>> >
>> > vnet9
>> >
>> > cloud0 8000.fe00a9fe1cea no vnet13
>> >
>> > vnet15
>> >
>> > vnet2
>> >
>> > vnet6
>> >
>> > cloudbr0 8000.047d7b600966 yes em1
>> >
>> > vnet3
>> >
>> > vnet5
>> >
>> > vnet7
>> >
>> > cloudbr1 8000.000000000000 yes
>> >
>> >
>> >
>> > I always have /var/log/cloudstack/agent/ this path , just inside
>> >
>> > ls /var/log/cloudstack/agent/
>> >
>> > agent.log agent.log.2020-09-26.gz agent.log.2020-09-27.gz
>> > agent.log.2020-09-28.gz resizevolume.log setup.log
>> >
>> > it doesnt have securitygroup folder/ log, all the log ip pasted is at
>> > agent.log
>> >
>> >
>> >
>> >
>> > On Tue, Sep 29, 2020 at 2:44 PM Pearl d'Silva <
>> pearl.dsilva@shapeblue.com>
>> > wrote:
>> >
>> >> Hi Hean,
>> >>
>> >> I was able to deploy a VM in a shared network in an advanced zone with
>> >> security groups enabled and could see the iptables rules getting added
>> for
>> >> the VM on the host, so it is probably not a bug that we are seeing.
>> Seems
>> >> like a setup issue.
>> >> Applying the rules can fail because:
>> >>
>> >> * The hypervisor host doesn't have iptables / ebtables command -
>> >> which is probably unlikely
>> >> * The VM deployed doesn't have an interface - can be verified by
>> >> dumping its configuration - virsh dumpxml <domain_name> | grep
>> interface
>> >> * or application of the default network rules for the VM itself
>> >> failed -If you are up for some debugging, can you try running the
>> script
>> >> manually in the hypervisor host:
>> >>
>> >> /usr/share/cloudstack-common/scripts/vm/network/security_group.py
>> >> default_network_rules --vmname <domain_name> --vmid <vm_id> --vmip
>> <vm_ip>
>> >> --vmmac <vm_mac> --vif <device/networkname> --brname <bridge_name>
>> >> --nicsecips 0; --isFirstNic --check
>> >>
>> >> when you do a virsh dumpxml <vm_domain_name>, look out for an interface
>> >> section that looks something like:
>> >>
>> >> <interface type='bridge'>
>> >> <mac address='1e:00:86:00:00:ba'/>
>> >> <source bridge='breth1-11'/>
>> >> <bandwidth>
>> >> <inbound average='25600' peak='25600'/>
>> >> <outbound average='25600' peak='25600'/>
>> >> </bandwidth>
>> >> <target dev='vnet0'/>
>> >> <model type='virtio'/>
>> >> <link state='up'/>
>> >> <alias name='net0'/>
>> >> <address type='pci' domain='0x0000' bus='0x00' slot='0x03'
>> >> function='0x0'/>
>> >> </interface>
>> >>
>> >>
>> >> the inputs to the script can be found as follows:
>> >> domain_name - internal name of your vm (i-x-y-VM); can also be obtained
>> >> by "virsh list"
>> >> vm_id - the 'y' part of the internal name : i-x-y-VM
>> >> vm_ip - Now that debug logs have been enabled, you can search for
>> >> "StartCommand" in your logs get the ip from the 'nics' section or you
>> could
>> >> choose any ip from the shared network range
>> >> vm_mac - from the above bridge section, you will be able to get the MAC
>> >> address
>> >> vif / device name - taget dev name - in the above case - vnet0
>> >> bridge_name - again can be obtained from the above snippet (eg,
>> breth1-11)
>> >> if there are no secondary ips, then pass 0 for nicsecips otherwise
>> >> specify the seconday ips separated by a semicolon (;)
>> >>
>> >> Basically, at the end of it, it should look something like:
>> >> /usr/share/cloudstack-common/scripts/vm/network/security_group.py
>> >> default_network_rules --vmname i-2-3-VM --vmid 3 --vmip <ip> --vmmac
>> >> 1e:00:86:00:00:ba --vif vnet0 --brname breth1-11 --nicsecips 0;
>> --isFirstNic
>> >>
>> >> Hopefully, now you should be able to see a security_group.log file in
>> the
>> >> /var/log/cloudstack/agent/ path.
>> >>
>> >> Thanks,
>> >> Pearl
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> ________________________________
>> >> From: Hean Seng <he...@gmail.com>
>> >> Sent: Tuesday, September 29, 2020 10:45 AM
>> >> To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
>> >> Subject: Re: Cloudstack Advance with Security Group
>> >>
>> >> Hi Pearl,
>> >>
>> >> I had change to debug, following is the error captured:
>> >>
>> >>
>> >> 2020-09-29 01:13:30,090 DEBUG [cloud.agent.Agent]
>> >> (agentRequest-Handler-2:null) (logid:388b92e0) Processing command:
>> >> com.cloud.agent.api.SecurityGroupRulesCmd
>> >>
>> >> 2020-09-29 01:13:30,090 DEBUG [kvm.resource.LibvirtConnection]
>> >> (agentRequest-Handler-2:null) (logid:388b92e0) Looking for libvirtd
>> >> connection at: qemu:///system
>> >>
>> >> 2020-09-29 01:13:30,094 DEBUG [kvm.resource.LibvirtComputingResource]
>> >> (agentRequest-Handler-2:null) (logid:388b92e0) Checking default network
>> >> rules for vm i-23-77-VM
>> >>
>> >> 2020-09-29 01:13:30,094 ERROR [kvm.resource.LibvirtComputingResource]
>> >> (agentRequest-Handler-2:null) (logid:388b92e0) Unable to apply default
>> >> network rule for nic cloudbr0 for VM i-23-77-VM
>> >>
>> >> 2020-09-29 01:13:30,094 WARN
>> >> [resource.wrapper.LibvirtSecurityGroupRulesCommandWrapper]
>> >> (agentRequest-Handler-2:null) (logid:388b92e0) Failed to program
>> default
>> >> network rules for vm i-23-77-VM
>> >>
>> >> 2020-09-29 01:13:30,095 DEBUG [cloud.agent.Agent]
>> >> (agentRequest-Handler-2:null) (logid:388b92e0) Seq
>> >> 11-6057904448766738456: {
>> >> Ans: , MgmtId: 72752492851638, via: 11, Ver: v1, Flags: 110,
>> >>
>> >>
>> [{"com.cloud.agent.api.SecurityGroupRuleAnswer":{"logSequenceNumber":15,"vmId":77,"reason":"PROGRAMMING_FAILED","result":false,"details":"programming
>> >> default network rules failed","wait":0}}] }
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> On Tue, Sep 29, 2020 at 12:38 PM Pearl d'Silva <
>> >> pearl.dsilva@shapeblue.com>
>> >> wrote:
>> >>
>> >> > Hi Hean,
>> >> >
>> >> >
>> >> > Could you try enabling debug logging on your hypervisor hosts, by
>> >> running
>> >> > the following:
>> >> > *sed -i 's/INFO/DEBUG/g' /etc/cloudstack/agent/log4j-cloud.xml*
>> >> >
>> >> > And then restart the cloudstack-agent service on the hosts. Post
>> this,
>> >> try
>> >> > deploying a VM in the shared network. This may give a better insight
>> >> into
>> >> > where exactly the issue lies.
>> >> >
>> >> > Thanks,
>> >> > Pearl
>> >> >
>> >> >
>> >> > Software Engineer
>> >> > pearl.dsilva@shapeblue.com
>> >> > www.shapeblue.com<http://www.shapeblue.com>
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > ------------------------------
>> >> > *From:* Ivan Kudryavtsev <iv...@bw-sw.com>
>> >> > *Sent:* Monday, September 28, 2020 2:17 PM
>> >> > *To:* users <us...@cloudstack.apache.org>
>> >> > *Subject:* Re: Cloudstack Advance with Security Group
>> >> >
>> >> > This is it.
>> >> >
>> >>
>> >> pearl.dsilva@shapeblue.com
>> >> www.shapeblue.com<http://www.shapeblue.com>
>> >> 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK
>> >> @shapeblue
>> >>
>> >>
>> >>
>> >> > On Mon, Sep 28, 2020 at 3:46 PM Hean Seng <he...@gmail.com>
>> wrote:
>> >> >
>> >> > > In the log, I cannot see anything much, except a few lines showing
>> >> above
>> >> > > . Not sure if this is a bug on 4.14.
>> >> > >
>> >> > >
>> >> > >
>> >> > > 2020-09-28 03:53:36,585 ERROR
>> [kvm.resource.LibvirtComputingResource]
>> >> > > (agentRequest-Handler-4:null) (logid:b6a4c077) Unable to apply
>> default
>> >> > > network rule for nic cloudbr0 for VM i-2-81-VM
>> >> > >
>> >> > > 2020-09-28 03:53:36,858 ERROR
>> [kvm.resource.LibvirtComputingResource]
>> >> > > (agentRequest-Handler-3:null) (logid:74058678) Unable to apply
>> default
>> >> > > network rule for nic cloudbr0 for VM i-2-81-VM
>> >> > >
>> >> > > 2020-09-28 03:53:36,858 WARN
>> >> > > [resource.wrapper.LibvirtSecurityGroupRulesCommandWrapper]
>> >> > > (agentRequest-Handler-3:null) (logid:74058678) Failed to program
>> >> default
>> >> > > network rules for vm i-2-81-VM
>> >> > >
>> >> > >
>> >> > >
>> >> > >
>> >> > > On Mon, Sep 28, 2020 at 4:34 PM Ivan Kudryavtsev <iv...@bw-sw.com>
>> >> wrote:
>> >> > >
>> >> > > > Hi,
>> >> > > > no I'm on 4.11, so can not help with exact 4.14, and I'm on
>> Ubuntu,
>> >> > > though,
>> >> > > > but for any KVM hypervisor Linux distribution, the logic is the
>> >> same.
>> >> > > >
>> >> > > > On Mon, Sep 28, 2020 at 3:31 PM Hean Seng <he...@gmail.com>
>> >> wrote:
>> >> > > >
>> >> > > > > Hi
>> >> > > > >
>> >> > > > > Are you running on CentOS7 ?
>> >> > > > >
>> >> > > > > I am running on CentOS7 , ACS 4.14 , and seem there is no
>> log
>> >> at
>> >> > of
>> >> > > > > security_group.log
>> >> > > > >
>> >> > > > > # ls /var/log/cloudstack/agent/
>> >> > > > >
>> >> > > > > agent.log resizevolume.log setup.log
>> >> > > > >
>> >> > > > >
>> >> > > > > I recheck back the Intall guide, seems no missing anything.
>> >> > > > >
>> >> > > > >
>> >> > > > > Older intallation guide, 4.11 mentioned need , allow
>> >> > > > > /usr/lib/sysctl.d/00-system.conf
>> >> > > > >
>> >> > > > > # Enable netfilter on bridges.
>> >> net.bridge.bridge-nf-call-ip6tables =
>> >> > 1
>> >> > > > > net.bridge.bridge-nf-call-iptables = 1
>> >> > > > net.bridge.bridge-nf-call-arptables
>> >> > > > > = 1
>> >> > > > >
>> >> > > > > And it has been done too.
>> >> > > > >
>> >> > > > >
>> >> > > > >
>> >> > > > > On Mon, Sep 28, 2020 at 4:05 PM Ivan Kudryavtsev <
>> ivan@bw-sw.com>
>> >> > > wrote:
>> >> > > > >
>> >> > > > > > Hi,
>> >> > > > > >
>> >> > > > > > No, this is not the issue.
>> >> > > > > > It's a normal state of the system, as KVM hooks are a new and
>> >> > > optional
>> >> > > > > > feature of 4.14.
>> >> > > > > >
>> >> > > > > > You should find some sort of messages regarding
>> security_groups
>> >> at
>> >> > > > > > /var/log/cloudstack/agent/security_group.log
>> >> > > > > >
>> >> > > > > >
>> >> > > > > > On Mon, Sep 28, 2020 at 2:10 PM Hean Seng <
>> heanseng@gmail.com>
>> >> > > wrote:
>> >> > > > > >
>> >> > > > > > > I not sure where goes wrong, are you running on CentOS 7
>> ? I
>> >> > have
>> >> > > > this
>> >> > > > > > > error too, do you think is this contribute to the error as
>> >> well:
>> >> > > > > > >
>> >> > > > > > > 2020-09-28 03:04:52,762 WARN
>> >> [kvm.resource.LibvirtKvmAgentHook]
>> >> > > > > > > (agentRequest-Handler-5:null) (logid:4f23845b) Groovy
>> script
>> >> > > > > > >
>> >> '/etc/cloudstack/agent/hooks/libvirt-vm-xml-transformer.groovy'
>> >> > is
>> >> > > > not
>> >> > > > > > > available. Transformations will not be applied.
>> >> > > > > > >
>> >> > > > > > > 2020-09-28 03:04:52,762 WARN
>> >> [kvm.resource.LibvirtKvmAgentHook]
>> >> > > > > > > (agentRequest-Handler-5:null) (logid:4f23845b) Groovy
>> >> scripting
>> >> > > > engine
>> >> > > > > is
>> >> > > > > > > not initialized. Data transformation skipped.
>> >> > > > > > >
>> >> > > > > > > 2020-09-28 03:04:53,083 WARN
>> >> [kvm.resource.LibvirtKvmAgentHook]
>> >> > > > > > > (agentRequest-Handler-5:null) (logid:4f23845b) Groovy
>> script
>> >> > > > > > >
>> '/etc/cloudstack/agent/hooks/libvirt-vm-state-change.groovy'
>> >> is
>> >> > not
>> >> > > > > > > available. Transformations will not be applied.
>> >> > > > > > >
>> >> > > > > > > On Mon, Sep 28, 2020 at 2:27 PM Ivan Kudryavtsev <
>> >> ivan@bw-sw.com
>> >> > >
>> >> > > > > wrote:
>> >> > > > > > >
>> >> > > > > > > > This just means you installed it in the wrong way.
>> Ebtables
>> >> and
>> >> > > > > > Iptables
>> >> > > > > > > > must be filled with rules like
>> >> > > > > > > >
>> >> > > > > > > > -A i-6242-10304-def -m state --state RELATED,ESTABLISHED
>> -j
>> >> > > ACCEPT
>> >> > > > > > > > -A i-6242-10304-def -p udp -m physdev --physdev-in vnet18
>> >> > > > > > > > --physdev-is-bridged -m udp --sport 68 --dport 67 -j
>> ACCEPT
>> >> > > > > > > > -A i-6242-10304-def -p udp -m physdev --physdev-out
>> vnet18
>> >> > > > > > > > --physdev-is-bridged -m udp --sport 67 --dport 68 -j
>> ACCEPT
>> >> > > > > > > > -A i-6242-10304-def -m physdev --physdev-in vnet18
>> >> > > > > --physdev-is-bridged
>> >> > > > > > > -m
>> >> > > > > > > > set ! --match-set i-6242-10304-vm src -j DROP
>> >> > > > > > > > -A i-6242-10304-def -p udp -m physdev --physdev-in vnet18
>> >> > > > > > > > --physdev-is-bridged -m set --match-set i-6242-10304-vm
>> src
>> >> -m
>> >> > > udp
>> >> > > > > > > --dport
>> >> > > > > > > > 53 -j RETURN
>> >> > > > > > > > -A i-6242-10304-def -p tcp -m physdev --physdev-in vnet18
>> >> > > > > > > > --physdev-is-bridged -m set --match-set i-6242-10304-vm
>> src
>> >> -m
>> >> > > tcp
>> >> > > > > > > --dport
>> >> > > > > > > > 53 -j RETURN
>> >> > > > > > > > -A i-6242-10304-def -m physdev --physdev-in vnet18
>> >> > > > > --physdev-is-bridged
>> >> > > > > > > -m
>> >> > > > > > > > set --match-set i-6242-10304-vm src -j i-6242-10304-vm-eg
>> >> > > > > > > > -A i-6242-10304-def -m physdev --physdev-out vnet18
>> >> > > > > > --physdev-is-bridged
>> >> > > > > > > -j
>> >> > > > > > > > i-6242-10304-vm
>> >> > > > > > > > -A i-6242-10304-vm -p udp -m udp --dport 1:65535 -m state
>> >> > --state
>> >> > > > NEW
>> >> > > > > > -j
>> >> > > > > > > > ACCEPT
>> >> > > > > > > > -A i-6242-10304-vm -p tcp -m tcp --dport 1:65535 -m state
>> >> > --state
>> >> > > > NEW
>> >> > > > > > -j
>> >> > > > > > > > ACCEPT
>> >> > > > > > > > -A i-6242-10304-vm -p icmp -m icmp --icmp-type any -j
>> ACCEPT
>> >> > > > > > > > -A i-6242-10304-vm -j DROP
>> >> > > > > > > >
>> >> > > > > > > >
>> >> > > > > > > > Bridge chain: i-4435-8929-vm-in, entries: 7, policy:
>> ACCEPT
>> >> > > > > > > > -s ! 1e:0:32:0:2:2 -j DROP
>> >> > > > > > > > -p ARP -s ! 1e:0:32:0:2:2 -j DROP
>> >> > > > > > > > -p ARP --arp-mac-src ! 1e:0:32:0:2:2 -j DROP
>> >> > > > > > > > -p ARP -j i-4435-8929-vm-in-ips
>> >> > > > > > > > -p ARP --arp-op Request -j ACCEPT
>> >> > > > > > > > -p ARP --arp-op Reply -j ACCEPT
>> >> > > > > > > > -p ARP -j DROP
>> >> > > > > > > >
>> >> > > > > > > >
>> >> > > > > > > >
>> >> > > > > > > > On Mon, Sep 28, 2020 at 1:10 PM Hean Seng <
>> >> heanseng@gmail.com>
>> >> > > > > wrote:
>> >> > > > > > > >
>> >> > > > > > > > > I checked the hypervisor , it seems iptables is nothing
>> >> > inside
>> >> > > ,
>> >> > > > > > this
>> >> > > > > > > is
>> >> > > > > > > > > centos7 , initially i turnoff firewalld , but even i
>> >> turn
>> >> > on
>> >> > > it
>> >> > > > > now
>> >> > > > > > > and
>> >> > > > > > > > > try to update the security group rules, it seems empty
>> >> > iptable
>> >> > > > > rules
>> >> > > > > > :
>> >> > > > > > > > >
>> >> > > > > > > > > [root@kvm03 ~]# iptables -L -v -n
>> >> > > > > > > > >
>> >> > > > > > > > > Chain INPUT (policy ACCEPT 82903 packets, 1170M bytes)
>> >> > > > > > > > >
>> >> > > > > > > > > pkts bytes target prot opt in out source
>> >> > > > > > > > > destination
>> >> > > > > > > > >
>> >> > > > > > > > >
>> >> > > > > > > > > Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
>> >> > > > > > > > >
>> >> > > > > > > > > pkts bytes target prot opt in out source
>> >> > > > > > > > > destination
>> >> > > > > > > > >
>> >> > > > > > > > >
>> >> > > > > > > > > Chain OUTPUT (policy ACCEPT 80505 packets, 25M bytes)
>> >> > > > > > > > >
>> >> > > > > > > > > pkts bytes target prot opt in out source
>> >> > > > > > > > > destination
>> >> > > > > > > > >
>> >> > > > > > > > >
>> >> > > > > > > > >
>> >> > > > > > > > >
>> >> > > > > > > > >
>> >> > > > > > > > >
>> >> > > > > > > > >
>> >> > > > > > > > > On Mon, Sep 28, 2020 at 12:05 PM Pearl d'Silva <
>> >> > > > > > > > pearl.dsilva@shapeblue.com
>> >> > > > > > > > > >
>> >> > > > > > > > > wrote:
>> >> > > > > > > > >
>> >> > > > > > > > > > Hi Hean,
>> >> > > > > > > > > >
>> >> > > > > > > > > > In an Advanced Zone with Security Groups enabled, by
>> >> > default,
>> >> > > > > > egress
>> >> > > > > > > > > > traffic from the VM is allowed, while Ingress
>> traffic is
>> >> > > > denied.
>> >> > > > > > > Hence,
>> >> > > > > > > > > as
>> >> > > > > > > > > > you rightly mentioned, security group rules are added
>> >> > > > > accordingly.
>> >> > > > > > > > These
>> >> > > > > > > > > > rules get added on the hypervisor host, and you can
>> >> verify
>> >> > > > them,
>> >> > > > > by
>> >> > > > > > > > going
>> >> > > > > > > > > > into the host and searching for iptables rules
>> >> > corresponding
>> >> > > to
>> >> > > > > the
>> >> > > > > > > VM
>> >> > > > > > > > > > (internal name - i-x-y-VM).
>> >> > > > > > > > > > This blog maybe helpful in providing further details:
>> >> > > > > > > > > >
>> >> > > > > > > > > >
>> >> > > > > > > > >
>> >> > > > > > > >
>> >> > > > > > >
>> >> > > > > >
>> >> > > > >
>> >> > > >
>> >> > >
>> >> >
>> >>
>> https://shankerbalan.net/blog/cloudstack-advanced-zone-with-security-groups/
>> >> > > > > > > > > >
>> >> > > > > > > > > > Thanks,
>> >> > > > > > > > > > Pearl
>> >> > > > > > > > > > ________________________________
>> >> > > > > > > > > > From: Hean Seng <he...@gmail.com>
>> >> > > > > > > > > > Sent: Sunday, September 27, 2020 2:48 PM
>> >> > > > > > > > > > To: users@cloudstack.apache.org <
>> >> > users@cloudstack.apache.org
>> >> > > >
>> >> > > > > > > > > > Subject: Cloudstack Advance with Security Group
>> >> > > > > > > > > >
>> >> > > > > > > > > > Hi
>> >> > > > > > > > > >
>> >> > > > > > > > > > I created advance zone with security group, all
>> working
>> >> > fine.
>> >> > > > > > > > > >
>> >> > > > > > > > > > But VMcreated ， seems the default security group that
>> >> > > assigned
>> >> > > > to
>> >> > > > > > the
>> >> > > > > > > > VM.
>> >> > > > > > > > > > all accept policy , i understand is Default Deny,
>> and
>> >> once
>> >> > > add
>> >> > > > > in
>> >> > > > > > > the
>> >> > > > > > > > > port
>> >> > > > > > > > > > in Security Group Ingress and Egress, only is allowed
>> >> > > > > > > > > >
>> >> > > > > > > > > > Also, is this rules created at VirtualRouter of the
>> >> > > > > SharedNetwork,
>> >> > > > > > or
>> >> > > > > > > > at
>> >> > > > > > > > > > the Hypervisor?
>> >> > > > > > > > > >
>> >> > > > > > > > > >
>> >> > > > > > > > > >
>> >> > > > > > > > > > --
>> >> > > > > > > > > > Regards,
>> >> > > > > > > > > > Hean Seng
>> >> > > > > > > > > >
>> >> > > > > > > > > > pearl.dsilva@shapeblue.com
>> >> > > > > > > > > > www.shapeblue.com<http://www.shapeblue.com>
>> >> > > > > > > > > > 3 London Bridge Street, 3rd floor, News Building,
>> >> London
>> >> > > SE1
>> >> > > > > > 9SGUK
>> >> > > > > > > > > > @shapeblue
>> >> > > > > > > > > >
>> >> > > > > > > > > >
>> >> > > > > > > > > >
>> >> > > > > > > > > >
>> >> > > > > > > > >
>> >> > > > > > > > > --
>> >> > > > > > > > > Regards,
>> >> > > > > > > > > Hean Seng
>> >> > > > > > > > >
>> >> > > > > > > >
>> >> > > > > > >
>> >> > > > > > >
>> >> > > > > > > --
>> >> > > > > > > Regards,
>> >> > > > > > > Hean Seng
>> >> > > > > > >
>> >> > > > > >
>> >> > > > >
>> >> > > > >
>> >> > > > > --
>> >> > > > > Regards,
>> >> > > > > Hean Seng
>> >> > > > >
>> >> > > >
>> >> > >
>> >> > >
>> >> > > --
>> >> > > Regards,
>> >> > > Hean Seng
>> >> > >
>> >> >
>> >>
>> >>
>> >> --
>> >> Regards,
>> >> Hean Seng
>> >>
>> >
>> >
>> > --
>> > Regards,
>> > Hean Seng
>> >
>>
>>
>> --
>> Regards,
>> Hean Seng
>>
>
>
> --
> Regards,
> Hean Seng
>
--
Regards,
Hean Seng
Re: Cloudstack Advance with Security Group
Posted by Hean Seng <he...@gmail.com>.
Hi
It seem libvirt-phython already install previously
[root@kvm03 agent]# yum install libvirt-python -y
Failed to set locale, defaulting to C
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: hk.mirrors.thegigabit.com
* epel: hk.mirrors.thegigabit.com
* extras: mirror-hk.koddos.net
* updates: hk.mirrors.thegigabit.com
Package libvirt-python-4.5.0-1.el7.x86_64 already installed and latest
version
Nothing to do
This all installed
libvirt-libs-4.5.0-33.el7_8.1.x86_64
libvirt-daemon-4.5.0-33.el7_8.1.x86_64
libvirt-daemon-config-nwfilter-4.5.0-33.el7_8.1.x86_64
libvirt-daemon-driver-storage-rbd-4.5.0-33.el7_8.1.x86_64
libvirt-daemon-driver-storage-gluster-4.5.0-33.el7_8.1.x86_64
libvirt-daemon-driver-nodedev-4.5.0-33.el7_8.1.x86_64
libvirt-client-4.5.0-33.el7_8.1.x86_64
libvirt-daemon-driver-storage-core-4.5.0-33.el7_8.1.x86_64
libvirt-daemon-driver-nwfilter-4.5.0-33.el7_8.1.x86_64
libvirt-daemon-driver-qemu-4.5.0-33.el7_8.1.x86_64
libvirt-daemon-driver-lxc-4.5.0-33.el7_8.1.x86_64
libvirt-daemon-driver-storage-logical-4.5.0-33.el7_8.1.x86_64
libvirt-daemon-driver-storage-disk-4.5.0-33.el7_8.1.x86_64
libvirt-daemon-driver-storage-iscsi-4.5.0-33.el7_8.1.x86_64
libvirt-daemon-driver-storage-4.5.0-33.el7_8.1.x86_64
libvirt-daemon-driver-secret-4.5.0-33.el7_8.1.x86_64
libvirt-4.5.0-33.el7_8.1.x86_64
libvirt-bash-completion-4.5.0-33.el7_8.1.x86_64
libvirt-python-4.5.0-1.el7.x86_64
libvirt-daemon-driver-network-4.5.0-33.el7_8.1.x86_64
libvirt-daemon-config-network-4.5.0-33.el7_8.1.x86_64
libvirt-daemon-driver-storage-scsi-4.5.0-33.el7_8.1.x86_64
libvirt-daemon-driver-storage-mpath-4.5.0-33.el7_8.1.x86_64
libvirt-daemon-driver-interface-4.5.0-33.el7_8.1.x86_64
On Tue, Sep 29, 2020 at 3:46 PM Pearl d'Silva <pe...@shapeblue.com>
wrote:
> Hi,
>
> Yes, the script wouldn't do anything, if it's just run without args,
> however, it should prompt you with the args to be provided. Based on the
> error, it seems that you are missing a python library, hence the script
> terminates and that's why the log file isn't generated in the first place.
> You can install the package using:
> yum install libvirt-python
>
> Thanks,
> Pearl
> ________________________________
> From: Hean Seng <he...@gmail.com>
> Sent: Tuesday, September 29, 2020 1:10 PM
> To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> Subject: Re: Cloudstack Advance with Security Group
>
> The error when running manually:
>
> ]# /usr/share/cloudstack-common/scripts/vm/network/security_group.py
>
> Traceback (most recent call last):
>
> File "/usr/share/cloudstack-common/scripts/vm/network/security_group.py",
> line 26, in <module>
>
> import libvirt
>
> Package libvirt-4.5.0-33.el7_8.1.x86_64 already installed and latest
> version
>
>
> so I guess, cannot runjust like that ?
>
>
>
>
>
>
>
>
>
> pearl.dsilva@shapeblue.com
> www.shapeblue.com
> 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK
> @shapeblue
>
>
>
> On Tue, Sep 29, 2020 at 3:08 PM Hean Seng <he...@gmail.com> wrote:
>
> > Hi
> >
> > I am running on CentOS 7.8.2003
> >
> > Cloudstack Agent , and KVM hypervisor:
> >
> > cloudstack-agent-4.14.0.0-1.el7.x86_64
> >
> > iptables and ebtables aldy install, iptables -L and ebtables -L show
> > default rules .
> >
> > This is dumpxml, seem the bridge is there, the VLAN for Guest is VLAN401,
> > which the setup seem right :
> >
> > <interface type='bridge'>
> >
> > <mac address='02:00:21:d9:00:01'/>
> >
> > <source bridge='brem1-401'/>
> >
> > <bandwidth>
> >
> > <inbound average='1280' peak='1280'/>
> >
> > <outbound average='1280' peak='1280'/>
> >
> > </bandwidth>
> >
> > <target dev='vnet9'/>
> >
> > <model type='virtio'/>
> >
> > <link state='up'/>
> >
> > <alias name='net0'/>
> >
> > <address type='pci' domain='0x0000' bus='0x00' slot='0x03'
> > function='0x0'/>
> >
> >
> >
> > # ifconfig brem1-401
> >
> > brem1-401: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
> >
> > inet6 fe80::3824:b2ff:fe09:873f prefixlen 64 scopeid 0x20<link>
> >
> > ether 04:7d:7b:60:09:66 txqueuelen 1000 (Ethernet)
> >
> > RX packets 7191638 bytes 333185781 (317.7 MiB)
> >
> > RX errors 0 dropped 0 overruns 0 frame 0
> >
> > TX packets 8 bytes 656 (656.0 B)
> >
> > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
> >
> > brctl show
> >
> > bridge name bridge id STP enabled interfaces
> >
> > brem1-401 8000.047d7b600966 no em1.401
> >
> > vnet10
> >
> > vnet11
> >
> > vnet12
> >
> > vnet14
> >
> > vnet4
> >
> > vnet8
> >
> > vnet9
> >
> > cloud0 8000.fe00a9fe1cea no vnet13
> >
> > vnet15
> >
> > vnet2
> >
> > vnet6
> >
> > cloudbr0 8000.047d7b600966 yes em1
> >
> > vnet3
> >
> > vnet5
> >
> > vnet7
> >
> > cloudbr1 8000.000000000000 yes
> >
> >
> >
> > I always have /var/log/cloudstack/agent/ this path , just inside
> >
> > ls /var/log/cloudstack/agent/
> >
> > agent.log agent.log.2020-09-26.gz agent.log.2020-09-27.gz
> > agent.log.2020-09-28.gz resizevolume.log setup.log
> >
> > it doesnt have securitygroup folder/ log, all the log ip pasted is at
> > agent.log
> >
> >
> >
> >
> > On Tue, Sep 29, 2020 at 2:44 PM Pearl d'Silva <
> pearl.dsilva@shapeblue.com>
> > wrote:
> >
> >> Hi Hean,
> >>
> >> I was able to deploy a VM in a shared network in an advanced zone with
> >> security groups enabled and could see the iptables rules getting added
> for
> >> the VM on the host, so it is probably not a bug that we are seeing.
> Seems
> >> like a setup issue.
> >> Applying the rules can fail because:
> >>
> >> * The hypervisor host doesn't have iptables / ebtables command -
> >> which is probably unlikely
> >> * The VM deployed doesn't have an interface - can be verified by
> >> dumping its configuration - virsh dumpxml <domain_name> | grep interface
> >> * or application of the default network rules for the VM itself
> >> failed -If you are up for some debugging, can you try running the script
> >> manually in the hypervisor host:
> >>
> >> /usr/share/cloudstack-common/scripts/vm/network/security_group.py
> >> default_network_rules --vmname <domain_name> --vmid <vm_id> --vmip
> <vm_ip>
> >> --vmmac <vm_mac> --vif <device/networkname> --brname <bridge_name>
> >> --nicsecips 0; --isFirstNic --check
> >>
> >> when you do a virsh dumpxml <vm_domain_name>, look out for an interface
> >> section that looks something like:
> >>
> >> <interface type='bridge'>
> >> <mac address='1e:00:86:00:00:ba'/>
> >> <source bridge='breth1-11'/>
> >> <bandwidth>
> >> <inbound average='25600' peak='25600'/>
> >> <outbound average='25600' peak='25600'/>
> >> </bandwidth>
> >> <target dev='vnet0'/>
> >> <model type='virtio'/>
> >> <link state='up'/>
> >> <alias name='net0'/>
> >> <address type='pci' domain='0x0000' bus='0x00' slot='0x03'
> >> function='0x0'/>
> >> </interface>
> >>
> >>
> >> the inputs to the script can be found as follows:
> >> domain_name - internal name of your vm (i-x-y-VM); can also be obtained
> >> by "virsh list"
> >> vm_id - the 'y' part of the internal name : i-x-y-VM
> >> vm_ip - Now that debug logs have been enabled, you can search for
> >> "StartCommand" in your logs get the ip from the 'nics' section or you
> could
> >> choose any ip from the shared network range
> >> vm_mac - from the above bridge section, you will be able to get the MAC
> >> address
> >> vif / device name - taget dev name - in the above case - vnet0
> >> bridge_name - again can be obtained from the above snippet (eg,
> breth1-11)
> >> if there are no secondary ips, then pass 0 for nicsecips otherwise
> >> specify the seconday ips separated by a semicolon (;)
> >>
> >> Basically, at the end of it, it should look something like:
> >> /usr/share/cloudstack-common/scripts/vm/network/security_group.py
> >> default_network_rules --vmname i-2-3-VM --vmid 3 --vmip <ip> --vmmac
> >> 1e:00:86:00:00:ba --vif vnet0 --brname breth1-11 --nicsecips 0;
> --isFirstNic
> >>
> >> Hopefully, now you should be able to see a security_group.log file in
> the
> >> /var/log/cloudstack/agent/ path.
> >>
> >> Thanks,
> >> Pearl
> >>
> >>
> >>
> >>
> >>
> >> ________________________________
> >> From: Hean Seng <he...@gmail.com>
> >> Sent: Tuesday, September 29, 2020 10:45 AM
> >> To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> >> Subject: Re: Cloudstack Advance with Security Group
> >>
> >> Hi Pearl,
> >>
> >> I had change to debug, following is the error captured:
> >>
> >>
> >> 2020-09-29 01:13:30,090 DEBUG [cloud.agent.Agent]
> >> (agentRequest-Handler-2:null) (logid:388b92e0) Processing command:
> >> com.cloud.agent.api.SecurityGroupRulesCmd
> >>
> >> 2020-09-29 01:13:30,090 DEBUG [kvm.resource.LibvirtConnection]
> >> (agentRequest-Handler-2:null) (logid:388b92e0) Looking for libvirtd
> >> connection at: qemu:///system
> >>
> >> 2020-09-29 01:13:30,094 DEBUG [kvm.resource.LibvirtComputingResource]
> >> (agentRequest-Handler-2:null) (logid:388b92e0) Checking default network
> >> rules for vm i-23-77-VM
> >>
> >> 2020-09-29 01:13:30,094 ERROR [kvm.resource.LibvirtComputingResource]
> >> (agentRequest-Handler-2:null) (logid:388b92e0) Unable to apply default
> >> network rule for nic cloudbr0 for VM i-23-77-VM
> >>
> >> 2020-09-29 01:13:30,094 WARN
> >> [resource.wrapper.LibvirtSecurityGroupRulesCommandWrapper]
> >> (agentRequest-Handler-2:null) (logid:388b92e0) Failed to program default
> >> network rules for vm i-23-77-VM
> >>
> >> 2020-09-29 01:13:30,095 DEBUG [cloud.agent.Agent]
> >> (agentRequest-Handler-2:null) (logid:388b92e0) Seq
> >> 11-6057904448766738456: {
> >> Ans: , MgmtId: 72752492851638, via: 11, Ver: v1, Flags: 110,
> >>
> >>
> [{"com.cloud.agent.api.SecurityGroupRuleAnswer":{"logSequenceNumber":15,"vmId":77,"reason":"PROGRAMMING_FAILED","result":false,"details":"programming
> >> default network rules failed","wait":0}}] }
> >>
> >>
> >>
> >>
> >>
> >> On Tue, Sep 29, 2020 at 12:38 PM Pearl d'Silva <
> >> pearl.dsilva@shapeblue.com>
> >> wrote:
> >>
> >> > Hi Hean,
> >> >
> >> >
> >> > Could you try enabling debug logging on your hypervisor hosts, by
> >> running
> >> > the following:
> >> > *sed -i 's/INFO/DEBUG/g' /etc/cloudstack/agent/log4j-cloud.xml*
> >> >
> >> > And then restart the cloudstack-agent service on the hosts. Post this,
> >> try
> >> > deploying a VM in the shared network. This may give a better insight
> >> into
> >> > where exactly the issue lies.
> >> >
> >> > Thanks,
> >> > Pearl
> >> >
> >> >
> >> > Software Engineer
> >> > pearl.dsilva@shapeblue.com
> >> > www.shapeblue.com<http://www.shapeblue.com>
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > ------------------------------
> >> > *From:* Ivan Kudryavtsev <iv...@bw-sw.com>
> >> > *Sent:* Monday, September 28, 2020 2:17 PM
> >> > *To:* users <us...@cloudstack.apache.org>
> >> > *Subject:* Re: Cloudstack Advance with Security Group
> >> >
> >> > This is it.
> >> >
> >>
> >> pearl.dsilva@shapeblue.com
> >> www.shapeblue.com<http://www.shapeblue.com>
> >> 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK
> >> @shapeblue
> >>
> >>
> >>
> >> > On Mon, Sep 28, 2020 at 3:46 PM Hean Seng <he...@gmail.com> wrote:
> >> >
> >> > > In the log, I cannot see anything much, except a few lines showing
> >> above
> >> > > . Not sure if this is a bug on 4.14.
> >> > >
> >> > >
> >> > >
> >> > > 2020-09-28 03:53:36,585 ERROR
> [kvm.resource.LibvirtComputingResource]
> >> > > (agentRequest-Handler-4:null) (logid:b6a4c077) Unable to apply
> default
> >> > > network rule for nic cloudbr0 for VM i-2-81-VM
> >> > >
> >> > > 2020-09-28 03:53:36,858 ERROR
> [kvm.resource.LibvirtComputingResource]
> >> > > (agentRequest-Handler-3:null) (logid:74058678) Unable to apply
> default
> >> > > network rule for nic cloudbr0 for VM i-2-81-VM
> >> > >
> >> > > 2020-09-28 03:53:36,858 WARN
> >> > > [resource.wrapper.LibvirtSecurityGroupRulesCommandWrapper]
> >> > > (agentRequest-Handler-3:null) (logid:74058678) Failed to program
> >> default
> >> > > network rules for vm i-2-81-VM
> >> > >
> >> > >
> >> > >
> >> > >
> >> > > On Mon, Sep 28, 2020 at 4:34 PM Ivan Kudryavtsev <iv...@bw-sw.com>
> >> wrote:
> >> > >
> >> > > > Hi,
> >> > > > no I'm on 4.11, so can not help with exact 4.14, and I'm on
> Ubuntu,
> >> > > though,
> >> > > > but for any KVM hypervisor Linux distribution, the logic is the
> >> same.
> >> > > >
> >> > > > On Mon, Sep 28, 2020 at 3:31 PM Hean Seng <he...@gmail.com>
> >> wrote:
> >> > > >
> >> > > > > Hi
> >> > > > >
> >> > > > > Are you running on CentOS7 ?
> >> > > > >
> >> > > > > I am running on CentOS7 , ACS 4.14 , and seem there is no log
> >> at
> >> > of
> >> > > > > security_group.log
> >> > > > >
> >> > > > > # ls /var/log/cloudstack/agent/
> >> > > > >
> >> > > > > agent.log resizevolume.log setup.log
> >> > > > >
> >> > > > >
> >> > > > > I recheck back the Intall guide, seems no missing anything.
> >> > > > >
> >> > > > >
> >> > > > > Older intallation guide, 4.11 mentioned need , allow
> >> > > > > /usr/lib/sysctl.d/00-system.conf
> >> > > > >
> >> > > > > # Enable netfilter on bridges.
> >> net.bridge.bridge-nf-call-ip6tables =
> >> > 1
> >> > > > > net.bridge.bridge-nf-call-iptables = 1
> >> > > > net.bridge.bridge-nf-call-arptables
> >> > > > > = 1
> >> > > > >
> >> > > > > And it has been done too.
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > > On Mon, Sep 28, 2020 at 4:05 PM Ivan Kudryavtsev <
> ivan@bw-sw.com>
> >> > > wrote:
> >> > > > >
> >> > > > > > Hi,
> >> > > > > >
> >> > > > > > No, this is not the issue.
> >> > > > > > It's a normal state of the system, as KVM hooks are a new and
> >> > > optional
> >> > > > > > feature of 4.14.
> >> > > > > >
> >> > > > > > You should find some sort of messages regarding
> security_groups
> >> at
> >> > > > > > /var/log/cloudstack/agent/security_group.log
> >> > > > > >
> >> > > > > >
> >> > > > > > On Mon, Sep 28, 2020 at 2:10 PM Hean Seng <heanseng@gmail.com
> >
> >> > > wrote:
> >> > > > > >
> >> > > > > > > I not sure where goes wrong, are you running on CentOS 7 ?
> I
> >> > have
> >> > > > this
> >> > > > > > > error too, do you think is this contribute to the error as
> >> well:
> >> > > > > > >
> >> > > > > > > 2020-09-28 03:04:52,762 WARN
> >> [kvm.resource.LibvirtKvmAgentHook]
> >> > > > > > > (agentRequest-Handler-5:null) (logid:4f23845b) Groovy script
> >> > > > > > >
> >> '/etc/cloudstack/agent/hooks/libvirt-vm-xml-transformer.groovy'
> >> > is
> >> > > > not
> >> > > > > > > available. Transformations will not be applied.
> >> > > > > > >
> >> > > > > > > 2020-09-28 03:04:52,762 WARN
> >> [kvm.resource.LibvirtKvmAgentHook]
> >> > > > > > > (agentRequest-Handler-5:null) (logid:4f23845b) Groovy
> >> scripting
> >> > > > engine
> >> > > > > is
> >> > > > > > > not initialized. Data transformation skipped.
> >> > > > > > >
> >> > > > > > > 2020-09-28 03:04:53,083 WARN
> >> [kvm.resource.LibvirtKvmAgentHook]
> >> > > > > > > (agentRequest-Handler-5:null) (logid:4f23845b) Groovy script
> >> > > > > > > '/etc/cloudstack/agent/hooks/libvirt-vm-state-change.groovy'
> >> is
> >> > not
> >> > > > > > > available. Transformations will not be applied.
> >> > > > > > >
> >> > > > > > > On Mon, Sep 28, 2020 at 2:27 PM Ivan Kudryavtsev <
> >> ivan@bw-sw.com
> >> > >
> >> > > > > wrote:
> >> > > > > > >
> >> > > > > > > > This just means you installed it in the wrong way.
> Ebtables
> >> and
> >> > > > > > Iptables
> >> > > > > > > > must be filled with rules like
> >> > > > > > > >
> >> > > > > > > > -A i-6242-10304-def -m state --state RELATED,ESTABLISHED
> -j
> >> > > ACCEPT
> >> > > > > > > > -A i-6242-10304-def -p udp -m physdev --physdev-in vnet18
> >> > > > > > > > --physdev-is-bridged -m udp --sport 68 --dport 67 -j
> ACCEPT
> >> > > > > > > > -A i-6242-10304-def -p udp -m physdev --physdev-out vnet18
> >> > > > > > > > --physdev-is-bridged -m udp --sport 67 --dport 68 -j
> ACCEPT
> >> > > > > > > > -A i-6242-10304-def -m physdev --physdev-in vnet18
> >> > > > > --physdev-is-bridged
> >> > > > > > > -m
> >> > > > > > > > set ! --match-set i-6242-10304-vm src -j DROP
> >> > > > > > > > -A i-6242-10304-def -p udp -m physdev --physdev-in vnet18
> >> > > > > > > > --physdev-is-bridged -m set --match-set i-6242-10304-vm
> src
> >> -m
> >> > > udp
> >> > > > > > > --dport
> >> > > > > > > > 53 -j RETURN
> >> > > > > > > > -A i-6242-10304-def -p tcp -m physdev --physdev-in vnet18
> >> > > > > > > > --physdev-is-bridged -m set --match-set i-6242-10304-vm
> src
> >> -m
> >> > > tcp
> >> > > > > > > --dport
> >> > > > > > > > 53 -j RETURN
> >> > > > > > > > -A i-6242-10304-def -m physdev --physdev-in vnet18
> >> > > > > --physdev-is-bridged
> >> > > > > > > -m
> >> > > > > > > > set --match-set i-6242-10304-vm src -j i-6242-10304-vm-eg
> >> > > > > > > > -A i-6242-10304-def -m physdev --physdev-out vnet18
> >> > > > > > --physdev-is-bridged
> >> > > > > > > -j
> >> > > > > > > > i-6242-10304-vm
> >> > > > > > > > -A i-6242-10304-vm -p udp -m udp --dport 1:65535 -m state
> >> > --state
> >> > > > NEW
> >> > > > > > -j
> >> > > > > > > > ACCEPT
> >> > > > > > > > -A i-6242-10304-vm -p tcp -m tcp --dport 1:65535 -m state
> >> > --state
> >> > > > NEW
> >> > > > > > -j
> >> > > > > > > > ACCEPT
> >> > > > > > > > -A i-6242-10304-vm -p icmp -m icmp --icmp-type any -j
> ACCEPT
> >> > > > > > > > -A i-6242-10304-vm -j DROP
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > Bridge chain: i-4435-8929-vm-in, entries: 7, policy:
> ACCEPT
> >> > > > > > > > -s ! 1e:0:32:0:2:2 -j DROP
> >> > > > > > > > -p ARP -s ! 1e:0:32:0:2:2 -j DROP
> >> > > > > > > > -p ARP --arp-mac-src ! 1e:0:32:0:2:2 -j DROP
> >> > > > > > > > -p ARP -j i-4435-8929-vm-in-ips
> >> > > > > > > > -p ARP --arp-op Request -j ACCEPT
> >> > > > > > > > -p ARP --arp-op Reply -j ACCEPT
> >> > > > > > > > -p ARP -j DROP
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > On Mon, Sep 28, 2020 at 1:10 PM Hean Seng <
> >> heanseng@gmail.com>
> >> > > > > wrote:
> >> > > > > > > >
> >> > > > > > > > > I checked the hypervisor , it seems iptables is nothing
> >> > inside
> >> > > ,
> >> > > > > > this
> >> > > > > > > is
> >> > > > > > > > > centos7 , initially i turnoff firewalld , but even i
> >> turn
> >> > on
> >> > > it
> >> > > > > now
> >> > > > > > > and
> >> > > > > > > > > try to update the security group rules, it seems empty
> >> > iptable
> >> > > > > rules
> >> > > > > > :
> >> > > > > > > > >
> >> > > > > > > > > [root@kvm03 ~]# iptables -L -v -n
> >> > > > > > > > >
> >> > > > > > > > > Chain INPUT (policy ACCEPT 82903 packets, 1170M bytes)
> >> > > > > > > > >
> >> > > > > > > > > pkts bytes target prot opt in out source
> >> > > > > > > > > destination
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
> >> > > > > > > > >
> >> > > > > > > > > pkts bytes target prot opt in out source
> >> > > > > > > > > destination
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > Chain OUTPUT (policy ACCEPT 80505 packets, 25M bytes)
> >> > > > > > > > >
> >> > > > > > > > > pkts bytes target prot opt in out source
> >> > > > > > > > > destination
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > On Mon, Sep 28, 2020 at 12:05 PM Pearl d'Silva <
> >> > > > > > > > pearl.dsilva@shapeblue.com
> >> > > > > > > > > >
> >> > > > > > > > > wrote:
> >> > > > > > > > >
> >> > > > > > > > > > Hi Hean,
> >> > > > > > > > > >
> >> > > > > > > > > > In an Advanced Zone with Security Groups enabled, by
> >> > default,
> >> > > > > > egress
> >> > > > > > > > > > traffic from the VM is allowed, while Ingress traffic
> is
> >> > > > denied.
> >> > > > > > > Hence,
> >> > > > > > > > > as
> >> > > > > > > > > > you rightly mentioned, security group rules are added
> >> > > > > accordingly.
> >> > > > > > > > These
> >> > > > > > > > > > rules get added on the hypervisor host, and you can
> >> verify
> >> > > > them,
> >> > > > > by
> >> > > > > > > > going
> >> > > > > > > > > > into the host and searching for iptables rules
> >> > corresponding
> >> > > to
> >> > > > > the
> >> > > > > > > VM
> >> > > > > > > > > > (internal name - i-x-y-VM).
> >> > > > > > > > > > This blog maybe helpful in providing further details:
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://shankerbalan.net/blog/cloudstack-advanced-zone-with-security-groups/
> >> > > > > > > > > >
> >> > > > > > > > > > Thanks,
> >> > > > > > > > > > Pearl
> >> > > > > > > > > > ________________________________
> >> > > > > > > > > > From: Hean Seng <he...@gmail.com>
> >> > > > > > > > > > Sent: Sunday, September 27, 2020 2:48 PM
> >> > > > > > > > > > To: users@cloudstack.apache.org <
> >> > users@cloudstack.apache.org
> >> > > >
> >> > > > > > > > > > Subject: Cloudstack Advance with Security Group
> >> > > > > > > > > >
> >> > > > > > > > > > Hi
> >> > > > > > > > > >
> >> > > > > > > > > > I created advance zone with security group, all
> working
> >> > fine.
> >> > > > > > > > > >
> >> > > > > > > > > > But VMcreated ， seems the default security group that
> >> > > assigned
> >> > > > to
> >> > > > > > the
> >> > > > > > > > VM.
> >> > > > > > > > > > all accept policy , i understand is Default Deny, and
> >> once
> >> > > add
> >> > > > > in
> >> > > > > > > the
> >> > > > > > > > > port
> >> > > > > > > > > > in Security Group Ingress and Egress, only is allowed
> >> > > > > > > > > >
> >> > > > > > > > > > Also, is this rules created at VirtualRouter of the
> >> > > > > SharedNetwork,
> >> > > > > > or
> >> > > > > > > > at
> >> > > > > > > > > > the Hypervisor?
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > --
> >> > > > > > > > > > Regards,
> >> > > > > > > > > > Hean Seng
> >> > > > > > > > > >
> >> > > > > > > > > > pearl.dsilva@shapeblue.com
> >> > > > > > > > > > www.shapeblue.com<http://www.shapeblue.com>
> >> > > > > > > > > > 3 London Bridge Street, 3rd floor, News Building,
> >> London
> >> > > SE1
> >> > > > > > 9SGUK
> >> > > > > > > > > > @shapeblue
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > --
> >> > > > > > > > > Regards,
> >> > > > > > > > > Hean Seng
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > --
> >> > > > > > > Regards,
> >> > > > > > > Hean Seng
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > > >
> >> > > > > --
> >> > > > > Regards,
> >> > > > > Hean Seng
> >> > > > >
> >> > > >
> >> > >
> >> > >
> >> > > --
> >> > > Regards,
> >> > > Hean Seng
> >> > >
> >> >
> >>
> >>
> >> --
> >> Regards,
> >> Hean Seng
> >>
> >
> >
> > --
> > Regards,
> > Hean Seng
> >
>
>
> --
> Regards,
> Hean Seng
>
--
Regards,
Hean Seng
Re: Cloudstack Advance with Security Group
Posted by Pearl d'Silva <pe...@shapeblue.com>.
Hi,
Yes, the script wouldn't do anything, if it's just run without args, however, it should prompt you with the args to be provided. Based on the error, it seems that you are missing a python library, hence the script terminates and that's why the log file isn't generated in the first place. You can install the package using:
yum install libvirt-python
Thanks,
Pearl
________________________________
From: Hean Seng <he...@gmail.com>
Sent: Tuesday, September 29, 2020 1:10 PM
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: Re: Cloudstack Advance with Security Group
The error when running manually:
]# /usr/share/cloudstack-common/scripts/vm/network/security_group.py
Traceback (most recent call last):
File "/usr/share/cloudstack-common/scripts/vm/network/security_group.py",
line 26, in <module>
import libvirt
Package libvirt-4.5.0-33.el7_8.1.x86_64 already installed and latest version
so I guess, cannot runjust like that ?
pearl.dsilva@shapeblue.com
www.shapeblue.com
3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK
@shapeblue
On Tue, Sep 29, 2020 at 3:08 PM Hean Seng <he...@gmail.com> wrote:
> Hi
>
> I am running on CentOS 7.8.2003
>
> Cloudstack Agent , and KVM hypervisor:
>
> cloudstack-agent-4.14.0.0-1.el7.x86_64
>
> iptables and ebtables aldy install, iptables -L and ebtables -L show
> default rules .
>
> This is dumpxml, seem the bridge is there, the VLAN for Guest is VLAN401,
> which the setup seem right :
>
> <interface type='bridge'>
>
> <mac address='02:00:21:d9:00:01'/>
>
> <source bridge='brem1-401'/>
>
> <bandwidth>
>
> <inbound average='1280' peak='1280'/>
>
> <outbound average='1280' peak='1280'/>
>
> </bandwidth>
>
> <target dev='vnet9'/>
>
> <model type='virtio'/>
>
> <link state='up'/>
>
> <alias name='net0'/>
>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x03'
> function='0x0'/>
>
>
>
> # ifconfig brem1-401
>
> brem1-401: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
>
> inet6 fe80::3824:b2ff:fe09:873f prefixlen 64 scopeid 0x20<link>
>
> ether 04:7d:7b:60:09:66 txqueuelen 1000 (Ethernet)
>
> RX packets 7191638 bytes 333185781 (317.7 MiB)
>
> RX errors 0 dropped 0 overruns 0 frame 0
>
> TX packets 8 bytes 656 (656.0 B)
>
> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
>
> brctl show
>
> bridge name bridge id STP enabled interfaces
>
> brem1-401 8000.047d7b600966 no em1.401
>
> vnet10
>
> vnet11
>
> vnet12
>
> vnet14
>
> vnet4
>
> vnet8
>
> vnet9
>
> cloud0 8000.fe00a9fe1cea no vnet13
>
> vnet15
>
> vnet2
>
> vnet6
>
> cloudbr0 8000.047d7b600966 yes em1
>
> vnet3
>
> vnet5
>
> vnet7
>
> cloudbr1 8000.000000000000 yes
>
>
>
> I always have /var/log/cloudstack/agent/ this path , just inside
>
> ls /var/log/cloudstack/agent/
>
> agent.log agent.log.2020-09-26.gz agent.log.2020-09-27.gz
> agent.log.2020-09-28.gz resizevolume.log setup.log
>
> it doesnt have securitygroup folder/ log, all the log ip pasted is at
> agent.log
>
>
>
>
> On Tue, Sep 29, 2020 at 2:44 PM Pearl d'Silva <pe...@shapeblue.com>
> wrote:
>
>> Hi Hean,
>>
>> I was able to deploy a VM in a shared network in an advanced zone with
>> security groups enabled and could see the iptables rules getting added for
>> the VM on the host, so it is probably not a bug that we are seeing. Seems
>> like a setup issue.
>> Applying the rules can fail because:
>>
>> * The hypervisor host doesn't have iptables / ebtables command -
>> which is probably unlikely
>> * The VM deployed doesn't have an interface - can be verified by
>> dumping its configuration - virsh dumpxml <domain_name> | grep interface
>> * or application of the default network rules for the VM itself
>> failed -If you are up for some debugging, can you try running the script
>> manually in the hypervisor host:
>>
>> /usr/share/cloudstack-common/scripts/vm/network/security_group.py
>> default_network_rules --vmname <domain_name> --vmid <vm_id> --vmip <vm_ip>
>> --vmmac <vm_mac> --vif <device/networkname> --brname <bridge_name>
>> --nicsecips 0; --isFirstNic --check
>>
>> when you do a virsh dumpxml <vm_domain_name>, look out for an interface
>> section that looks something like:
>>
>> <interface type='bridge'>
>> <mac address='1e:00:86:00:00:ba'/>
>> <source bridge='breth1-11'/>
>> <bandwidth>
>> <inbound average='25600' peak='25600'/>
>> <outbound average='25600' peak='25600'/>
>> </bandwidth>
>> <target dev='vnet0'/>
>> <model type='virtio'/>
>> <link state='up'/>
>> <alias name='net0'/>
>> <address type='pci' domain='0x0000' bus='0x00' slot='0x03'
>> function='0x0'/>
>> </interface>
>>
>>
>> the inputs to the script can be found as follows:
>> domain_name - internal name of your vm (i-x-y-VM); can also be obtained
>> by "virsh list"
>> vm_id - the 'y' part of the internal name : i-x-y-VM
>> vm_ip - Now that debug logs have been enabled, you can search for
>> "StartCommand" in your logs get the ip from the 'nics' section or you could
>> choose any ip from the shared network range
>> vm_mac - from the above bridge section, you will be able to get the MAC
>> address
>> vif / device name - taget dev name - in the above case - vnet0
>> bridge_name - again can be obtained from the above snippet (eg, breth1-11)
>> if there are no secondary ips, then pass 0 for nicsecips otherwise
>> specify the seconday ips separated by a semicolon (;)
>>
>> Basically, at the end of it, it should look something like:
>> /usr/share/cloudstack-common/scripts/vm/network/security_group.py
>> default_network_rules --vmname i-2-3-VM --vmid 3 --vmip <ip> --vmmac
>> 1e:00:86:00:00:ba --vif vnet0 --brname breth1-11 --nicsecips 0; --isFirstNic
>>
>> Hopefully, now you should be able to see a security_group.log file in the
>> /var/log/cloudstack/agent/ path.
>>
>> Thanks,
>> Pearl
>>
>>
>>
>>
>>
>> ________________________________
>> From: Hean Seng <he...@gmail.com>
>> Sent: Tuesday, September 29, 2020 10:45 AM
>> To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
>> Subject: Re: Cloudstack Advance with Security Group
>>
>> Hi Pearl,
>>
>> I had change to debug, following is the error captured:
>>
>>
>> 2020-09-29 01:13:30,090 DEBUG [cloud.agent.Agent]
>> (agentRequest-Handler-2:null) (logid:388b92e0) Processing command:
>> com.cloud.agent.api.SecurityGroupRulesCmd
>>
>> 2020-09-29 01:13:30,090 DEBUG [kvm.resource.LibvirtConnection]
>> (agentRequest-Handler-2:null) (logid:388b92e0) Looking for libvirtd
>> connection at: qemu:///system
>>
>> 2020-09-29 01:13:30,094 DEBUG [kvm.resource.LibvirtComputingResource]
>> (agentRequest-Handler-2:null) (logid:388b92e0) Checking default network
>> rules for vm i-23-77-VM
>>
>> 2020-09-29 01:13:30,094 ERROR [kvm.resource.LibvirtComputingResource]
>> (agentRequest-Handler-2:null) (logid:388b92e0) Unable to apply default
>> network rule for nic cloudbr0 for VM i-23-77-VM
>>
>> 2020-09-29 01:13:30,094 WARN
>> [resource.wrapper.LibvirtSecurityGroupRulesCommandWrapper]
>> (agentRequest-Handler-2:null) (logid:388b92e0) Failed to program default
>> network rules for vm i-23-77-VM
>>
>> 2020-09-29 01:13:30,095 DEBUG [cloud.agent.Agent]
>> (agentRequest-Handler-2:null) (logid:388b92e0) Seq
>> 11-6057904448766738456: {
>> Ans: , MgmtId: 72752492851638, via: 11, Ver: v1, Flags: 110,
>>
>> [{"com.cloud.agent.api.SecurityGroupRuleAnswer":{"logSequenceNumber":15,"vmId":77,"reason":"PROGRAMMING_FAILED","result":false,"details":"programming
>> default network rules failed","wait":0}}] }
>>
>>
>>
>>
>>
>> On Tue, Sep 29, 2020 at 12:38 PM Pearl d'Silva <
>> pearl.dsilva@shapeblue.com>
>> wrote:
>>
>> > Hi Hean,
>> >
>> >
>> > Could you try enabling debug logging on your hypervisor hosts, by
>> running
>> > the following:
>> > *sed -i 's/INFO/DEBUG/g' /etc/cloudstack/agent/log4j-cloud.xml*
>> >
>> > And then restart the cloudstack-agent service on the hosts. Post this,
>> try
>> > deploying a VM in the shared network. This may give a better insight
>> into
>> > where exactly the issue lies.
>> >
>> > Thanks,
>> > Pearl
>> >
>> >
>> > Software Engineer
>> > pearl.dsilva@shapeblue.com
>> > www.shapeblue.com<http://www.shapeblue.com>
>> >
>> >
>> >
>> >
>> >
>> >
>> > ------------------------------
>> > *From:* Ivan Kudryavtsev <iv...@bw-sw.com>
>> > *Sent:* Monday, September 28, 2020 2:17 PM
>> > *To:* users <us...@cloudstack.apache.org>
>> > *Subject:* Re: Cloudstack Advance with Security Group
>> >
>> > This is it.
>> >
>>
>> pearl.dsilva@shapeblue.com
>> www.shapeblue.com<http://www.shapeblue.com>
>> 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK
>> @shapeblue
>>
>>
>>
>> > On Mon, Sep 28, 2020 at 3:46 PM Hean Seng <he...@gmail.com> wrote:
>> >
>> > > In the log, I cannot see anything much, except a few lines showing
>> above
>> > > . Not sure if this is a bug on 4.14.
>> > >
>> > >
>> > >
>> > > 2020-09-28 03:53:36,585 ERROR [kvm.resource.LibvirtComputingResource]
>> > > (agentRequest-Handler-4:null) (logid:b6a4c077) Unable to apply default
>> > > network rule for nic cloudbr0 for VM i-2-81-VM
>> > >
>> > > 2020-09-28 03:53:36,858 ERROR [kvm.resource.LibvirtComputingResource]
>> > > (agentRequest-Handler-3:null) (logid:74058678) Unable to apply default
>> > > network rule for nic cloudbr0 for VM i-2-81-VM
>> > >
>> > > 2020-09-28 03:53:36,858 WARN
>> > > [resource.wrapper.LibvirtSecurityGroupRulesCommandWrapper]
>> > > (agentRequest-Handler-3:null) (logid:74058678) Failed to program
>> default
>> > > network rules for vm i-2-81-VM
>> > >
>> > >
>> > >
>> > >
>> > > On Mon, Sep 28, 2020 at 4:34 PM Ivan Kudryavtsev <iv...@bw-sw.com>
>> wrote:
>> > >
>> > > > Hi,
>> > > > no I'm on 4.11, so can not help with exact 4.14, and I'm on Ubuntu,
>> > > though,
>> > > > but for any KVM hypervisor Linux distribution, the logic is the
>> same.
>> > > >
>> > > > On Mon, Sep 28, 2020 at 3:31 PM Hean Seng <he...@gmail.com>
>> wrote:
>> > > >
>> > > > > Hi
>> > > > >
>> > > > > Are you running on CentOS7 ?
>> > > > >
>> > > > > I am running on CentOS7 , ACS 4.14 , and seem there is no log
>> at
>> > of
>> > > > > security_group.log
>> > > > >
>> > > > > # ls /var/log/cloudstack/agent/
>> > > > >
>> > > > > agent.log resizevolume.log setup.log
>> > > > >
>> > > > >
>> > > > > I recheck back the Intall guide, seems no missing anything.
>> > > > >
>> > > > >
>> > > > > Older intallation guide, 4.11 mentioned need , allow
>> > > > > /usr/lib/sysctl.d/00-system.conf
>> > > > >
>> > > > > # Enable netfilter on bridges.
>> net.bridge.bridge-nf-call-ip6tables =
>> > 1
>> > > > > net.bridge.bridge-nf-call-iptables = 1
>> > > > net.bridge.bridge-nf-call-arptables
>> > > > > = 1
>> > > > >
>> > > > > And it has been done too.
>> > > > >
>> > > > >
>> > > > >
>> > > > > On Mon, Sep 28, 2020 at 4:05 PM Ivan Kudryavtsev <iv...@bw-sw.com>
>> > > wrote:
>> > > > >
>> > > > > > Hi,
>> > > > > >
>> > > > > > No, this is not the issue.
>> > > > > > It's a normal state of the system, as KVM hooks are a new and
>> > > optional
>> > > > > > feature of 4.14.
>> > > > > >
>> > > > > > You should find some sort of messages regarding security_groups
>> at
>> > > > > > /var/log/cloudstack/agent/security_group.log
>> > > > > >
>> > > > > >
>> > > > > > On Mon, Sep 28, 2020 at 2:10 PM Hean Seng <he...@gmail.com>
>> > > wrote:
>> > > > > >
>> > > > > > > I not sure where goes wrong, are you running on CentOS 7 ? I
>> > have
>> > > > this
>> > > > > > > error too, do you think is this contribute to the error as
>> well:
>> > > > > > >
>> > > > > > > 2020-09-28 03:04:52,762 WARN
>> [kvm.resource.LibvirtKvmAgentHook]
>> > > > > > > (agentRequest-Handler-5:null) (logid:4f23845b) Groovy script
>> > > > > > >
>> '/etc/cloudstack/agent/hooks/libvirt-vm-xml-transformer.groovy'
>> > is
>> > > > not
>> > > > > > > available. Transformations will not be applied.
>> > > > > > >
>> > > > > > > 2020-09-28 03:04:52,762 WARN
>> [kvm.resource.LibvirtKvmAgentHook]
>> > > > > > > (agentRequest-Handler-5:null) (logid:4f23845b) Groovy
>> scripting
>> > > > engine
>> > > > > is
>> > > > > > > not initialized. Data transformation skipped.
>> > > > > > >
>> > > > > > > 2020-09-28 03:04:53,083 WARN
>> [kvm.resource.LibvirtKvmAgentHook]
>> > > > > > > (agentRequest-Handler-5:null) (logid:4f23845b) Groovy script
>> > > > > > > '/etc/cloudstack/agent/hooks/libvirt-vm-state-change.groovy'
>> is
>> > not
>> > > > > > > available. Transformations will not be applied.
>> > > > > > >
>> > > > > > > On Mon, Sep 28, 2020 at 2:27 PM Ivan Kudryavtsev <
>> ivan@bw-sw.com
>> > >
>> > > > > wrote:
>> > > > > > >
>> > > > > > > > This just means you installed it in the wrong way. Ebtables
>> and
>> > > > > > Iptables
>> > > > > > > > must be filled with rules like
>> > > > > > > >
>> > > > > > > > -A i-6242-10304-def -m state --state RELATED,ESTABLISHED -j
>> > > ACCEPT
>> > > > > > > > -A i-6242-10304-def -p udp -m physdev --physdev-in vnet18
>> > > > > > > > --physdev-is-bridged -m udp --sport 68 --dport 67 -j ACCEPT
>> > > > > > > > -A i-6242-10304-def -p udp -m physdev --physdev-out vnet18
>> > > > > > > > --physdev-is-bridged -m udp --sport 67 --dport 68 -j ACCEPT
>> > > > > > > > -A i-6242-10304-def -m physdev --physdev-in vnet18
>> > > > > --physdev-is-bridged
>> > > > > > > -m
>> > > > > > > > set ! --match-set i-6242-10304-vm src -j DROP
>> > > > > > > > -A i-6242-10304-def -p udp -m physdev --physdev-in vnet18
>> > > > > > > > --physdev-is-bridged -m set --match-set i-6242-10304-vm src
>> -m
>> > > udp
>> > > > > > > --dport
>> > > > > > > > 53 -j RETURN
>> > > > > > > > -A i-6242-10304-def -p tcp -m physdev --physdev-in vnet18
>> > > > > > > > --physdev-is-bridged -m set --match-set i-6242-10304-vm src
>> -m
>> > > tcp
>> > > > > > > --dport
>> > > > > > > > 53 -j RETURN
>> > > > > > > > -A i-6242-10304-def -m physdev --physdev-in vnet18
>> > > > > --physdev-is-bridged
>> > > > > > > -m
>> > > > > > > > set --match-set i-6242-10304-vm src -j i-6242-10304-vm-eg
>> > > > > > > > -A i-6242-10304-def -m physdev --physdev-out vnet18
>> > > > > > --physdev-is-bridged
>> > > > > > > -j
>> > > > > > > > i-6242-10304-vm
>> > > > > > > > -A i-6242-10304-vm -p udp -m udp --dport 1:65535 -m state
>> > --state
>> > > > NEW
>> > > > > > -j
>> > > > > > > > ACCEPT
>> > > > > > > > -A i-6242-10304-vm -p tcp -m tcp --dport 1:65535 -m state
>> > --state
>> > > > NEW
>> > > > > > -j
>> > > > > > > > ACCEPT
>> > > > > > > > -A i-6242-10304-vm -p icmp -m icmp --icmp-type any -j ACCEPT
>> > > > > > > > -A i-6242-10304-vm -j DROP
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > Bridge chain: i-4435-8929-vm-in, entries: 7, policy: ACCEPT
>> > > > > > > > -s ! 1e:0:32:0:2:2 -j DROP
>> > > > > > > > -p ARP -s ! 1e:0:32:0:2:2 -j DROP
>> > > > > > > > -p ARP --arp-mac-src ! 1e:0:32:0:2:2 -j DROP
>> > > > > > > > -p ARP -j i-4435-8929-vm-in-ips
>> > > > > > > > -p ARP --arp-op Request -j ACCEPT
>> > > > > > > > -p ARP --arp-op Reply -j ACCEPT
>> > > > > > > > -p ARP -j DROP
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > On Mon, Sep 28, 2020 at 1:10 PM Hean Seng <
>> heanseng@gmail.com>
>> > > > > wrote:
>> > > > > > > >
>> > > > > > > > > I checked the hypervisor , it seems iptables is nothing
>> > inside
>> > > ,
>> > > > > > this
>> > > > > > > is
>> > > > > > > > > centos7 , initially i turnoff firewalld , but even i
>> turn
>> > on
>> > > it
>> > > > > now
>> > > > > > > and
>> > > > > > > > > try to update the security group rules, it seems empty
>> > iptable
>> > > > > rules
>> > > > > > :
>> > > > > > > > >
>> > > > > > > > > [root@kvm03 ~]# iptables -L -v -n
>> > > > > > > > >
>> > > > > > > > > Chain INPUT (policy ACCEPT 82903 packets, 1170M bytes)
>> > > > > > > > >
>> > > > > > > > > pkts bytes target prot opt in out source
>> > > > > > > > > destination
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
>> > > > > > > > >
>> > > > > > > > > pkts bytes target prot opt in out source
>> > > > > > > > > destination
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > Chain OUTPUT (policy ACCEPT 80505 packets, 25M bytes)
>> > > > > > > > >
>> > > > > > > > > pkts bytes target prot opt in out source
>> > > > > > > > > destination
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > On Mon, Sep 28, 2020 at 12:05 PM Pearl d'Silva <
>> > > > > > > > pearl.dsilva@shapeblue.com
>> > > > > > > > > >
>> > > > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > > Hi Hean,
>> > > > > > > > > >
>> > > > > > > > > > In an Advanced Zone with Security Groups enabled, by
>> > default,
>> > > > > > egress
>> > > > > > > > > > traffic from the VM is allowed, while Ingress traffic is
>> > > > denied.
>> > > > > > > Hence,
>> > > > > > > > > as
>> > > > > > > > > > you rightly mentioned, security group rules are added
>> > > > > accordingly.
>> > > > > > > > These
>> > > > > > > > > > rules get added on the hypervisor host, and you can
>> verify
>> > > > them,
>> > > > > by
>> > > > > > > > going
>> > > > > > > > > > into the host and searching for iptables rules
>> > corresponding
>> > > to
>> > > > > the
>> > > > > > > VM
>> > > > > > > > > > (internal name - i-x-y-VM).
>> > > > > > > > > > This blog maybe helpful in providing further details:
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://shankerbalan.net/blog/cloudstack-advanced-zone-with-security-groups/
>> > > > > > > > > >
>> > > > > > > > > > Thanks,
>> > > > > > > > > > Pearl
>> > > > > > > > > > ________________________________
>> > > > > > > > > > From: Hean Seng <he...@gmail.com>
>> > > > > > > > > > Sent: Sunday, September 27, 2020 2:48 PM
>> > > > > > > > > > To: users@cloudstack.apache.org <
>> > users@cloudstack.apache.org
>> > > >
>> > > > > > > > > > Subject: Cloudstack Advance with Security Group
>> > > > > > > > > >
>> > > > > > > > > > Hi
>> > > > > > > > > >
>> > > > > > > > > > I created advance zone with security group, all working
>> > fine.
>> > > > > > > > > >
>> > > > > > > > > > But VMcreated ， seems the default security group that
>> > > assigned
>> > > > to
>> > > > > > the
>> > > > > > > > VM.
>> > > > > > > > > > all accept policy , i understand is Default Deny, and
>> once
>> > > add
>> > > > > in
>> > > > > > > the
>> > > > > > > > > port
>> > > > > > > > > > in Security Group Ingress and Egress, only is allowed
>> > > > > > > > > >
>> > > > > > > > > > Also, is this rules created at VirtualRouter of the
>> > > > > SharedNetwork,
>> > > > > > or
>> > > > > > > > at
>> > > > > > > > > > the Hypervisor?
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > --
>> > > > > > > > > > Regards,
>> > > > > > > > > > Hean Seng
>> > > > > > > > > >
>> > > > > > > > > > pearl.dsilva@shapeblue.com
>> > > > > > > > > > www.shapeblue.com<http://www.shapeblue.com>
>> > > > > > > > > > 3 London Bridge Street, 3rd floor, News Building,
>> London
>> > > SE1
>> > > > > > 9SGUK
>> > > > > > > > > > @shapeblue
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > --
>> > > > > > > > > Regards,
>> > > > > > > > > Hean Seng
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > --
>> > > > > > > Regards,
>> > > > > > > Hean Seng
>> > > > > > >
>> > > > > >
>> > > > >
>> > > > >
>> > > > > --
>> > > > > Regards,
>> > > > > Hean Seng
>> > > > >
>> > > >
>> > >
>> > >
>> > > --
>> > > Regards,
>> > > Hean Seng
>> > >
>> >
>>
>>
>> --
>> Regards,
>> Hean Seng
>>
>
>
> --
> Regards,
> Hean Seng
>
--
Regards,
Hean Seng
Re: Cloudstack Advance with Security Group
Posted by Hean Seng <he...@gmail.com>.
The error when running manually:
]# /usr/share/cloudstack-common/scripts/vm/network/security_group.py
Traceback (most recent call last):
File "/usr/share/cloudstack-common/scripts/vm/network/security_group.py",
line 26, in <module>
import libvirt
Package libvirt-4.5.0-33.el7_8.1.x86_64 already installed and latest version
so I guess, cannot runjust like that ?
On Tue, Sep 29, 2020 at 3:08 PM Hean Seng <he...@gmail.com> wrote:
> Hi
>
> I am running on CentOS 7.8.2003
>
> Cloudstack Agent , and KVM hypervisor:
>
> cloudstack-agent-4.14.0.0-1.el7.x86_64
>
> iptables and ebtables aldy install, iptables -L and ebtables -L show
> default rules .
>
> This is dumpxml, seem the bridge is there, the VLAN for Guest is VLAN401,
> which the setup seem right :
>
> <interface type='bridge'>
>
> <mac address='02:00:21:d9:00:01'/>
>
> <source bridge='brem1-401'/>
>
> <bandwidth>
>
> <inbound average='1280' peak='1280'/>
>
> <outbound average='1280' peak='1280'/>
>
> </bandwidth>
>
> <target dev='vnet9'/>
>
> <model type='virtio'/>
>
> <link state='up'/>
>
> <alias name='net0'/>
>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x03'
> function='0x0'/>
>
>
>
> # ifconfig brem1-401
>
> brem1-401: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
>
> inet6 fe80::3824:b2ff:fe09:873f prefixlen 64 scopeid 0x20<link>
>
> ether 04:7d:7b:60:09:66 txqueuelen 1000 (Ethernet)
>
> RX packets 7191638 bytes 333185781 (317.7 MiB)
>
> RX errors 0 dropped 0 overruns 0 frame 0
>
> TX packets 8 bytes 656 (656.0 B)
>
> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
>
> brctl show
>
> bridge name bridge id STP enabled interfaces
>
> brem1-401 8000.047d7b600966 no em1.401
>
> vnet10
>
> vnet11
>
> vnet12
>
> vnet14
>
> vnet4
>
> vnet8
>
> vnet9
>
> cloud0 8000.fe00a9fe1cea no vnet13
>
> vnet15
>
> vnet2
>
> vnet6
>
> cloudbr0 8000.047d7b600966 yes em1
>
> vnet3
>
> vnet5
>
> vnet7
>
> cloudbr1 8000.000000000000 yes
>
>
>
> I always have /var/log/cloudstack/agent/ this path , just inside
>
> ls /var/log/cloudstack/agent/
>
> agent.log agent.log.2020-09-26.gz agent.log.2020-09-27.gz
> agent.log.2020-09-28.gz resizevolume.log setup.log
>
> it doesnt have securitygroup folder/ log, all the log ip pasted is at
> agent.log
>
>
>
>
> On Tue, Sep 29, 2020 at 2:44 PM Pearl d'Silva <pe...@shapeblue.com>
> wrote:
>
>> Hi Hean,
>>
>> I was able to deploy a VM in a shared network in an advanced zone with
>> security groups enabled and could see the iptables rules getting added for
>> the VM on the host, so it is probably not a bug that we are seeing. Seems
>> like a setup issue.
>> Applying the rules can fail because:
>>
>> * The hypervisor host doesn't have iptables / ebtables command -
>> which is probably unlikely
>> * The VM deployed doesn't have an interface - can be verified by
>> dumping its configuration - virsh dumpxml <domain_name> | grep interface
>> * or application of the default network rules for the VM itself
>> failed -If you are up for some debugging, can you try running the script
>> manually in the hypervisor host:
>>
>> /usr/share/cloudstack-common/scripts/vm/network/security_group.py
>> default_network_rules --vmname <domain_name> --vmid <vm_id> --vmip <vm_ip>
>> --vmmac <vm_mac> --vif <device/networkname> --brname <bridge_name>
>> --nicsecips 0; --isFirstNic --check
>>
>> when you do a virsh dumpxml <vm_domain_name>, look out for an interface
>> section that looks something like:
>>
>> <interface type='bridge'>
>> <mac address='1e:00:86:00:00:ba'/>
>> <source bridge='breth1-11'/>
>> <bandwidth>
>> <inbound average='25600' peak='25600'/>
>> <outbound average='25600' peak='25600'/>
>> </bandwidth>
>> <target dev='vnet0'/>
>> <model type='virtio'/>
>> <link state='up'/>
>> <alias name='net0'/>
>> <address type='pci' domain='0x0000' bus='0x00' slot='0x03'
>> function='0x0'/>
>> </interface>
>>
>>
>> the inputs to the script can be found as follows:
>> domain_name - internal name of your vm (i-x-y-VM); can also be obtained
>> by "virsh list"
>> vm_id - the 'y' part of the internal name : i-x-y-VM
>> vm_ip - Now that debug logs have been enabled, you can search for
>> "StartCommand" in your logs get the ip from the 'nics' section or you could
>> choose any ip from the shared network range
>> vm_mac - from the above bridge section, you will be able to get the MAC
>> address
>> vif / device name - taget dev name - in the above case - vnet0
>> bridge_name - again can be obtained from the above snippet (eg, breth1-11)
>> if there are no secondary ips, then pass 0 for nicsecips otherwise
>> specify the seconday ips separated by a semicolon (;)
>>
>> Basically, at the end of it, it should look something like:
>> /usr/share/cloudstack-common/scripts/vm/network/security_group.py
>> default_network_rules --vmname i-2-3-VM --vmid 3 --vmip <ip> --vmmac
>> 1e:00:86:00:00:ba --vif vnet0 --brname breth1-11 --nicsecips 0; --isFirstNic
>>
>> Hopefully, now you should be able to see a security_group.log file in the
>> /var/log/cloudstack/agent/ path.
>>
>> Thanks,
>> Pearl
>>
>>
>>
>>
>>
>> ________________________________
>> From: Hean Seng <he...@gmail.com>
>> Sent: Tuesday, September 29, 2020 10:45 AM
>> To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
>> Subject: Re: Cloudstack Advance with Security Group
>>
>> Hi Pearl,
>>
>> I had change to debug, following is the error captured:
>>
>>
>> 2020-09-29 01:13:30,090 DEBUG [cloud.agent.Agent]
>> (agentRequest-Handler-2:null) (logid:388b92e0) Processing command:
>> com.cloud.agent.api.SecurityGroupRulesCmd
>>
>> 2020-09-29 01:13:30,090 DEBUG [kvm.resource.LibvirtConnection]
>> (agentRequest-Handler-2:null) (logid:388b92e0) Looking for libvirtd
>> connection at: qemu:///system
>>
>> 2020-09-29 01:13:30,094 DEBUG [kvm.resource.LibvirtComputingResource]
>> (agentRequest-Handler-2:null) (logid:388b92e0) Checking default network
>> rules for vm i-23-77-VM
>>
>> 2020-09-29 01:13:30,094 ERROR [kvm.resource.LibvirtComputingResource]
>> (agentRequest-Handler-2:null) (logid:388b92e0) Unable to apply default
>> network rule for nic cloudbr0 for VM i-23-77-VM
>>
>> 2020-09-29 01:13:30,094 WARN
>> [resource.wrapper.LibvirtSecurityGroupRulesCommandWrapper]
>> (agentRequest-Handler-2:null) (logid:388b92e0) Failed to program default
>> network rules for vm i-23-77-VM
>>
>> 2020-09-29 01:13:30,095 DEBUG [cloud.agent.Agent]
>> (agentRequest-Handler-2:null) (logid:388b92e0) Seq
>> 11-6057904448766738456: {
>> Ans: , MgmtId: 72752492851638, via: 11, Ver: v1, Flags: 110,
>>
>> [{"com.cloud.agent.api.SecurityGroupRuleAnswer":{"logSequenceNumber":15,"vmId":77,"reason":"PROGRAMMING_FAILED","result":false,"details":"programming
>> default network rules failed","wait":0}}] }
>>
>>
>>
>>
>>
>> On Tue, Sep 29, 2020 at 12:38 PM Pearl d'Silva <
>> pearl.dsilva@shapeblue.com>
>> wrote:
>>
>> > Hi Hean,
>> >
>> >
>> > Could you try enabling debug logging on your hypervisor hosts, by
>> running
>> > the following:
>> > *sed -i 's/INFO/DEBUG/g' /etc/cloudstack/agent/log4j-cloud.xml*
>> >
>> > And then restart the cloudstack-agent service on the hosts. Post this,
>> try
>> > deploying a VM in the shared network. This may give a better insight
>> into
>> > where exactly the issue lies.
>> >
>> > Thanks,
>> > Pearl
>> >
>> >
>> > Software Engineer
>> > pearl.dsilva@shapeblue.com
>> > www.shapeblue.com<http://www.shapeblue.com>
>> >
>> >
>> >
>> >
>> >
>> >
>> > ------------------------------
>> > *From:* Ivan Kudryavtsev <iv...@bw-sw.com>
>> > *Sent:* Monday, September 28, 2020 2:17 PM
>> > *To:* users <us...@cloudstack.apache.org>
>> > *Subject:* Re: Cloudstack Advance with Security Group
>> >
>> > This is it.
>> >
>>
>> pearl.dsilva@shapeblue.com
>> www.shapeblue.com
>> 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK
>> @shapeblue
>>
>>
>>
>> > On Mon, Sep 28, 2020 at 3:46 PM Hean Seng <he...@gmail.com> wrote:
>> >
>> > > In the log, I cannot see anything much, except a few lines showing
>> above
>> > > . Not sure if this is a bug on 4.14.
>> > >
>> > >
>> > >
>> > > 2020-09-28 03:53:36,585 ERROR [kvm.resource.LibvirtComputingResource]
>> > > (agentRequest-Handler-4:null) (logid:b6a4c077) Unable to apply default
>> > > network rule for nic cloudbr0 for VM i-2-81-VM
>> > >
>> > > 2020-09-28 03:53:36,858 ERROR [kvm.resource.LibvirtComputingResource]
>> > > (agentRequest-Handler-3:null) (logid:74058678) Unable to apply default
>> > > network rule for nic cloudbr0 for VM i-2-81-VM
>> > >
>> > > 2020-09-28 03:53:36,858 WARN
>> > > [resource.wrapper.LibvirtSecurityGroupRulesCommandWrapper]
>> > > (agentRequest-Handler-3:null) (logid:74058678) Failed to program
>> default
>> > > network rules for vm i-2-81-VM
>> > >
>> > >
>> > >
>> > >
>> > > On Mon, Sep 28, 2020 at 4:34 PM Ivan Kudryavtsev <iv...@bw-sw.com>
>> wrote:
>> > >
>> > > > Hi,
>> > > > no I'm on 4.11, so can not help with exact 4.14, and I'm on Ubuntu,
>> > > though,
>> > > > but for any KVM hypervisor Linux distribution, the logic is the
>> same.
>> > > >
>> > > > On Mon, Sep 28, 2020 at 3:31 PM Hean Seng <he...@gmail.com>
>> wrote:
>> > > >
>> > > > > Hi
>> > > > >
>> > > > > Are you running on CentOS7 ?
>> > > > >
>> > > > > I am running on CentOS7 , ACS 4.14 , and seem there is no log
>> at
>> > of
>> > > > > security_group.log
>> > > > >
>> > > > > # ls /var/log/cloudstack/agent/
>> > > > >
>> > > > > agent.log resizevolume.log setup.log
>> > > > >
>> > > > >
>> > > > > I recheck back the Intall guide, seems no missing anything.
>> > > > >
>> > > > >
>> > > > > Older intallation guide, 4.11 mentioned need , allow
>> > > > > /usr/lib/sysctl.d/00-system.conf
>> > > > >
>> > > > > # Enable netfilter on bridges.
>> net.bridge.bridge-nf-call-ip6tables =
>> > 1
>> > > > > net.bridge.bridge-nf-call-iptables = 1
>> > > > net.bridge.bridge-nf-call-arptables
>> > > > > = 1
>> > > > >
>> > > > > And it has been done too.
>> > > > >
>> > > > >
>> > > > >
>> > > > > On Mon, Sep 28, 2020 at 4:05 PM Ivan Kudryavtsev <iv...@bw-sw.com>
>> > > wrote:
>> > > > >
>> > > > > > Hi,
>> > > > > >
>> > > > > > No, this is not the issue.
>> > > > > > It's a normal state of the system, as KVM hooks are a new and
>> > > optional
>> > > > > > feature of 4.14.
>> > > > > >
>> > > > > > You should find some sort of messages regarding security_groups
>> at
>> > > > > > /var/log/cloudstack/agent/security_group.log
>> > > > > >
>> > > > > >
>> > > > > > On Mon, Sep 28, 2020 at 2:10 PM Hean Seng <he...@gmail.com>
>> > > wrote:
>> > > > > >
>> > > > > > > I not sure where goes wrong, are you running on CentOS 7 ? I
>> > have
>> > > > this
>> > > > > > > error too, do you think is this contribute to the error as
>> well:
>> > > > > > >
>> > > > > > > 2020-09-28 03:04:52,762 WARN
>> [kvm.resource.LibvirtKvmAgentHook]
>> > > > > > > (agentRequest-Handler-5:null) (logid:4f23845b) Groovy script
>> > > > > > >
>> '/etc/cloudstack/agent/hooks/libvirt-vm-xml-transformer.groovy'
>> > is
>> > > > not
>> > > > > > > available. Transformations will not be applied.
>> > > > > > >
>> > > > > > > 2020-09-28 03:04:52,762 WARN
>> [kvm.resource.LibvirtKvmAgentHook]
>> > > > > > > (agentRequest-Handler-5:null) (logid:4f23845b) Groovy
>> scripting
>> > > > engine
>> > > > > is
>> > > > > > > not initialized. Data transformation skipped.
>> > > > > > >
>> > > > > > > 2020-09-28 03:04:53,083 WARN
>> [kvm.resource.LibvirtKvmAgentHook]
>> > > > > > > (agentRequest-Handler-5:null) (logid:4f23845b) Groovy script
>> > > > > > > '/etc/cloudstack/agent/hooks/libvirt-vm-state-change.groovy'
>> is
>> > not
>> > > > > > > available. Transformations will not be applied.
>> > > > > > >
>> > > > > > > On Mon, Sep 28, 2020 at 2:27 PM Ivan Kudryavtsev <
>> ivan@bw-sw.com
>> > >
>> > > > > wrote:
>> > > > > > >
>> > > > > > > > This just means you installed it in the wrong way. Ebtables
>> and
>> > > > > > Iptables
>> > > > > > > > must be filled with rules like
>> > > > > > > >
>> > > > > > > > -A i-6242-10304-def -m state --state RELATED,ESTABLISHED -j
>> > > ACCEPT
>> > > > > > > > -A i-6242-10304-def -p udp -m physdev --physdev-in vnet18
>> > > > > > > > --physdev-is-bridged -m udp --sport 68 --dport 67 -j ACCEPT
>> > > > > > > > -A i-6242-10304-def -p udp -m physdev --physdev-out vnet18
>> > > > > > > > --physdev-is-bridged -m udp --sport 67 --dport 68 -j ACCEPT
>> > > > > > > > -A i-6242-10304-def -m physdev --physdev-in vnet18
>> > > > > --physdev-is-bridged
>> > > > > > > -m
>> > > > > > > > set ! --match-set i-6242-10304-vm src -j DROP
>> > > > > > > > -A i-6242-10304-def -p udp -m physdev --physdev-in vnet18
>> > > > > > > > --physdev-is-bridged -m set --match-set i-6242-10304-vm src
>> -m
>> > > udp
>> > > > > > > --dport
>> > > > > > > > 53 -j RETURN
>> > > > > > > > -A i-6242-10304-def -p tcp -m physdev --physdev-in vnet18
>> > > > > > > > --physdev-is-bridged -m set --match-set i-6242-10304-vm src
>> -m
>> > > tcp
>> > > > > > > --dport
>> > > > > > > > 53 -j RETURN
>> > > > > > > > -A i-6242-10304-def -m physdev --physdev-in vnet18
>> > > > > --physdev-is-bridged
>> > > > > > > -m
>> > > > > > > > set --match-set i-6242-10304-vm src -j i-6242-10304-vm-eg
>> > > > > > > > -A i-6242-10304-def -m physdev --physdev-out vnet18
>> > > > > > --physdev-is-bridged
>> > > > > > > -j
>> > > > > > > > i-6242-10304-vm
>> > > > > > > > -A i-6242-10304-vm -p udp -m udp --dport 1:65535 -m state
>> > --state
>> > > > NEW
>> > > > > > -j
>> > > > > > > > ACCEPT
>> > > > > > > > -A i-6242-10304-vm -p tcp -m tcp --dport 1:65535 -m state
>> > --state
>> > > > NEW
>> > > > > > -j
>> > > > > > > > ACCEPT
>> > > > > > > > -A i-6242-10304-vm -p icmp -m icmp --icmp-type any -j ACCEPT
>> > > > > > > > -A i-6242-10304-vm -j DROP
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > Bridge chain: i-4435-8929-vm-in, entries: 7, policy: ACCEPT
>> > > > > > > > -s ! 1e:0:32:0:2:2 -j DROP
>> > > > > > > > -p ARP -s ! 1e:0:32:0:2:2 -j DROP
>> > > > > > > > -p ARP --arp-mac-src ! 1e:0:32:0:2:2 -j DROP
>> > > > > > > > -p ARP -j i-4435-8929-vm-in-ips
>> > > > > > > > -p ARP --arp-op Request -j ACCEPT
>> > > > > > > > -p ARP --arp-op Reply -j ACCEPT
>> > > > > > > > -p ARP -j DROP
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > On Mon, Sep 28, 2020 at 1:10 PM Hean Seng <
>> heanseng@gmail.com>
>> > > > > wrote:
>> > > > > > > >
>> > > > > > > > > I checked the hypervisor , it seems iptables is nothing
>> > inside
>> > > ,
>> > > > > > this
>> > > > > > > is
>> > > > > > > > > centos7 , initially i turnoff firewalld , but even i
>> turn
>> > on
>> > > it
>> > > > > now
>> > > > > > > and
>> > > > > > > > > try to update the security group rules, it seems empty
>> > iptable
>> > > > > rules
>> > > > > > :
>> > > > > > > > >
>> > > > > > > > > [root@kvm03 ~]# iptables -L -v -n
>> > > > > > > > >
>> > > > > > > > > Chain INPUT (policy ACCEPT 82903 packets, 1170M bytes)
>> > > > > > > > >
>> > > > > > > > > pkts bytes target prot opt in out source
>> > > > > > > > > destination
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
>> > > > > > > > >
>> > > > > > > > > pkts bytes target prot opt in out source
>> > > > > > > > > destination
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > Chain OUTPUT (policy ACCEPT 80505 packets, 25M bytes)
>> > > > > > > > >
>> > > > > > > > > pkts bytes target prot opt in out source
>> > > > > > > > > destination
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > On Mon, Sep 28, 2020 at 12:05 PM Pearl d'Silva <
>> > > > > > > > pearl.dsilva@shapeblue.com
>> > > > > > > > > >
>> > > > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > > Hi Hean,
>> > > > > > > > > >
>> > > > > > > > > > In an Advanced Zone with Security Groups enabled, by
>> > default,
>> > > > > > egress
>> > > > > > > > > > traffic from the VM is allowed, while Ingress traffic is
>> > > > denied.
>> > > > > > > Hence,
>> > > > > > > > > as
>> > > > > > > > > > you rightly mentioned, security group rules are added
>> > > > > accordingly.
>> > > > > > > > These
>> > > > > > > > > > rules get added on the hypervisor host, and you can
>> verify
>> > > > them,
>> > > > > by
>> > > > > > > > going
>> > > > > > > > > > into the host and searching for iptables rules
>> > corresponding
>> > > to
>> > > > > the
>> > > > > > > VM
>> > > > > > > > > > (internal name - i-x-y-VM).
>> > > > > > > > > > This blog maybe helpful in providing further details:
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://shankerbalan.net/blog/cloudstack-advanced-zone-with-security-groups/
>> > > > > > > > > >
>> > > > > > > > > > Thanks,
>> > > > > > > > > > Pearl
>> > > > > > > > > > ________________________________
>> > > > > > > > > > From: Hean Seng <he...@gmail.com>
>> > > > > > > > > > Sent: Sunday, September 27, 2020 2:48 PM
>> > > > > > > > > > To: users@cloudstack.apache.org <
>> > users@cloudstack.apache.org
>> > > >
>> > > > > > > > > > Subject: Cloudstack Advance with Security Group
>> > > > > > > > > >
>> > > > > > > > > > Hi
>> > > > > > > > > >
>> > > > > > > > > > I created advance zone with security group, all working
>> > fine.
>> > > > > > > > > >
>> > > > > > > > > > But VMcreated ， seems the default security group that
>> > > assigned
>> > > > to
>> > > > > > the
>> > > > > > > > VM.
>> > > > > > > > > > all accept policy , i understand is Default Deny, and
>> once
>> > > add
>> > > > > in
>> > > > > > > the
>> > > > > > > > > port
>> > > > > > > > > > in Security Group Ingress and Egress, only is allowed
>> > > > > > > > > >
>> > > > > > > > > > Also, is this rules created at VirtualRouter of the
>> > > > > SharedNetwork,
>> > > > > > or
>> > > > > > > > at
>> > > > > > > > > > the Hypervisor?
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > --
>> > > > > > > > > > Regards,
>> > > > > > > > > > Hean Seng
>> > > > > > > > > >
>> > > > > > > > > > pearl.dsilva@shapeblue.com
>> > > > > > > > > > www.shapeblue.com<http://www.shapeblue.com>
>> > > > > > > > > > 3 London Bridge Street, 3rd floor, News Building,
>> London
>> > > SE1
>> > > > > > 9SGUK
>> > > > > > > > > > @shapeblue
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > --
>> > > > > > > > > Regards,
>> > > > > > > > > Hean Seng
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > --
>> > > > > > > Regards,
>> > > > > > > Hean Seng
>> > > > > > >
>> > > > > >
>> > > > >
>> > > > >
>> > > > > --
>> > > > > Regards,
>> > > > > Hean Seng
>> > > > >
>> > > >
>> > >
>> > >
>> > > --
>> > > Regards,
>> > > Hean Seng
>> > >
>> >
>>
>>
>> --
>> Regards,
>> Hean Seng
>>
>
>
> --
> Regards,
> Hean Seng
>
--
Regards,
Hean Seng
Re: Cloudstack Advance with Security Group
Posted by Hean Seng <he...@gmail.com>.
Hi
I am running on CentOS 7.8.2003
Cloudstack Agent , and KVM hypervisor:
cloudstack-agent-4.14.0.0-1.el7.x86_64
iptables and ebtables aldy install, iptables -L and ebtables -L show
default rules .
This is dumpxml, seem the bridge is there, the VLAN for Guest is VLAN401,
which the setup seem right :
<interface type='bridge'>
<mac address='02:00:21:d9:00:01'/>
<source bridge='brem1-401'/>
<bandwidth>
<inbound average='1280' peak='1280'/>
<outbound average='1280' peak='1280'/>
</bandwidth>
<target dev='vnet9'/>
<model type='virtio'/>
<link state='up'/>
<alias name='net0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03'
function='0x0'/>
# ifconfig brem1-401
brem1-401: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet6 fe80::3824:b2ff:fe09:873f prefixlen 64 scopeid 0x20<link>
ether 04:7d:7b:60:09:66 txqueuelen 1000 (Ethernet)
RX packets 7191638 bytes 333185781 (317.7 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 8 bytes 656 (656.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
brctl show
bridge name bridge id STP enabled interfaces
brem1-401 8000.047d7b600966 no em1.401
vnet10
vnet11
vnet12
vnet14
vnet4
vnet8
vnet9
cloud0 8000.fe00a9fe1cea no vnet13
vnet15
vnet2
vnet6
cloudbr0 8000.047d7b600966 yes em1
vnet3
vnet5
vnet7
cloudbr1 8000.000000000000 yes
I always have /var/log/cloudstack/agent/ this path , just inside
ls /var/log/cloudstack/agent/
agent.log agent.log.2020-09-26.gz agent.log.2020-09-27.gz
agent.log.2020-09-28.gz resizevolume.log setup.log
it doesnt have securitygroup folder/ log, all the log ip pasted is at
agent.log
On Tue, Sep 29, 2020 at 2:44 PM Pearl d'Silva <pe...@shapeblue.com>
wrote:
> Hi Hean,
>
> I was able to deploy a VM in a shared network in an advanced zone with
> security groups enabled and could see the iptables rules getting added for
> the VM on the host, so it is probably not a bug that we are seeing. Seems
> like a setup issue.
> Applying the rules can fail because:
>
> * The hypervisor host doesn't have iptables / ebtables command - which
> is probably unlikely
> * The VM deployed doesn't have an interface - can be verified by
> dumping its configuration - virsh dumpxml <domain_name> | grep interface
> * or application of the default network rules for the VM itself failed
> -If you are up for some debugging, can you try running the script manually
> in the hypervisor host:
>
> /usr/share/cloudstack-common/scripts/vm/network/security_group.py
> default_network_rules --vmname <domain_name> --vmid <vm_id> --vmip <vm_ip>
> --vmmac <vm_mac> --vif <device/networkname> --brname <bridge_name>
> --nicsecips 0; --isFirstNic --check
>
> when you do a virsh dumpxml <vm_domain_name>, look out for an interface
> section that looks something like:
>
> <interface type='bridge'>
> <mac address='1e:00:86:00:00:ba'/>
> <source bridge='breth1-11'/>
> <bandwidth>
> <inbound average='25600' peak='25600'/>
> <outbound average='25600' peak='25600'/>
> </bandwidth>
> <target dev='vnet0'/>
> <model type='virtio'/>
> <link state='up'/>
> <alias name='net0'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x03'
> function='0x0'/>
> </interface>
>
>
> the inputs to the script can be found as follows:
> domain_name - internal name of your vm (i-x-y-VM); can also be obtained by
> "virsh list"
> vm_id - the 'y' part of the internal name : i-x-y-VM
> vm_ip - Now that debug logs have been enabled, you can search for
> "StartCommand" in your logs get the ip from the 'nics' section or you could
> choose any ip from the shared network range
> vm_mac - from the above bridge section, you will be able to get the MAC
> address
> vif / device name - taget dev name - in the above case - vnet0
> bridge_name - again can be obtained from the above snippet (eg, breth1-11)
> if there are no secondary ips, then pass 0 for nicsecips otherwise specify
> the seconday ips separated by a semicolon (;)
>
> Basically, at the end of it, it should look something like:
> /usr/share/cloudstack-common/scripts/vm/network/security_group.py
> default_network_rules --vmname i-2-3-VM --vmid 3 --vmip <ip> --vmmac
> 1e:00:86:00:00:ba --vif vnet0 --brname breth1-11 --nicsecips 0; --isFirstNic
>
> Hopefully, now you should be able to see a security_group.log file in the
> /var/log/cloudstack/agent/ path.
>
> Thanks,
> Pearl
>
>
>
>
>
> ________________________________
> From: Hean Seng <he...@gmail.com>
> Sent: Tuesday, September 29, 2020 10:45 AM
> To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> Subject: Re: Cloudstack Advance with Security Group
>
> Hi Pearl,
>
> I had change to debug, following is the error captured:
>
>
> 2020-09-29 01:13:30,090 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-2:null) (logid:388b92e0) Processing command:
> com.cloud.agent.api.SecurityGroupRulesCmd
>
> 2020-09-29 01:13:30,090 DEBUG [kvm.resource.LibvirtConnection]
> (agentRequest-Handler-2:null) (logid:388b92e0) Looking for libvirtd
> connection at: qemu:///system
>
> 2020-09-29 01:13:30,094 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-2:null) (logid:388b92e0) Checking default network
> rules for vm i-23-77-VM
>
> 2020-09-29 01:13:30,094 ERROR [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-2:null) (logid:388b92e0) Unable to apply default
> network rule for nic cloudbr0 for VM i-23-77-VM
>
> 2020-09-29 01:13:30,094 WARN
> [resource.wrapper.LibvirtSecurityGroupRulesCommandWrapper]
> (agentRequest-Handler-2:null) (logid:388b92e0) Failed to program default
> network rules for vm i-23-77-VM
>
> 2020-09-29 01:13:30,095 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-2:null) (logid:388b92e0) Seq
> 11-6057904448766738456: {
> Ans: , MgmtId: 72752492851638, via: 11, Ver: v1, Flags: 110,
>
> [{"com.cloud.agent.api.SecurityGroupRuleAnswer":{"logSequenceNumber":15,"vmId":77,"reason":"PROGRAMMING_FAILED","result":false,"details":"programming
> default network rules failed","wait":0}}] }
>
>
>
>
>
> On Tue, Sep 29, 2020 at 12:38 PM Pearl d'Silva <pearl.dsilva@shapeblue.com
> >
> wrote:
>
> > Hi Hean,
> >
> >
> > Could you try enabling debug logging on your hypervisor hosts, by running
> > the following:
> > *sed -i 's/INFO/DEBUG/g' /etc/cloudstack/agent/log4j-cloud.xml*
> >
> > And then restart the cloudstack-agent service on the hosts. Post this,
> try
> > deploying a VM in the shared network. This may give a better insight into
> > where exactly the issue lies.
> >
> > Thanks,
> > Pearl
> >
> >
> > Software Engineer
> > pearl.dsilva@shapeblue.com
> > www.shapeblue.com<http://www.shapeblue.com>
> >
> >
> >
> >
> >
> >
> > ------------------------------
> > *From:* Ivan Kudryavtsev <iv...@bw-sw.com>
> > *Sent:* Monday, September 28, 2020 2:17 PM
> > *To:* users <us...@cloudstack.apache.org>
> > *Subject:* Re: Cloudstack Advance with Security Group
> >
> > This is it.
> >
>
> pearl.dsilva@shapeblue.com
> www.shapeblue.com
> 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK
> @shapeblue
>
>
>
> > On Mon, Sep 28, 2020 at 3:46 PM Hean Seng <he...@gmail.com> wrote:
> >
> > > In the log, I cannot see anything much, except a few lines showing
> above
> > > . Not sure if this is a bug on 4.14.
> > >
> > >
> > >
> > > 2020-09-28 03:53:36,585 ERROR [kvm.resource.LibvirtComputingResource]
> > > (agentRequest-Handler-4:null) (logid:b6a4c077) Unable to apply default
> > > network rule for nic cloudbr0 for VM i-2-81-VM
> > >
> > > 2020-09-28 03:53:36,858 ERROR [kvm.resource.LibvirtComputingResource]
> > > (agentRequest-Handler-3:null) (logid:74058678) Unable to apply default
> > > network rule for nic cloudbr0 for VM i-2-81-VM
> > >
> > > 2020-09-28 03:53:36,858 WARN
> > > [resource.wrapper.LibvirtSecurityGroupRulesCommandWrapper]
> > > (agentRequest-Handler-3:null) (logid:74058678) Failed to program
> default
> > > network rules for vm i-2-81-VM
> > >
> > >
> > >
> > >
> > > On Mon, Sep 28, 2020 at 4:34 PM Ivan Kudryavtsev <iv...@bw-sw.com>
> wrote:
> > >
> > > > Hi,
> > > > no I'm on 4.11, so can not help with exact 4.14, and I'm on Ubuntu,
> > > though,
> > > > but for any KVM hypervisor Linux distribution, the logic is the same.
> > > >
> > > > On Mon, Sep 28, 2020 at 3:31 PM Hean Seng <he...@gmail.com>
> wrote:
> > > >
> > > > > Hi
> > > > >
> > > > > Are you running on CentOS7 ?
> > > > >
> > > > > I am running on CentOS7 , ACS 4.14 , and seem there is no log at
> > of
> > > > > security_group.log
> > > > >
> > > > > # ls /var/log/cloudstack/agent/
> > > > >
> > > > > agent.log resizevolume.log setup.log
> > > > >
> > > > >
> > > > > I recheck back the Intall guide, seems no missing anything.
> > > > >
> > > > >
> > > > > Older intallation guide, 4.11 mentioned need , allow
> > > > > /usr/lib/sysctl.d/00-system.conf
> > > > >
> > > > > # Enable netfilter on bridges. net.bridge.bridge-nf-call-ip6tables
> =
> > 1
> > > > > net.bridge.bridge-nf-call-iptables = 1
> > > > net.bridge.bridge-nf-call-arptables
> > > > > = 1
> > > > >
> > > > > And it has been done too.
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Sep 28, 2020 at 4:05 PM Ivan Kudryavtsev <iv...@bw-sw.com>
> > > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > No, this is not the issue.
> > > > > > It's a normal state of the system, as KVM hooks are a new and
> > > optional
> > > > > > feature of 4.14.
> > > > > >
> > > > > > You should find some sort of messages regarding security_groups
> at
> > > > > > /var/log/cloudstack/agent/security_group.log
> > > > > >
> > > > > >
> > > > > > On Mon, Sep 28, 2020 at 2:10 PM Hean Seng <he...@gmail.com>
> > > wrote:
> > > > > >
> > > > > > > I not sure where goes wrong, are you running on CentOS 7 ? I
> > have
> > > > this
> > > > > > > error too, do you think is this contribute to the error as
> well:
> > > > > > >
> > > > > > > 2020-09-28 03:04:52,762 WARN
> [kvm.resource.LibvirtKvmAgentHook]
> > > > > > > (agentRequest-Handler-5:null) (logid:4f23845b) Groovy script
> > > > > > > '/etc/cloudstack/agent/hooks/libvirt-vm-xml-transformer.groovy'
> > is
> > > > not
> > > > > > > available. Transformations will not be applied.
> > > > > > >
> > > > > > > 2020-09-28 03:04:52,762 WARN
> [kvm.resource.LibvirtKvmAgentHook]
> > > > > > > (agentRequest-Handler-5:null) (logid:4f23845b) Groovy scripting
> > > > engine
> > > > > is
> > > > > > > not initialized. Data transformation skipped.
> > > > > > >
> > > > > > > 2020-09-28 03:04:53,083 WARN
> [kvm.resource.LibvirtKvmAgentHook]
> > > > > > > (agentRequest-Handler-5:null) (logid:4f23845b) Groovy script
> > > > > > > '/etc/cloudstack/agent/hooks/libvirt-vm-state-change.groovy' is
> > not
> > > > > > > available. Transformations will not be applied.
> > > > > > >
> > > > > > > On Mon, Sep 28, 2020 at 2:27 PM Ivan Kudryavtsev <
> ivan@bw-sw.com
> > >
> > > > > wrote:
> > > > > > >
> > > > > > > > This just means you installed it in the wrong way. Ebtables
> and
> > > > > > Iptables
> > > > > > > > must be filled with rules like
> > > > > > > >
> > > > > > > > -A i-6242-10304-def -m state --state RELATED,ESTABLISHED -j
> > > ACCEPT
> > > > > > > > -A i-6242-10304-def -p udp -m physdev --physdev-in vnet18
> > > > > > > > --physdev-is-bridged -m udp --sport 68 --dport 67 -j ACCEPT
> > > > > > > > -A i-6242-10304-def -p udp -m physdev --physdev-out vnet18
> > > > > > > > --physdev-is-bridged -m udp --sport 67 --dport 68 -j ACCEPT
> > > > > > > > -A i-6242-10304-def -m physdev --physdev-in vnet18
> > > > > --physdev-is-bridged
> > > > > > > -m
> > > > > > > > set ! --match-set i-6242-10304-vm src -j DROP
> > > > > > > > -A i-6242-10304-def -p udp -m physdev --physdev-in vnet18
> > > > > > > > --physdev-is-bridged -m set --match-set i-6242-10304-vm src
> -m
> > > udp
> > > > > > > --dport
> > > > > > > > 53 -j RETURN
> > > > > > > > -A i-6242-10304-def -p tcp -m physdev --physdev-in vnet18
> > > > > > > > --physdev-is-bridged -m set --match-set i-6242-10304-vm src
> -m
> > > tcp
> > > > > > > --dport
> > > > > > > > 53 -j RETURN
> > > > > > > > -A i-6242-10304-def -m physdev --physdev-in vnet18
> > > > > --physdev-is-bridged
> > > > > > > -m
> > > > > > > > set --match-set i-6242-10304-vm src -j i-6242-10304-vm-eg
> > > > > > > > -A i-6242-10304-def -m physdev --physdev-out vnet18
> > > > > > --physdev-is-bridged
> > > > > > > -j
> > > > > > > > i-6242-10304-vm
> > > > > > > > -A i-6242-10304-vm -p udp -m udp --dport 1:65535 -m state
> > --state
> > > > NEW
> > > > > > -j
> > > > > > > > ACCEPT
> > > > > > > > -A i-6242-10304-vm -p tcp -m tcp --dport 1:65535 -m state
> > --state
> > > > NEW
> > > > > > -j
> > > > > > > > ACCEPT
> > > > > > > > -A i-6242-10304-vm -p icmp -m icmp --icmp-type any -j ACCEPT
> > > > > > > > -A i-6242-10304-vm -j DROP
> > > > > > > >
> > > > > > > >
> > > > > > > > Bridge chain: i-4435-8929-vm-in, entries: 7, policy: ACCEPT
> > > > > > > > -s ! 1e:0:32:0:2:2 -j DROP
> > > > > > > > -p ARP -s ! 1e:0:32:0:2:2 -j DROP
> > > > > > > > -p ARP --arp-mac-src ! 1e:0:32:0:2:2 -j DROP
> > > > > > > > -p ARP -j i-4435-8929-vm-in-ips
> > > > > > > > -p ARP --arp-op Request -j ACCEPT
> > > > > > > > -p ARP --arp-op Reply -j ACCEPT
> > > > > > > > -p ARP -j DROP
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Mon, Sep 28, 2020 at 1:10 PM Hean Seng <
> heanseng@gmail.com>
> > > > > wrote:
> > > > > > > >
> > > > > > > > > I checked the hypervisor , it seems iptables is nothing
> > inside
> > > ,
> > > > > > this
> > > > > > > is
> > > > > > > > > centos7 , initially i turnoff firewalld , but even i turn
> > on
> > > it
> > > > > now
> > > > > > > and
> > > > > > > > > try to update the security group rules, it seems empty
> > iptable
> > > > > rules
> > > > > > :
> > > > > > > > >
> > > > > > > > > [root@kvm03 ~]# iptables -L -v -n
> > > > > > > > >
> > > > > > > > > Chain INPUT (policy ACCEPT 82903 packets, 1170M bytes)
> > > > > > > > >
> > > > > > > > > pkts bytes target prot opt in out source
> > > > > > > > > destination
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
> > > > > > > > >
> > > > > > > > > pkts bytes target prot opt in out source
> > > > > > > > > destination
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Chain OUTPUT (policy ACCEPT 80505 packets, 25M bytes)
> > > > > > > > >
> > > > > > > > > pkts bytes target prot opt in out source
> > > > > > > > > destination
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Mon, Sep 28, 2020 at 12:05 PM Pearl d'Silva <
> > > > > > > > pearl.dsilva@shapeblue.com
> > > > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Hean,
> > > > > > > > > >
> > > > > > > > > > In an Advanced Zone with Security Groups enabled, by
> > default,
> > > > > > egress
> > > > > > > > > > traffic from the VM is allowed, while Ingress traffic is
> > > > denied.
> > > > > > > Hence,
> > > > > > > > > as
> > > > > > > > > > you rightly mentioned, security group rules are added
> > > > > accordingly.
> > > > > > > > These
> > > > > > > > > > rules get added on the hypervisor host, and you can
> verify
> > > > them,
> > > > > by
> > > > > > > > going
> > > > > > > > > > into the host and searching for iptables rules
> > corresponding
> > > to
> > > > > the
> > > > > > > VM
> > > > > > > > > > (internal name - i-x-y-VM).
> > > > > > > > > > This blog maybe helpful in providing further details:
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://shankerbalan.net/blog/cloudstack-advanced-zone-with-security-groups/
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Pearl
> > > > > > > > > > ________________________________
> > > > > > > > > > From: Hean Seng <he...@gmail.com>
> > > > > > > > > > Sent: Sunday, September 27, 2020 2:48 PM
> > > > > > > > > > To: users@cloudstack.apache.org <
> > users@cloudstack.apache.org
> > > >
> > > > > > > > > > Subject: Cloudstack Advance with Security Group
> > > > > > > > > >
> > > > > > > > > > Hi
> > > > > > > > > >
> > > > > > > > > > I created advance zone with security group, all working
> > fine.
> > > > > > > > > >
> > > > > > > > > > But VMcreated ， seems the default security group that
> > > assigned
> > > > to
> > > > > > the
> > > > > > > > VM.
> > > > > > > > > > all accept policy , i understand is Default Deny, and
> once
> > > add
> > > > > in
> > > > > > > the
> > > > > > > > > port
> > > > > > > > > > in Security Group Ingress and Egress, only is allowed
> > > > > > > > > >
> > > > > > > > > > Also, is this rules created at VirtualRouter of the
> > > > > SharedNetwork,
> > > > > > or
> > > > > > > > at
> > > > > > > > > > the Hypervisor?
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > Regards,
> > > > > > > > > > Hean Seng
> > > > > > > > > >
> > > > > > > > > > pearl.dsilva@shapeblue.com
> > > > > > > > > > www.shapeblue.com<http://www.shapeblue.com>
> > > > > > > > > > 3 London Bridge Street, 3rd floor, News Building, London
> > > SE1
> > > > > > 9SGUK
> > > > > > > > > > @shapeblue
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Regards,
> > > > > > > > > Hean Seng
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Regards,
> > > > > > > Hean Seng
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Regards,
> > > > > Hean Seng
> > > > >
> > > >
> > >
> > >
> > > --
> > > Regards,
> > > Hean Seng
> > >
> >
>
>
> --
> Regards,
> Hean Seng
>
--
Regards,
Hean Seng
Re: Cloudstack Advance with Security Group
Posted by Pearl d'Silva <pe...@shapeblue.com>.
Hi Hean,
I was able to deploy a VM in a shared network in an advanced zone with security groups enabled and could see the iptables rules getting added for the VM on the host, so it is probably not a bug that we are seeing. Seems like a setup issue.
Applying the rules can fail because:
* The hypervisor host doesn't have iptables / ebtables command - which is probably unlikely
* The VM deployed doesn't have an interface - can be verified by dumping its configuration - virsh dumpxml <domain_name> | grep interface
* or application of the default network rules for the VM itself failed -If you are up for some debugging, can you try running the script manually in the hypervisor host:
/usr/share/cloudstack-common/scripts/vm/network/security_group.py default_network_rules --vmname <domain_name> --vmid <vm_id> --vmip <vm_ip> --vmmac <vm_mac> --vif <device/networkname> --brname <bridge_name> --nicsecips 0; --isFirstNic --check
when you do a virsh dumpxml <vm_domain_name>, look out for an interface section that looks something like:
<interface type='bridge'>
<mac address='1e:00:86:00:00:ba'/>
<source bridge='breth1-11'/>
<bandwidth>
<inbound average='25600' peak='25600'/>
<outbound average='25600' peak='25600'/>
</bandwidth>
<target dev='vnet0'/>
<model type='virtio'/>
<link state='up'/>
<alias name='net0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
the inputs to the script can be found as follows:
domain_name - internal name of your vm (i-x-y-VM); can also be obtained by "virsh list"
vm_id - the 'y' part of the internal name : i-x-y-VM
vm_ip - Now that debug logs have been enabled, you can search for "StartCommand" in your logs get the ip from the 'nics' section or you could choose any ip from the shared network range
vm_mac - from the above bridge section, you will be able to get the MAC address
vif / device name - taget dev name - in the above case - vnet0
bridge_name - again can be obtained from the above snippet (eg, breth1-11)
if there are no secondary ips, then pass 0 for nicsecips otherwise specify the seconday ips separated by a semicolon (;)
Basically, at the end of it, it should look something like:
/usr/share/cloudstack-common/scripts/vm/network/security_group.py default_network_rules --vmname i-2-3-VM --vmid 3 --vmip <ip> --vmmac 1e:00:86:00:00:ba --vif vnet0 --brname breth1-11 --nicsecips 0; --isFirstNic
Hopefully, now you should be able to see a security_group.log file in the /var/log/cloudstack/agent/ path.
Thanks,
Pearl
________________________________
From: Hean Seng <he...@gmail.com>
Sent: Tuesday, September 29, 2020 10:45 AM
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: Re: Cloudstack Advance with Security Group
Hi Pearl,
I had change to debug, following is the error captured:
2020-09-29 01:13:30,090 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-2:null) (logid:388b92e0) Processing command:
com.cloud.agent.api.SecurityGroupRulesCmd
2020-09-29 01:13:30,090 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-2:null) (logid:388b92e0) Looking for libvirtd
connection at: qemu:///system
2020-09-29 01:13:30,094 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-2:null) (logid:388b92e0) Checking default network
rules for vm i-23-77-VM
2020-09-29 01:13:30,094 ERROR [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-2:null) (logid:388b92e0) Unable to apply default
network rule for nic cloudbr0 for VM i-23-77-VM
2020-09-29 01:13:30,094 WARN
[resource.wrapper.LibvirtSecurityGroupRulesCommandWrapper]
(agentRequest-Handler-2:null) (logid:388b92e0) Failed to program default
network rules for vm i-23-77-VM
2020-09-29 01:13:30,095 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-2:null) (logid:388b92e0) Seq 11-6057904448766738456: {
Ans: , MgmtId: 72752492851638, via: 11, Ver: v1, Flags: 110,
[{"com.cloud.agent.api.SecurityGroupRuleAnswer":{"logSequenceNumber":15,"vmId":77,"reason":"PROGRAMMING_FAILED","result":false,"details":"programming
default network rules failed","wait":0}}] }
On Tue, Sep 29, 2020 at 12:38 PM Pearl d'Silva <pe...@shapeblue.com>
wrote:
> Hi Hean,
>
>
> Could you try enabling debug logging on your hypervisor hosts, by running
> the following:
> *sed -i 's/INFO/DEBUG/g' /etc/cloudstack/agent/log4j-cloud.xml*
>
> And then restart the cloudstack-agent service on the hosts. Post this, try
> deploying a VM in the shared network. This may give a better insight into
> where exactly the issue lies.
>
> Thanks,
> Pearl
>
>
> Software Engineer
> pearl.dsilva@shapeblue.com
> www.shapeblue.com<http://www.shapeblue.com>
>
>
>
>
>
>
> ------------------------------
> *From:* Ivan Kudryavtsev <iv...@bw-sw.com>
> *Sent:* Monday, September 28, 2020 2:17 PM
> *To:* users <us...@cloudstack.apache.org>
> *Subject:* Re: Cloudstack Advance with Security Group
>
> This is it.
>
pearl.dsilva@shapeblue.com
www.shapeblue.com
3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK
@shapeblue
> On Mon, Sep 28, 2020 at 3:46 PM Hean Seng <he...@gmail.com> wrote:
>
> > In the log, I cannot see anything much, except a few lines showing above
> > . Not sure if this is a bug on 4.14.
> >
> >
> >
> > 2020-09-28 03:53:36,585 ERROR [kvm.resource.LibvirtComputingResource]
> > (agentRequest-Handler-4:null) (logid:b6a4c077) Unable to apply default
> > network rule for nic cloudbr0 for VM i-2-81-VM
> >
> > 2020-09-28 03:53:36,858 ERROR [kvm.resource.LibvirtComputingResource]
> > (agentRequest-Handler-3:null) (logid:74058678) Unable to apply default
> > network rule for nic cloudbr0 for VM i-2-81-VM
> >
> > 2020-09-28 03:53:36,858 WARN
> > [resource.wrapper.LibvirtSecurityGroupRulesCommandWrapper]
> > (agentRequest-Handler-3:null) (logid:74058678) Failed to program default
> > network rules for vm i-2-81-VM
> >
> >
> >
> >
> > On Mon, Sep 28, 2020 at 4:34 PM Ivan Kudryavtsev <iv...@bw-sw.com> wrote:
> >
> > > Hi,
> > > no I'm on 4.11, so can not help with exact 4.14, and I'm on Ubuntu,
> > though,
> > > but for any KVM hypervisor Linux distribution, the logic is the same.
> > >
> > > On Mon, Sep 28, 2020 at 3:31 PM Hean Seng <he...@gmail.com> wrote:
> > >
> > > > Hi
> > > >
> > > > Are you running on CentOS7 ?
> > > >
> > > > I am running on CentOS7 , ACS 4.14 , and seem there is no log at
> of
> > > > security_group.log
> > > >
> > > > # ls /var/log/cloudstack/agent/
> > > >
> > > > agent.log resizevolume.log setup.log
> > > >
> > > >
> > > > I recheck back the Intall guide, seems no missing anything.
> > > >
> > > >
> > > > Older intallation guide, 4.11 mentioned need , allow
> > > > /usr/lib/sysctl.d/00-system.conf
> > > >
> > > > # Enable netfilter on bridges. net.bridge.bridge-nf-call-ip6tables =
> 1
> > > > net.bridge.bridge-nf-call-iptables = 1
> > > net.bridge.bridge-nf-call-arptables
> > > > = 1
> > > >
> > > > And it has been done too.
> > > >
> > > >
> > > >
> > > > On Mon, Sep 28, 2020 at 4:05 PM Ivan Kudryavtsev <iv...@bw-sw.com>
> > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > No, this is not the issue.
> > > > > It's a normal state of the system, as KVM hooks are a new and
> > optional
> > > > > feature of 4.14.
> > > > >
> > > > > You should find some sort of messages regarding security_groups at
> > > > > /var/log/cloudstack/agent/security_group.log
> > > > >
> > > > >
> > > > > On Mon, Sep 28, 2020 at 2:10 PM Hean Seng <he...@gmail.com>
> > wrote:
> > > > >
> > > > > > I not sure where goes wrong, are you running on CentOS 7 ? I
> have
> > > this
> > > > > > error too, do you think is this contribute to the error as well:
> > > > > >
> > > > > > 2020-09-28 03:04:52,762 WARN [kvm.resource.LibvirtKvmAgentHook]
> > > > > > (agentRequest-Handler-5:null) (logid:4f23845b) Groovy script
> > > > > > '/etc/cloudstack/agent/hooks/libvirt-vm-xml-transformer.groovy'
> is
> > > not
> > > > > > available. Transformations will not be applied.
> > > > > >
> > > > > > 2020-09-28 03:04:52,762 WARN [kvm.resource.LibvirtKvmAgentHook]
> > > > > > (agentRequest-Handler-5:null) (logid:4f23845b) Groovy scripting
> > > engine
> > > > is
> > > > > > not initialized. Data transformation skipped.
> > > > > >
> > > > > > 2020-09-28 03:04:53,083 WARN [kvm.resource.LibvirtKvmAgentHook]
> > > > > > (agentRequest-Handler-5:null) (logid:4f23845b) Groovy script
> > > > > > '/etc/cloudstack/agent/hooks/libvirt-vm-state-change.groovy' is
> not
> > > > > > available. Transformations will not be applied.
> > > > > >
> > > > > > On Mon, Sep 28, 2020 at 2:27 PM Ivan Kudryavtsev <ivan@bw-sw.com
> >
> > > > wrote:
> > > > > >
> > > > > > > This just means you installed it in the wrong way. Ebtables and
> > > > > Iptables
> > > > > > > must be filled with rules like
> > > > > > >
> > > > > > > -A i-6242-10304-def -m state --state RELATED,ESTABLISHED -j
> > ACCEPT
> > > > > > > -A i-6242-10304-def -p udp -m physdev --physdev-in vnet18
> > > > > > > --physdev-is-bridged -m udp --sport 68 --dport 67 -j ACCEPT
> > > > > > > -A i-6242-10304-def -p udp -m physdev --physdev-out vnet18
> > > > > > > --physdev-is-bridged -m udp --sport 67 --dport 68 -j ACCEPT
> > > > > > > -A i-6242-10304-def -m physdev --physdev-in vnet18
> > > > --physdev-is-bridged
> > > > > > -m
> > > > > > > set ! --match-set i-6242-10304-vm src -j DROP
> > > > > > > -A i-6242-10304-def -p udp -m physdev --physdev-in vnet18
> > > > > > > --physdev-is-bridged -m set --match-set i-6242-10304-vm src -m
> > udp
> > > > > > --dport
> > > > > > > 53 -j RETURN
> > > > > > > -A i-6242-10304-def -p tcp -m physdev --physdev-in vnet18
> > > > > > > --physdev-is-bridged -m set --match-set i-6242-10304-vm src -m
> > tcp
> > > > > > --dport
> > > > > > > 53 -j RETURN
> > > > > > > -A i-6242-10304-def -m physdev --physdev-in vnet18
> > > > --physdev-is-bridged
> > > > > > -m
> > > > > > > set --match-set i-6242-10304-vm src -j i-6242-10304-vm-eg
> > > > > > > -A i-6242-10304-def -m physdev --physdev-out vnet18
> > > > > --physdev-is-bridged
> > > > > > -j
> > > > > > > i-6242-10304-vm
> > > > > > > -A i-6242-10304-vm -p udp -m udp --dport 1:65535 -m state
> --state
> > > NEW
> > > > > -j
> > > > > > > ACCEPT
> > > > > > > -A i-6242-10304-vm -p tcp -m tcp --dport 1:65535 -m state
> --state
> > > NEW
> > > > > -j
> > > > > > > ACCEPT
> > > > > > > -A i-6242-10304-vm -p icmp -m icmp --icmp-type any -j ACCEPT
> > > > > > > -A i-6242-10304-vm -j DROP
> > > > > > >
> > > > > > >
> > > > > > > Bridge chain: i-4435-8929-vm-in, entries: 7, policy: ACCEPT
> > > > > > > -s ! 1e:0:32:0:2:2 -j DROP
> > > > > > > -p ARP -s ! 1e:0:32:0:2:2 -j DROP
> > > > > > > -p ARP --arp-mac-src ! 1e:0:32:0:2:2 -j DROP
> > > > > > > -p ARP -j i-4435-8929-vm-in-ips
> > > > > > > -p ARP --arp-op Request -j ACCEPT
> > > > > > > -p ARP --arp-op Reply -j ACCEPT
> > > > > > > -p ARP -j DROP
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Sep 28, 2020 at 1:10 PM Hean Seng <he...@gmail.com>
> > > > wrote:
> > > > > > >
> > > > > > > > I checked the hypervisor , it seems iptables is nothing
> inside
> > ,
> > > > > this
> > > > > > is
> > > > > > > > centos7 , initially i turnoff firewalld , but even i turn
> on
> > it
> > > > now
> > > > > > and
> > > > > > > > try to update the security group rules, it seems empty
> iptable
> > > > rules
> > > > > :
> > > > > > > >
> > > > > > > > [root@kvm03 ~]# iptables -L -v -n
> > > > > > > >
> > > > > > > > Chain INPUT (policy ACCEPT 82903 packets, 1170M bytes)
> > > > > > > >
> > > > > > > > pkts bytes target prot opt in out source
> > > > > > > > destination
> > > > > > > >
> > > > > > > >
> > > > > > > > Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
> > > > > > > >
> > > > > > > > pkts bytes target prot opt in out source
> > > > > > > > destination
> > > > > > > >
> > > > > > > >
> > > > > > > > Chain OUTPUT (policy ACCEPT 80505 packets, 25M bytes)
> > > > > > > >
> > > > > > > > pkts bytes target prot opt in out source
> > > > > > > > destination
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Mon, Sep 28, 2020 at 12:05 PM Pearl d'Silva <
> > > > > > > pearl.dsilva@shapeblue.com
> > > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Hean,
> > > > > > > > >
> > > > > > > > > In an Advanced Zone with Security Groups enabled, by
> default,
> > > > > egress
> > > > > > > > > traffic from the VM is allowed, while Ingress traffic is
> > > denied.
> > > > > > Hence,
> > > > > > > > as
> > > > > > > > > you rightly mentioned, security group rules are added
> > > > accordingly.
> > > > > > > These
> > > > > > > > > rules get added on the hypervisor host, and you can verify
> > > them,
> > > > by
> > > > > > > going
> > > > > > > > > into the host and searching for iptables rules
> corresponding
> > to
> > > > the
> > > > > > VM
> > > > > > > > > (internal name - i-x-y-VM).
> > > > > > > > > This blog maybe helpful in providing further details:
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://shankerbalan.net/blog/cloudstack-advanced-zone-with-security-groups/
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Pearl
> > > > > > > > > ________________________________
> > > > > > > > > From: Hean Seng <he...@gmail.com>
> > > > > > > > > Sent: Sunday, September 27, 2020 2:48 PM
> > > > > > > > > To: users@cloudstack.apache.org <
> users@cloudstack.apache.org
> > >
> > > > > > > > > Subject: Cloudstack Advance with Security Group
> > > > > > > > >
> > > > > > > > > Hi
> > > > > > > > >
> > > > > > > > > I created advance zone with security group, all working
> fine.
> > > > > > > > >
> > > > > > > > > But VMcreated ， seems the default security group that
> > assigned
> > > to
> > > > > the
> > > > > > > VM.
> > > > > > > > > all accept policy , i understand is Default Deny, and once
> > add
> > > > in
> > > > > > the
> > > > > > > > port
> > > > > > > > > in Security Group Ingress and Egress, only is allowed
> > > > > > > > >
> > > > > > > > > Also, is this rules created at VirtualRouter of the
> > > > SharedNetwork,
> > > > > or
> > > > > > > at
> > > > > > > > > the Hypervisor?
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Regards,
> > > > > > > > > Hean Seng
> > > > > > > > >
> > > > > > > > > pearl.dsilva@shapeblue.com
> > > > > > > > > www.shapeblue.com<http://www.shapeblue.com>
> > > > > > > > > 3 London Bridge Street, 3rd floor, News Building, London
> > SE1
> > > > > 9SGUK
> > > > > > > > > @shapeblue
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Regards,
> > > > > > > > Hean Seng
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Regards,
> > > > > > Hean Seng
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Regards,
> > > > Hean Seng
> > > >
> > >
> >
> >
> > --
> > Regards,
> > Hean Seng
> >
>
--
Regards,
Hean Seng
Re: Cloudstack Advance with Security Group
Posted by Hean Seng <he...@gmail.com>.
Hi Pearl,
I had change to debug, following is the error captured:
2020-09-29 01:13:30,090 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-2:null) (logid:388b92e0) Processing command:
com.cloud.agent.api.SecurityGroupRulesCmd
2020-09-29 01:13:30,090 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-2:null) (logid:388b92e0) Looking for libvirtd
connection at: qemu:///system
2020-09-29 01:13:30,094 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-2:null) (logid:388b92e0) Checking default network
rules for vm i-23-77-VM
2020-09-29 01:13:30,094 ERROR [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-2:null) (logid:388b92e0) Unable to apply default
network rule for nic cloudbr0 for VM i-23-77-VM
2020-09-29 01:13:30,094 WARN
[resource.wrapper.LibvirtSecurityGroupRulesCommandWrapper]
(agentRequest-Handler-2:null) (logid:388b92e0) Failed to program default
network rules for vm i-23-77-VM
2020-09-29 01:13:30,095 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-2:null) (logid:388b92e0) Seq 11-6057904448766738456: {
Ans: , MgmtId: 72752492851638, via: 11, Ver: v1, Flags: 110,
[{"com.cloud.agent.api.SecurityGroupRuleAnswer":{"logSequenceNumber":15,"vmId":77,"reason":"PROGRAMMING_FAILED","result":false,"details":"programming
default network rules failed","wait":0}}] }
On Tue, Sep 29, 2020 at 12:38 PM Pearl d'Silva <pe...@shapeblue.com>
wrote:
> Hi Hean,
>
>
> Could you try enabling debug logging on your hypervisor hosts, by running
> the following:
> *sed -i 's/INFO/DEBUG/g' /etc/cloudstack/agent/log4j-cloud.xml*
>
> And then restart the cloudstack-agent service on the hosts. Post this, try
> deploying a VM in the shared network. This may give a better insight into
> where exactly the issue lies.
>
> Thanks,
> Pearl
>
>
> Software Engineer
> pearl.dsilva@shapeblue.com
> www.shapeblue.com
>
>
>
>
>
>
> ------------------------------
> *From:* Ivan Kudryavtsev <iv...@bw-sw.com>
> *Sent:* Monday, September 28, 2020 2:17 PM
> *To:* users <us...@cloudstack.apache.org>
> *Subject:* Re: Cloudstack Advance with Security Group
>
> This is it.
>
> On Mon, Sep 28, 2020 at 3:46 PM Hean Seng <he...@gmail.com> wrote:
>
> > In the log, I cannot see anything much, except a few lines showing above
> > . Not sure if this is a bug on 4.14.
> >
> >
> >
> > 2020-09-28 03:53:36,585 ERROR [kvm.resource.LibvirtComputingResource]
> > (agentRequest-Handler-4:null) (logid:b6a4c077) Unable to apply default
> > network rule for nic cloudbr0 for VM i-2-81-VM
> >
> > 2020-09-28 03:53:36,858 ERROR [kvm.resource.LibvirtComputingResource]
> > (agentRequest-Handler-3:null) (logid:74058678) Unable to apply default
> > network rule for nic cloudbr0 for VM i-2-81-VM
> >
> > 2020-09-28 03:53:36,858 WARN
> > [resource.wrapper.LibvirtSecurityGroupRulesCommandWrapper]
> > (agentRequest-Handler-3:null) (logid:74058678) Failed to program default
> > network rules for vm i-2-81-VM
> >
> >
> >
> >
> > On Mon, Sep 28, 2020 at 4:34 PM Ivan Kudryavtsev <iv...@bw-sw.com> wrote:
> >
> > > Hi,
> > > no I'm on 4.11, so can not help with exact 4.14, and I'm on Ubuntu,
> > though,
> > > but for any KVM hypervisor Linux distribution, the logic is the same.
> > >
> > > On Mon, Sep 28, 2020 at 3:31 PM Hean Seng <he...@gmail.com> wrote:
> > >
> > > > Hi
> > > >
> > > > Are you running on CentOS7 ?
> > > >
> > > > I am running on CentOS7 , ACS 4.14 , and seem there is no log at
> of
> > > > security_group.log
> > > >
> > > > # ls /var/log/cloudstack/agent/
> > > >
> > > > agent.log resizevolume.log setup.log
> > > >
> > > >
> > > > I recheck back the Intall guide, seems no missing anything.
> > > >
> > > >
> > > > Older intallation guide, 4.11 mentioned need , allow
> > > > /usr/lib/sysctl.d/00-system.conf
> > > >
> > > > # Enable netfilter on bridges. net.bridge.bridge-nf-call-ip6tables =
> 1
> > > > net.bridge.bridge-nf-call-iptables = 1
> > > net.bridge.bridge-nf-call-arptables
> > > > = 1
> > > >
> > > > And it has been done too.
> > > >
> > > >
> > > >
> > > > On Mon, Sep 28, 2020 at 4:05 PM Ivan Kudryavtsev <iv...@bw-sw.com>
> > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > No, this is not the issue.
> > > > > It's a normal state of the system, as KVM hooks are a new and
> > optional
> > > > > feature of 4.14.
> > > > >
> > > > > You should find some sort of messages regarding security_groups at
> > > > > /var/log/cloudstack/agent/security_group.log
> > > > >
> > > > >
> > > > > On Mon, Sep 28, 2020 at 2:10 PM Hean Seng <he...@gmail.com>
> > wrote:
> > > > >
> > > > > > I not sure where goes wrong, are you running on CentOS 7 ? I
> have
> > > this
> > > > > > error too, do you think is this contribute to the error as well:
> > > > > >
> > > > > > 2020-09-28 03:04:52,762 WARN [kvm.resource.LibvirtKvmAgentHook]
> > > > > > (agentRequest-Handler-5:null) (logid:4f23845b) Groovy script
> > > > > > '/etc/cloudstack/agent/hooks/libvirt-vm-xml-transformer.groovy'
> is
> > > not
> > > > > > available. Transformations will not be applied.
> > > > > >
> > > > > > 2020-09-28 03:04:52,762 WARN [kvm.resource.LibvirtKvmAgentHook]
> > > > > > (agentRequest-Handler-5:null) (logid:4f23845b) Groovy scripting
> > > engine
> > > > is
> > > > > > not initialized. Data transformation skipped.
> > > > > >
> > > > > > 2020-09-28 03:04:53,083 WARN [kvm.resource.LibvirtKvmAgentHook]
> > > > > > (agentRequest-Handler-5:null) (logid:4f23845b) Groovy script
> > > > > > '/etc/cloudstack/agent/hooks/libvirt-vm-state-change.groovy' is
> not
> > > > > > available. Transformations will not be applied.
> > > > > >
> > > > > > On Mon, Sep 28, 2020 at 2:27 PM Ivan Kudryavtsev <ivan@bw-sw.com
> >
> > > > wrote:
> > > > > >
> > > > > > > This just means you installed it in the wrong way. Ebtables and
> > > > > Iptables
> > > > > > > must be filled with rules like
> > > > > > >
> > > > > > > -A i-6242-10304-def -m state --state RELATED,ESTABLISHED -j
> > ACCEPT
> > > > > > > -A i-6242-10304-def -p udp -m physdev --physdev-in vnet18
> > > > > > > --physdev-is-bridged -m udp --sport 68 --dport 67 -j ACCEPT
> > > > > > > -A i-6242-10304-def -p udp -m physdev --physdev-out vnet18
> > > > > > > --physdev-is-bridged -m udp --sport 67 --dport 68 -j ACCEPT
> > > > > > > -A i-6242-10304-def -m physdev --physdev-in vnet18
> > > > --physdev-is-bridged
> > > > > > -m
> > > > > > > set ! --match-set i-6242-10304-vm src -j DROP
> > > > > > > -A i-6242-10304-def -p udp -m physdev --physdev-in vnet18
> > > > > > > --physdev-is-bridged -m set --match-set i-6242-10304-vm src -m
> > udp
> > > > > > --dport
> > > > > > > 53 -j RETURN
> > > > > > > -A i-6242-10304-def -p tcp -m physdev --physdev-in vnet18
> > > > > > > --physdev-is-bridged -m set --match-set i-6242-10304-vm src -m
> > tcp
> > > > > > --dport
> > > > > > > 53 -j RETURN
> > > > > > > -A i-6242-10304-def -m physdev --physdev-in vnet18
> > > > --physdev-is-bridged
> > > > > > -m
> > > > > > > set --match-set i-6242-10304-vm src -j i-6242-10304-vm-eg
> > > > > > > -A i-6242-10304-def -m physdev --physdev-out vnet18
> > > > > --physdev-is-bridged
> > > > > > -j
> > > > > > > i-6242-10304-vm
> > > > > > > -A i-6242-10304-vm -p udp -m udp --dport 1:65535 -m state
> --state
> > > NEW
> > > > > -j
> > > > > > > ACCEPT
> > > > > > > -A i-6242-10304-vm -p tcp -m tcp --dport 1:65535 -m state
> --state
> > > NEW
> > > > > -j
> > > > > > > ACCEPT
> > > > > > > -A i-6242-10304-vm -p icmp -m icmp --icmp-type any -j ACCEPT
> > > > > > > -A i-6242-10304-vm -j DROP
> > > > > > >
> > > > > > >
> > > > > > > Bridge chain: i-4435-8929-vm-in, entries: 7, policy: ACCEPT
> > > > > > > -s ! 1e:0:32:0:2:2 -j DROP
> > > > > > > -p ARP -s ! 1e:0:32:0:2:2 -j DROP
> > > > > > > -p ARP --arp-mac-src ! 1e:0:32:0:2:2 -j DROP
> > > > > > > -p ARP -j i-4435-8929-vm-in-ips
> > > > > > > -p ARP --arp-op Request -j ACCEPT
> > > > > > > -p ARP --arp-op Reply -j ACCEPT
> > > > > > > -p ARP -j DROP
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Sep 28, 2020 at 1:10 PM Hean Seng <he...@gmail.com>
> > > > wrote:
> > > > > > >
> > > > > > > > I checked the hypervisor , it seems iptables is nothing
> inside
> > ,
> > > > > this
> > > > > > is
> > > > > > > > centos7 , initially i turnoff firewalld , but even i turn
> on
> > it
> > > > now
> > > > > > and
> > > > > > > > try to update the security group rules, it seems empty
> iptable
> > > > rules
> > > > > :
> > > > > > > >
> > > > > > > > [root@kvm03 ~]# iptables -L -v -n
> > > > > > > >
> > > > > > > > Chain INPUT (policy ACCEPT 82903 packets, 1170M bytes)
> > > > > > > >
> > > > > > > > pkts bytes target prot opt in out source
> > > > > > > > destination
> > > > > > > >
> > > > > > > >
> > > > > > > > Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
> > > > > > > >
> > > > > > > > pkts bytes target prot opt in out source
> > > > > > > > destination
> > > > > > > >
> > > > > > > >
> > > > > > > > Chain OUTPUT (policy ACCEPT 80505 packets, 25M bytes)
> > > > > > > >
> > > > > > > > pkts bytes target prot opt in out source
> > > > > > > > destination
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Mon, Sep 28, 2020 at 12:05 PM Pearl d'Silva <
> > > > > > > pearl.dsilva@shapeblue.com
> > > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Hean,
> > > > > > > > >
> > > > > > > > > In an Advanced Zone with Security Groups enabled, by
> default,
> > > > > egress
> > > > > > > > > traffic from the VM is allowed, while Ingress traffic is
> > > denied.
> > > > > > Hence,
> > > > > > > > as
> > > > > > > > > you rightly mentioned, security group rules are added
> > > > accordingly.
> > > > > > > These
> > > > > > > > > rules get added on the hypervisor host, and you can verify
> > > them,
> > > > by
> > > > > > > going
> > > > > > > > > into the host and searching for iptables rules
> corresponding
> > to
> > > > the
> > > > > > VM
> > > > > > > > > (internal name - i-x-y-VM).
> > > > > > > > > This blog maybe helpful in providing further details:
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://shankerbalan.net/blog/cloudstack-advanced-zone-with-security-groups/
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Pearl
> > > > > > > > > ________________________________
> > > > > > > > > From: Hean Seng <he...@gmail.com>
> > > > > > > > > Sent: Sunday, September 27, 2020 2:48 PM
> > > > > > > > > To: users@cloudstack.apache.org <
> users@cloudstack.apache.org
> > >
> > > > > > > > > Subject: Cloudstack Advance with Security Group
> > > > > > > > >
> > > > > > > > > Hi
> > > > > > > > >
> > > > > > > > > I created advance zone with security group, all working
> fine.
> > > > > > > > >
> > > > > > > > > But VMcreated ， seems the default security group that
> > assigned
> > > to
> > > > > the
> > > > > > > VM.
> > > > > > > > > all accept policy , i understand is Default Deny, and once
> > add
> > > > in
> > > > > > the
> > > > > > > > port
> > > > > > > > > in Security Group Ingress and Egress, only is allowed
> > > > > > > > >
> > > > > > > > > Also, is this rules created at VirtualRouter of the
> > > > SharedNetwork,
> > > > > or
> > > > > > > at
> > > > > > > > > the Hypervisor?
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Regards,
> > > > > > > > > Hean Seng
> > > > > > > > >
> > > > > > > > > pearl.dsilva@shapeblue.com
> > > > > > > > > www.shapeblue.com
> > > > > > > > > 3 London Bridge Street, 3rd floor, News Building, London
> > SE1
> > > > > 9SGUK
> > > > > > > > > @shapeblue
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Regards,
> > > > > > > > Hean Seng
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Regards,
> > > > > > Hean Seng
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Regards,
> > > > Hean Seng
> > > >
> > >
> >
> >
> > --
> > Regards,
> > Hean Seng
> >
>
--
Regards,
Hean Seng
Re: Cloudstack Advance with Security Group
Posted by Pearl d'Silva <pe...@shapeblue.com>.
Hi Hean,
Could you try enabling debug logging on your hypervisor hosts, by running the following:
sed -i 's/INFO/DEBUG/g' /etc/cloudstack/agent/log4j-cloud.xml
And then restart the cloudstack-agent service on the hosts. Post this, try deploying a VM in the shared network. This may give a better insight into where exactly the issue lies.
Thanks,
Pearl
________________________________
From: Ivan Kudryavtsev <iv...@bw-sw.com>
Sent: Monday, September 28, 2020 2:17 PM
To: users <us...@cloudstack.apache.org>
Subject: Re: Cloudstack Advance with Security Group
This is it.
Software Engineer
pearl.dsilva@shapeblue.com
www.shapeblue.com
On Mon, Sep 28, 2020 at 3:46 PM Hean Seng <he...@gmail.com> wrote:
> In the log, I cannot see anything much, except a few lines showing above
> . Not sure if this is a bug on 4.14.
>
>
>
> 2020-09-28 03:53:36,585 ERROR [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-4:null) (logid:b6a4c077) Unable to apply default
> network rule for nic cloudbr0 for VM i-2-81-VM
>
> 2020-09-28 03:53:36,858 ERROR [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-3:null) (logid:74058678) Unable to apply default
> network rule for nic cloudbr0 for VM i-2-81-VM
>
> 2020-09-28 03:53:36,858 WARN
> [resource.wrapper.LibvirtSecurityGroupRulesCommandWrapper]
> (agentRequest-Handler-3:null) (logid:74058678) Failed to program default
> network rules for vm i-2-81-VM
>
>
>
>
> On Mon, Sep 28, 2020 at 4:34 PM Ivan Kudryavtsev <iv...@bw-sw.com> wrote:
>
> > Hi,
> > no I'm on 4.11, so can not help with exact 4.14, and I'm on Ubuntu,
> though,
> > but for any KVM hypervisor Linux distribution, the logic is the same.
> >
> > On Mon, Sep 28, 2020 at 3:31 PM Hean Seng <he...@gmail.com> wrote:
> >
> > > Hi
> > >
> > > Are you running on CentOS7 ?
> > >
> > > I am running on CentOS7 , ACS 4.14 , and seem there is no log at of
> > > security_group.log
> > >
> > > # ls /var/log/cloudstack/agent/
> > >
> > > agent.log resizevolume.log setup.log
> > >
> > >
> > > I recheck back the Intall guide, seems no missing anything.
> > >
> > >
> > > Older intallation guide, 4.11 mentioned need , allow
> > > /usr/lib/sysctl.d/00-system.conf
> > >
> > > # Enable netfilter on bridges. net.bridge.bridge-nf-call-ip6tables = 1
> > > net.bridge.bridge-nf-call-iptables = 1
> > net.bridge.bridge-nf-call-arptables
> > > = 1
> > >
> > > And it has been done too.
> > >
> > >
> > >
> > > On Mon, Sep 28, 2020 at 4:05 PM Ivan Kudryavtsev <iv...@bw-sw.com>
> wrote:
> > >
> > > > Hi,
> > > >
> > > > No, this is not the issue.
> > > > It's a normal state of the system, as KVM hooks are a new and
> optional
> > > > feature of 4.14.
> > > >
> > > > You should find some sort of messages regarding security_groups at
> > > > /var/log/cloudstack/agent/security_group.log
> > > >
> > > >
> > > > On Mon, Sep 28, 2020 at 2:10 PM Hean Seng <he...@gmail.com>
> wrote:
> > > >
> > > > > I not sure where goes wrong, are you running on CentOS 7 ? I have
> > this
> > > > > error too, do you think is this contribute to the error as well:
> > > > >
> > > > > 2020-09-28 03:04:52,762 WARN [kvm.resource.LibvirtKvmAgentHook]
> > > > > (agentRequest-Handler-5:null) (logid:4f23845b) Groovy script
> > > > > '/etc/cloudstack/agent/hooks/libvirt-vm-xml-transformer.groovy' is
> > not
> > > > > available. Transformations will not be applied.
> > > > >
> > > > > 2020-09-28 03:04:52,762 WARN [kvm.resource.LibvirtKvmAgentHook]
> > > > > (agentRequest-Handler-5:null) (logid:4f23845b) Groovy scripting
> > engine
> > > is
> > > > > not initialized. Data transformation skipped.
> > > > >
> > > > > 2020-09-28 03:04:53,083 WARN [kvm.resource.LibvirtKvmAgentHook]
> > > > > (agentRequest-Handler-5:null) (logid:4f23845b) Groovy script
> > > > > '/etc/cloudstack/agent/hooks/libvirt-vm-state-change.groovy' is not
> > > > > available. Transformations will not be applied.
> > > > >
> > > > > On Mon, Sep 28, 2020 at 2:27 PM Ivan Kudryavtsev <iv...@bw-sw.com>
> > > wrote:
> > > > >
> > > > > > This just means you installed it in the wrong way. Ebtables and
> > > > Iptables
> > > > > > must be filled with rules like
> > > > > >
> > > > > > -A i-6242-10304-def -m state --state RELATED,ESTABLISHED -j
> ACCEPT
> > > > > > -A i-6242-10304-def -p udp -m physdev --physdev-in vnet18
> > > > > > --physdev-is-bridged -m udp --sport 68 --dport 67 -j ACCEPT
> > > > > > -A i-6242-10304-def -p udp -m physdev --physdev-out vnet18
> > > > > > --physdev-is-bridged -m udp --sport 67 --dport 68 -j ACCEPT
> > > > > > -A i-6242-10304-def -m physdev --physdev-in vnet18
> > > --physdev-is-bridged
> > > > > -m
> > > > > > set ! --match-set i-6242-10304-vm src -j DROP
> > > > > > -A i-6242-10304-def -p udp -m physdev --physdev-in vnet18
> > > > > > --physdev-is-bridged -m set --match-set i-6242-10304-vm src -m
> udp
> > > > > --dport
> > > > > > 53 -j RETURN
> > > > > > -A i-6242-10304-def -p tcp -m physdev --physdev-in vnet18
> > > > > > --physdev-is-bridged -m set --match-set i-6242-10304-vm src -m
> tcp
> > > > > --dport
> > > > > > 53 -j RETURN
> > > > > > -A i-6242-10304-def -m physdev --physdev-in vnet18
> > > --physdev-is-bridged
> > > > > -m
> > > > > > set --match-set i-6242-10304-vm src -j i-6242-10304-vm-eg
> > > > > > -A i-6242-10304-def -m physdev --physdev-out vnet18
> > > > --physdev-is-bridged
> > > > > -j
> > > > > > i-6242-10304-vm
> > > > > > -A i-6242-10304-vm -p udp -m udp --dport 1:65535 -m state --state
> > NEW
> > > > -j
> > > > > > ACCEPT
> > > > > > -A i-6242-10304-vm -p tcp -m tcp --dport 1:65535 -m state --state
> > NEW
> > > > -j
> > > > > > ACCEPT
> > > > > > -A i-6242-10304-vm -p icmp -m icmp --icmp-type any -j ACCEPT
> > > > > > -A i-6242-10304-vm -j DROP
> > > > > >
> > > > > >
> > > > > > Bridge chain: i-4435-8929-vm-in, entries: 7, policy: ACCEPT
> > > > > > -s ! 1e:0:32:0:2:2 -j DROP
> > > > > > -p ARP -s ! 1e:0:32:0:2:2 -j DROP
> > > > > > -p ARP --arp-mac-src ! 1e:0:32:0:2:2 -j DROP
> > > > > > -p ARP -j i-4435-8929-vm-in-ips
> > > > > > -p ARP --arp-op Request -j ACCEPT
> > > > > > -p ARP --arp-op Reply -j ACCEPT
> > > > > > -p ARP -j DROP
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Mon, Sep 28, 2020 at 1:10 PM Hean Seng <he...@gmail.com>
> > > wrote:
> > > > > >
> > > > > > > I checked the hypervisor , it seems iptables is nothing inside
> ,
> > > > this
> > > > > is
> > > > > > > centos7 , initially i turnoff firewalld , but even i turn on
> it
> > > now
> > > > > and
> > > > > > > try to update the security group rules, it seems empty iptable
> > > rules
> > > > :
> > > > > > >
> > > > > > > [root@kvm03 ~]# iptables -L -v -n
> > > > > > >
> > > > > > > Chain INPUT (policy ACCEPT 82903 packets, 1170M bytes)
> > > > > > >
> > > > > > > pkts bytes target prot opt in out source
> > > > > > > destination
> > > > > > >
> > > > > > >
> > > > > > > Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
> > > > > > >
> > > > > > > pkts bytes target prot opt in out source
> > > > > > > destination
> > > > > > >
> > > > > > >
> > > > > > > Chain OUTPUT (policy ACCEPT 80505 packets, 25M bytes)
> > > > > > >
> > > > > > > pkts bytes target prot opt in out source
> > > > > > > destination
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Sep 28, 2020 at 12:05 PM Pearl d'Silva <
> > > > > > pearl.dsilva@shapeblue.com
> > > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Hean,
> > > > > > > >
> > > > > > > > In an Advanced Zone with Security Groups enabled, by default,
> > > > egress
> > > > > > > > traffic from the VM is allowed, while Ingress traffic is
> > denied.
> > > > > Hence,
> > > > > > > as
> > > > > > > > you rightly mentioned, security group rules are added
> > > accordingly.
> > > > > > These
> > > > > > > > rules get added on the hypervisor host, and you can verify
> > them,
> > > by
> > > > > > going
> > > > > > > > into the host and searching for iptables rules corresponding
> to
> > > the
> > > > > VM
> > > > > > > > (internal name - i-x-y-VM).
> > > > > > > > This blog maybe helpful in providing further details:
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://shankerbalan.net/blog/cloudstack-advanced-zone-with-security-groups/
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Pearl
> > > > > > > > ________________________________
> > > > > > > > From: Hean Seng <he...@gmail.com>
> > > > > > > > Sent: Sunday, September 27, 2020 2:48 PM
> > > > > > > > To: users@cloudstack.apache.org <users@cloudstack.apache.org
> >
> > > > > > > > Subject: Cloudstack Advance with Security Group
> > > > > > > >
> > > > > > > > Hi
> > > > > > > >
> > > > > > > > I created advance zone with security group, all working fine.
> > > > > > > >
> > > > > > > > But VMcreated ， seems the default security group that
> assigned
> > to
> > > > the
> > > > > > VM.
> > > > > > > > all accept policy , i understand is Default Deny, and once
> add
> > > in
> > > > > the
> > > > > > > port
> > > > > > > > in Security Group Ingress and Egress, only is allowed
> > > > > > > >
> > > > > > > > Also, is this rules created at VirtualRouter of the
> > > SharedNetwork,
> > > > or
> > > > > > at
> > > > > > > > the Hypervisor?
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Regards,
> > > > > > > > Hean Seng
> > > > > > > >
> > > > > > > > pearl.dsilva@shapeblue.com
> > > > > > > > www.shapeblue.com<http://www.shapeblue.com>
> > > > > > > > 3 London Bridge Street, 3rd floor, News Building, London
> SE1
> > > > 9SGUK
> > > > > > > > @shapeblue
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Regards,
> > > > > > > Hean Seng
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Regards,
> > > > > Hean Seng
> > > > >
> > > >
> > >
> > >
> > > --
> > > Regards,
> > > Hean Seng
> > >
> >
>
>
> --
> Regards,
> Hean Seng
>
Re: Cloudstack Advance with Security Group
Posted by Ivan Kudryavtsev <iv...@bw-sw.com>.
This is it.
On Mon, Sep 28, 2020 at 3:46 PM Hean Seng <he...@gmail.com> wrote:
> In the log, I cannot see anything much, except a few lines showing above
> . Not sure if this is a bug on 4.14.
>
>
>
> 2020-09-28 03:53:36,585 ERROR [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-4:null) (logid:b6a4c077) Unable to apply default
> network rule for nic cloudbr0 for VM i-2-81-VM
>
> 2020-09-28 03:53:36,858 ERROR [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-3:null) (logid:74058678) Unable to apply default
> network rule for nic cloudbr0 for VM i-2-81-VM
>
> 2020-09-28 03:53:36,858 WARN
> [resource.wrapper.LibvirtSecurityGroupRulesCommandWrapper]
> (agentRequest-Handler-3:null) (logid:74058678) Failed to program default
> network rules for vm i-2-81-VM
>
>
>
>
> On Mon, Sep 28, 2020 at 4:34 PM Ivan Kudryavtsev <iv...@bw-sw.com> wrote:
>
> > Hi,
> > no I'm on 4.11, so can not help with exact 4.14, and I'm on Ubuntu,
> though,
> > but for any KVM hypervisor Linux distribution, the logic is the same.
> >
> > On Mon, Sep 28, 2020 at 3:31 PM Hean Seng <he...@gmail.com> wrote:
> >
> > > Hi
> > >
> > > Are you running on CentOS7 ?
> > >
> > > I am running on CentOS7 , ACS 4.14 , and seem there is no log at of
> > > security_group.log
> > >
> > > # ls /var/log/cloudstack/agent/
> > >
> > > agent.log resizevolume.log setup.log
> > >
> > >
> > > I recheck back the Intall guide, seems no missing anything.
> > >
> > >
> > > Older intallation guide, 4.11 mentioned need , allow
> > > /usr/lib/sysctl.d/00-system.conf
> > >
> > > # Enable netfilter on bridges. net.bridge.bridge-nf-call-ip6tables = 1
> > > net.bridge.bridge-nf-call-iptables = 1
> > net.bridge.bridge-nf-call-arptables
> > > = 1
> > >
> > > And it has been done too.
> > >
> > >
> > >
> > > On Mon, Sep 28, 2020 at 4:05 PM Ivan Kudryavtsev <iv...@bw-sw.com>
> wrote:
> > >
> > > > Hi,
> > > >
> > > > No, this is not the issue.
> > > > It's a normal state of the system, as KVM hooks are a new and
> optional
> > > > feature of 4.14.
> > > >
> > > > You should find some sort of messages regarding security_groups at
> > > > /var/log/cloudstack/agent/security_group.log
> > > >
> > > >
> > > > On Mon, Sep 28, 2020 at 2:10 PM Hean Seng <he...@gmail.com>
> wrote:
> > > >
> > > > > I not sure where goes wrong, are you running on CentOS 7 ? I have
> > this
> > > > > error too, do you think is this contribute to the error as well:
> > > > >
> > > > > 2020-09-28 03:04:52,762 WARN [kvm.resource.LibvirtKvmAgentHook]
> > > > > (agentRequest-Handler-5:null) (logid:4f23845b) Groovy script
> > > > > '/etc/cloudstack/agent/hooks/libvirt-vm-xml-transformer.groovy' is
> > not
> > > > > available. Transformations will not be applied.
> > > > >
> > > > > 2020-09-28 03:04:52,762 WARN [kvm.resource.LibvirtKvmAgentHook]
> > > > > (agentRequest-Handler-5:null) (logid:4f23845b) Groovy scripting
> > engine
> > > is
> > > > > not initialized. Data transformation skipped.
> > > > >
> > > > > 2020-09-28 03:04:53,083 WARN [kvm.resource.LibvirtKvmAgentHook]
> > > > > (agentRequest-Handler-5:null) (logid:4f23845b) Groovy script
> > > > > '/etc/cloudstack/agent/hooks/libvirt-vm-state-change.groovy' is not
> > > > > available. Transformations will not be applied.
> > > > >
> > > > > On Mon, Sep 28, 2020 at 2:27 PM Ivan Kudryavtsev <iv...@bw-sw.com>
> > > wrote:
> > > > >
> > > > > > This just means you installed it in the wrong way. Ebtables and
> > > > Iptables
> > > > > > must be filled with rules like
> > > > > >
> > > > > > -A i-6242-10304-def -m state --state RELATED,ESTABLISHED -j
> ACCEPT
> > > > > > -A i-6242-10304-def -p udp -m physdev --physdev-in vnet18
> > > > > > --physdev-is-bridged -m udp --sport 68 --dport 67 -j ACCEPT
> > > > > > -A i-6242-10304-def -p udp -m physdev --physdev-out vnet18
> > > > > > --physdev-is-bridged -m udp --sport 67 --dport 68 -j ACCEPT
> > > > > > -A i-6242-10304-def -m physdev --physdev-in vnet18
> > > --physdev-is-bridged
> > > > > -m
> > > > > > set ! --match-set i-6242-10304-vm src -j DROP
> > > > > > -A i-6242-10304-def -p udp -m physdev --physdev-in vnet18
> > > > > > --physdev-is-bridged -m set --match-set i-6242-10304-vm src -m
> udp
> > > > > --dport
> > > > > > 53 -j RETURN
> > > > > > -A i-6242-10304-def -p tcp -m physdev --physdev-in vnet18
> > > > > > --physdev-is-bridged -m set --match-set i-6242-10304-vm src -m
> tcp
> > > > > --dport
> > > > > > 53 -j RETURN
> > > > > > -A i-6242-10304-def -m physdev --physdev-in vnet18
> > > --physdev-is-bridged
> > > > > -m
> > > > > > set --match-set i-6242-10304-vm src -j i-6242-10304-vm-eg
> > > > > > -A i-6242-10304-def -m physdev --physdev-out vnet18
> > > > --physdev-is-bridged
> > > > > -j
> > > > > > i-6242-10304-vm
> > > > > > -A i-6242-10304-vm -p udp -m udp --dport 1:65535 -m state --state
> > NEW
> > > > -j
> > > > > > ACCEPT
> > > > > > -A i-6242-10304-vm -p tcp -m tcp --dport 1:65535 -m state --state
> > NEW
> > > > -j
> > > > > > ACCEPT
> > > > > > -A i-6242-10304-vm -p icmp -m icmp --icmp-type any -j ACCEPT
> > > > > > -A i-6242-10304-vm -j DROP
> > > > > >
> > > > > >
> > > > > > Bridge chain: i-4435-8929-vm-in, entries: 7, policy: ACCEPT
> > > > > > -s ! 1e:0:32:0:2:2 -j DROP
> > > > > > -p ARP -s ! 1e:0:32:0:2:2 -j DROP
> > > > > > -p ARP --arp-mac-src ! 1e:0:32:0:2:2 -j DROP
> > > > > > -p ARP -j i-4435-8929-vm-in-ips
> > > > > > -p ARP --arp-op Request -j ACCEPT
> > > > > > -p ARP --arp-op Reply -j ACCEPT
> > > > > > -p ARP -j DROP
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Mon, Sep 28, 2020 at 1:10 PM Hean Seng <he...@gmail.com>
> > > wrote:
> > > > > >
> > > > > > > I checked the hypervisor , it seems iptables is nothing inside
> ,
> > > > this
> > > > > is
> > > > > > > centos7 , initially i turnoff firewalld , but even i turn on
> it
> > > now
> > > > > and
> > > > > > > try to update the security group rules, it seems empty iptable
> > > rules
> > > > :
> > > > > > >
> > > > > > > [root@kvm03 ~]# iptables -L -v -n
> > > > > > >
> > > > > > > Chain INPUT (policy ACCEPT 82903 packets, 1170M bytes)
> > > > > > >
> > > > > > > pkts bytes target prot opt in out source
> > > > > > > destination
> > > > > > >
> > > > > > >
> > > > > > > Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
> > > > > > >
> > > > > > > pkts bytes target prot opt in out source
> > > > > > > destination
> > > > > > >
> > > > > > >
> > > > > > > Chain OUTPUT (policy ACCEPT 80505 packets, 25M bytes)
> > > > > > >
> > > > > > > pkts bytes target prot opt in out source
> > > > > > > destination
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Sep 28, 2020 at 12:05 PM Pearl d'Silva <
> > > > > > pearl.dsilva@shapeblue.com
> > > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Hean,
> > > > > > > >
> > > > > > > > In an Advanced Zone with Security Groups enabled, by default,
> > > > egress
> > > > > > > > traffic from the VM is allowed, while Ingress traffic is
> > denied.
> > > > > Hence,
> > > > > > > as
> > > > > > > > you rightly mentioned, security group rules are added
> > > accordingly.
> > > > > > These
> > > > > > > > rules get added on the hypervisor host, and you can verify
> > them,
> > > by
> > > > > > going
> > > > > > > > into the host and searching for iptables rules corresponding
> to
> > > the
> > > > > VM
> > > > > > > > (internal name - i-x-y-VM).
> > > > > > > > This blog maybe helpful in providing further details:
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://shankerbalan.net/blog/cloudstack-advanced-zone-with-security-groups/
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Pearl
> > > > > > > > ________________________________
> > > > > > > > From: Hean Seng <he...@gmail.com>
> > > > > > > > Sent: Sunday, September 27, 2020 2:48 PM
> > > > > > > > To: users@cloudstack.apache.org <users@cloudstack.apache.org
> >
> > > > > > > > Subject: Cloudstack Advance with Security Group
> > > > > > > >
> > > > > > > > Hi
> > > > > > > >
> > > > > > > > I created advance zone with security group, all working fine.
> > > > > > > >
> > > > > > > > But VMcreated ， seems the default security group that
> assigned
> > to
> > > > the
> > > > > > VM.
> > > > > > > > all accept policy , i understand is Default Deny, and once
> add
> > > in
> > > > > the
> > > > > > > port
> > > > > > > > in Security Group Ingress and Egress, only is allowed
> > > > > > > >
> > > > > > > > Also, is this rules created at VirtualRouter of the
> > > SharedNetwork,
> > > > or
> > > > > > at
> > > > > > > > the Hypervisor?
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Regards,
> > > > > > > > Hean Seng
> > > > > > > >
> > > > > > > > pearl.dsilva@shapeblue.com
> > > > > > > > www.shapeblue.com
> > > > > > > > 3 London Bridge Street, 3rd floor, News Building, London
> SE1
> > > > 9SGUK
> > > > > > > > @shapeblue
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Regards,
> > > > > > > Hean Seng
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Regards,
> > > > > Hean Seng
> > > > >
> > > >
> > >
> > >
> > > --
> > > Regards,
> > > Hean Seng
> > >
> >
>
>
> --
> Regards,
> Hean Seng
>
Re: Cloudstack Advance with Security Group
Posted by Hean Seng <he...@gmail.com>.
In the log, I cannot see anything much, except a few lines showing above
. Not sure if this is a bug on 4.14.
2020-09-28 03:53:36,585 ERROR [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-4:null) (logid:b6a4c077) Unable to apply default
network rule for nic cloudbr0 for VM i-2-81-VM
2020-09-28 03:53:36,858 ERROR [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-3:null) (logid:74058678) Unable to apply default
network rule for nic cloudbr0 for VM i-2-81-VM
2020-09-28 03:53:36,858 WARN
[resource.wrapper.LibvirtSecurityGroupRulesCommandWrapper]
(agentRequest-Handler-3:null) (logid:74058678) Failed to program default
network rules for vm i-2-81-VM
On Mon, Sep 28, 2020 at 4:34 PM Ivan Kudryavtsev <iv...@bw-sw.com> wrote:
> Hi,
> no I'm on 4.11, so can not help with exact 4.14, and I'm on Ubuntu, though,
> but for any KVM hypervisor Linux distribution, the logic is the same.
>
> On Mon, Sep 28, 2020 at 3:31 PM Hean Seng <he...@gmail.com> wrote:
>
> > Hi
> >
> > Are you running on CentOS7 ?
> >
> > I am running on CentOS7 , ACS 4.14 , and seem there is no log at of
> > security_group.log
> >
> > # ls /var/log/cloudstack/agent/
> >
> > agent.log resizevolume.log setup.log
> >
> >
> > I recheck back the Intall guide, seems no missing anything.
> >
> >
> > Older intallation guide, 4.11 mentioned need , allow
> > /usr/lib/sysctl.d/00-system.conf
> >
> > # Enable netfilter on bridges. net.bridge.bridge-nf-call-ip6tables = 1
> > net.bridge.bridge-nf-call-iptables = 1
> net.bridge.bridge-nf-call-arptables
> > = 1
> >
> > And it has been done too.
> >
> >
> >
> > On Mon, Sep 28, 2020 at 4:05 PM Ivan Kudryavtsev <iv...@bw-sw.com> wrote:
> >
> > > Hi,
> > >
> > > No, this is not the issue.
> > > It's a normal state of the system, as KVM hooks are a new and optional
> > > feature of 4.14.
> > >
> > > You should find some sort of messages regarding security_groups at
> > > /var/log/cloudstack/agent/security_group.log
> > >
> > >
> > > On Mon, Sep 28, 2020 at 2:10 PM Hean Seng <he...@gmail.com> wrote:
> > >
> > > > I not sure where goes wrong, are you running on CentOS 7 ? I have
> this
> > > > error too, do you think is this contribute to the error as well:
> > > >
> > > > 2020-09-28 03:04:52,762 WARN [kvm.resource.LibvirtKvmAgentHook]
> > > > (agentRequest-Handler-5:null) (logid:4f23845b) Groovy script
> > > > '/etc/cloudstack/agent/hooks/libvirt-vm-xml-transformer.groovy' is
> not
> > > > available. Transformations will not be applied.
> > > >
> > > > 2020-09-28 03:04:52,762 WARN [kvm.resource.LibvirtKvmAgentHook]
> > > > (agentRequest-Handler-5:null) (logid:4f23845b) Groovy scripting
> engine
> > is
> > > > not initialized. Data transformation skipped.
> > > >
> > > > 2020-09-28 03:04:53,083 WARN [kvm.resource.LibvirtKvmAgentHook]
> > > > (agentRequest-Handler-5:null) (logid:4f23845b) Groovy script
> > > > '/etc/cloudstack/agent/hooks/libvirt-vm-state-change.groovy' is not
> > > > available. Transformations will not be applied.
> > > >
> > > > On Mon, Sep 28, 2020 at 2:27 PM Ivan Kudryavtsev <iv...@bw-sw.com>
> > wrote:
> > > >
> > > > > This just means you installed it in the wrong way. Ebtables and
> > > Iptables
> > > > > must be filled with rules like
> > > > >
> > > > > -A i-6242-10304-def -m state --state RELATED,ESTABLISHED -j ACCEPT
> > > > > -A i-6242-10304-def -p udp -m physdev --physdev-in vnet18
> > > > > --physdev-is-bridged -m udp --sport 68 --dport 67 -j ACCEPT
> > > > > -A i-6242-10304-def -p udp -m physdev --physdev-out vnet18
> > > > > --physdev-is-bridged -m udp --sport 67 --dport 68 -j ACCEPT
> > > > > -A i-6242-10304-def -m physdev --physdev-in vnet18
> > --physdev-is-bridged
> > > > -m
> > > > > set ! --match-set i-6242-10304-vm src -j DROP
> > > > > -A i-6242-10304-def -p udp -m physdev --physdev-in vnet18
> > > > > --physdev-is-bridged -m set --match-set i-6242-10304-vm src -m udp
> > > > --dport
> > > > > 53 -j RETURN
> > > > > -A i-6242-10304-def -p tcp -m physdev --physdev-in vnet18
> > > > > --physdev-is-bridged -m set --match-set i-6242-10304-vm src -m tcp
> > > > --dport
> > > > > 53 -j RETURN
> > > > > -A i-6242-10304-def -m physdev --physdev-in vnet18
> > --physdev-is-bridged
> > > > -m
> > > > > set --match-set i-6242-10304-vm src -j i-6242-10304-vm-eg
> > > > > -A i-6242-10304-def -m physdev --physdev-out vnet18
> > > --physdev-is-bridged
> > > > -j
> > > > > i-6242-10304-vm
> > > > > -A i-6242-10304-vm -p udp -m udp --dport 1:65535 -m state --state
> NEW
> > > -j
> > > > > ACCEPT
> > > > > -A i-6242-10304-vm -p tcp -m tcp --dport 1:65535 -m state --state
> NEW
> > > -j
> > > > > ACCEPT
> > > > > -A i-6242-10304-vm -p icmp -m icmp --icmp-type any -j ACCEPT
> > > > > -A i-6242-10304-vm -j DROP
> > > > >
> > > > >
> > > > > Bridge chain: i-4435-8929-vm-in, entries: 7, policy: ACCEPT
> > > > > -s ! 1e:0:32:0:2:2 -j DROP
> > > > > -p ARP -s ! 1e:0:32:0:2:2 -j DROP
> > > > > -p ARP --arp-mac-src ! 1e:0:32:0:2:2 -j DROP
> > > > > -p ARP -j i-4435-8929-vm-in-ips
> > > > > -p ARP --arp-op Request -j ACCEPT
> > > > > -p ARP --arp-op Reply -j ACCEPT
> > > > > -p ARP -j DROP
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Sep 28, 2020 at 1:10 PM Hean Seng <he...@gmail.com>
> > wrote:
> > > > >
> > > > > > I checked the hypervisor , it seems iptables is nothing inside ,
> > > this
> > > > is
> > > > > > centos7 , initially i turnoff firewalld , but even i turn on it
> > now
> > > > and
> > > > > > try to update the security group rules, it seems empty iptable
> > rules
> > > :
> > > > > >
> > > > > > [root@kvm03 ~]# iptables -L -v -n
> > > > > >
> > > > > > Chain INPUT (policy ACCEPT 82903 packets, 1170M bytes)
> > > > > >
> > > > > > pkts bytes target prot opt in out source
> > > > > > destination
> > > > > >
> > > > > >
> > > > > > Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
> > > > > >
> > > > > > pkts bytes target prot opt in out source
> > > > > > destination
> > > > > >
> > > > > >
> > > > > > Chain OUTPUT (policy ACCEPT 80505 packets, 25M bytes)
> > > > > >
> > > > > > pkts bytes target prot opt in out source
> > > > > > destination
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Mon, Sep 28, 2020 at 12:05 PM Pearl d'Silva <
> > > > > pearl.dsilva@shapeblue.com
> > > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Hean,
> > > > > > >
> > > > > > > In an Advanced Zone with Security Groups enabled, by default,
> > > egress
> > > > > > > traffic from the VM is allowed, while Ingress traffic is
> denied.
> > > > Hence,
> > > > > > as
> > > > > > > you rightly mentioned, security group rules are added
> > accordingly.
> > > > > These
> > > > > > > rules get added on the hypervisor host, and you can verify
> them,
> > by
> > > > > going
> > > > > > > into the host and searching for iptables rules corresponding to
> > the
> > > > VM
> > > > > > > (internal name - i-x-y-VM).
> > > > > > > This blog maybe helpful in providing further details:
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://shankerbalan.net/blog/cloudstack-advanced-zone-with-security-groups/
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Pearl
> > > > > > > ________________________________
> > > > > > > From: Hean Seng <he...@gmail.com>
> > > > > > > Sent: Sunday, September 27, 2020 2:48 PM
> > > > > > > To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> > > > > > > Subject: Cloudstack Advance with Security Group
> > > > > > >
> > > > > > > Hi
> > > > > > >
> > > > > > > I created advance zone with security group, all working fine.
> > > > > > >
> > > > > > > But VMcreated ， seems the default security group that assigned
> to
> > > the
> > > > > VM.
> > > > > > > all accept policy , i understand is Default Deny, and once add
> > in
> > > > the
> > > > > > port
> > > > > > > in Security Group Ingress and Egress, only is allowed
> > > > > > >
> > > > > > > Also, is this rules created at VirtualRouter of the
> > SharedNetwork,
> > > or
> > > > > at
> > > > > > > the Hypervisor?
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Regards,
> > > > > > > Hean Seng
> > > > > > >
> > > > > > > pearl.dsilva@shapeblue.com
> > > > > > > www.shapeblue.com
> > > > > > > 3 London Bridge Street, 3rd floor, News Building, London SE1
> > > 9SGUK
> > > > > > > @shapeblue
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > > --
> > > > > > Regards,
> > > > > > Hean Seng
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Regards,
> > > > Hean Seng
> > > >
> > >
> >
> >
> > --
> > Regards,
> > Hean Seng
> >
>
--
Regards,
Hean Seng
Re: Cloudstack Advance with Security Group
Posted by Ivan Kudryavtsev <iv...@bw-sw.com>.
Hi,
no I'm on 4.11, so can not help with exact 4.14, and I'm on Ubuntu, though,
but for any KVM hypervisor Linux distribution, the logic is the same.
On Mon, Sep 28, 2020 at 3:31 PM Hean Seng <he...@gmail.com> wrote:
> Hi
>
> Are you running on CentOS7 ?
>
> I am running on CentOS7 , ACS 4.14 , and seem there is no log at of
> security_group.log
>
> # ls /var/log/cloudstack/agent/
>
> agent.log resizevolume.log setup.log
>
>
> I recheck back the Intall guide, seems no missing anything.
>
>
> Older intallation guide, 4.11 mentioned need , allow
> /usr/lib/sysctl.d/00-system.conf
>
> # Enable netfilter on bridges. net.bridge.bridge-nf-call-ip6tables = 1
> net.bridge.bridge-nf-call-iptables = 1 net.bridge.bridge-nf-call-arptables
> = 1
>
> And it has been done too.
>
>
>
> On Mon, Sep 28, 2020 at 4:05 PM Ivan Kudryavtsev <iv...@bw-sw.com> wrote:
>
> > Hi,
> >
> > No, this is not the issue.
> > It's a normal state of the system, as KVM hooks are a new and optional
> > feature of 4.14.
> >
> > You should find some sort of messages regarding security_groups at
> > /var/log/cloudstack/agent/security_group.log
> >
> >
> > On Mon, Sep 28, 2020 at 2:10 PM Hean Seng <he...@gmail.com> wrote:
> >
> > > I not sure where goes wrong, are you running on CentOS 7 ? I have this
> > > error too, do you think is this contribute to the error as well:
> > >
> > > 2020-09-28 03:04:52,762 WARN [kvm.resource.LibvirtKvmAgentHook]
> > > (agentRequest-Handler-5:null) (logid:4f23845b) Groovy script
> > > '/etc/cloudstack/agent/hooks/libvirt-vm-xml-transformer.groovy' is not
> > > available. Transformations will not be applied.
> > >
> > > 2020-09-28 03:04:52,762 WARN [kvm.resource.LibvirtKvmAgentHook]
> > > (agentRequest-Handler-5:null) (logid:4f23845b) Groovy scripting engine
> is
> > > not initialized. Data transformation skipped.
> > >
> > > 2020-09-28 03:04:53,083 WARN [kvm.resource.LibvirtKvmAgentHook]
> > > (agentRequest-Handler-5:null) (logid:4f23845b) Groovy script
> > > '/etc/cloudstack/agent/hooks/libvirt-vm-state-change.groovy' is not
> > > available. Transformations will not be applied.
> > >
> > > On Mon, Sep 28, 2020 at 2:27 PM Ivan Kudryavtsev <iv...@bw-sw.com>
> wrote:
> > >
> > > > This just means you installed it in the wrong way. Ebtables and
> > Iptables
> > > > must be filled with rules like
> > > >
> > > > -A i-6242-10304-def -m state --state RELATED,ESTABLISHED -j ACCEPT
> > > > -A i-6242-10304-def -p udp -m physdev --physdev-in vnet18
> > > > --physdev-is-bridged -m udp --sport 68 --dport 67 -j ACCEPT
> > > > -A i-6242-10304-def -p udp -m physdev --physdev-out vnet18
> > > > --physdev-is-bridged -m udp --sport 67 --dport 68 -j ACCEPT
> > > > -A i-6242-10304-def -m physdev --physdev-in vnet18
> --physdev-is-bridged
> > > -m
> > > > set ! --match-set i-6242-10304-vm src -j DROP
> > > > -A i-6242-10304-def -p udp -m physdev --physdev-in vnet18
> > > > --physdev-is-bridged -m set --match-set i-6242-10304-vm src -m udp
> > > --dport
> > > > 53 -j RETURN
> > > > -A i-6242-10304-def -p tcp -m physdev --physdev-in vnet18
> > > > --physdev-is-bridged -m set --match-set i-6242-10304-vm src -m tcp
> > > --dport
> > > > 53 -j RETURN
> > > > -A i-6242-10304-def -m physdev --physdev-in vnet18
> --physdev-is-bridged
> > > -m
> > > > set --match-set i-6242-10304-vm src -j i-6242-10304-vm-eg
> > > > -A i-6242-10304-def -m physdev --physdev-out vnet18
> > --physdev-is-bridged
> > > -j
> > > > i-6242-10304-vm
> > > > -A i-6242-10304-vm -p udp -m udp --dport 1:65535 -m state --state NEW
> > -j
> > > > ACCEPT
> > > > -A i-6242-10304-vm -p tcp -m tcp --dport 1:65535 -m state --state NEW
> > -j
> > > > ACCEPT
> > > > -A i-6242-10304-vm -p icmp -m icmp --icmp-type any -j ACCEPT
> > > > -A i-6242-10304-vm -j DROP
> > > >
> > > >
> > > > Bridge chain: i-4435-8929-vm-in, entries: 7, policy: ACCEPT
> > > > -s ! 1e:0:32:0:2:2 -j DROP
> > > > -p ARP -s ! 1e:0:32:0:2:2 -j DROP
> > > > -p ARP --arp-mac-src ! 1e:0:32:0:2:2 -j DROP
> > > > -p ARP -j i-4435-8929-vm-in-ips
> > > > -p ARP --arp-op Request -j ACCEPT
> > > > -p ARP --arp-op Reply -j ACCEPT
> > > > -p ARP -j DROP
> > > >
> > > >
> > > >
> > > > On Mon, Sep 28, 2020 at 1:10 PM Hean Seng <he...@gmail.com>
> wrote:
> > > >
> > > > > I checked the hypervisor , it seems iptables is nothing inside ,
> > this
> > > is
> > > > > centos7 , initially i turnoff firewalld , but even i turn on it
> now
> > > and
> > > > > try to update the security group rules, it seems empty iptable
> rules
> > :
> > > > >
> > > > > [root@kvm03 ~]# iptables -L -v -n
> > > > >
> > > > > Chain INPUT (policy ACCEPT 82903 packets, 1170M bytes)
> > > > >
> > > > > pkts bytes target prot opt in out source
> > > > > destination
> > > > >
> > > > >
> > > > > Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
> > > > >
> > > > > pkts bytes target prot opt in out source
> > > > > destination
> > > > >
> > > > >
> > > > > Chain OUTPUT (policy ACCEPT 80505 packets, 25M bytes)
> > > > >
> > > > > pkts bytes target prot opt in out source
> > > > > destination
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Sep 28, 2020 at 12:05 PM Pearl d'Silva <
> > > > pearl.dsilva@shapeblue.com
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > Hi Hean,
> > > > > >
> > > > > > In an Advanced Zone with Security Groups enabled, by default,
> > egress
> > > > > > traffic from the VM is allowed, while Ingress traffic is denied.
> > > Hence,
> > > > > as
> > > > > > you rightly mentioned, security group rules are added
> accordingly.
> > > > These
> > > > > > rules get added on the hypervisor host, and you can verify them,
> by
> > > > going
> > > > > > into the host and searching for iptables rules corresponding to
> the
> > > VM
> > > > > > (internal name - i-x-y-VM).
> > > > > > This blog maybe helpful in providing further details:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://shankerbalan.net/blog/cloudstack-advanced-zone-with-security-groups/
> > > > > >
> > > > > > Thanks,
> > > > > > Pearl
> > > > > > ________________________________
> > > > > > From: Hean Seng <he...@gmail.com>
> > > > > > Sent: Sunday, September 27, 2020 2:48 PM
> > > > > > To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> > > > > > Subject: Cloudstack Advance with Security Group
> > > > > >
> > > > > > Hi
> > > > > >
> > > > > > I created advance zone with security group, all working fine.
> > > > > >
> > > > > > But VMcreated ， seems the default security group that assigned to
> > the
> > > > VM.
> > > > > > all accept policy , i understand is Default Deny, and once add
> in
> > > the
> > > > > port
> > > > > > in Security Group Ingress and Egress, only is allowed
> > > > > >
> > > > > > Also, is this rules created at VirtualRouter of the
> SharedNetwork,
> > or
> > > > at
> > > > > > the Hypervisor?
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Regards,
> > > > > > Hean Seng
> > > > > >
> > > > > > pearl.dsilva@shapeblue.com
> > > > > > www.shapeblue.com
> > > > > > 3 London Bridge Street, 3rd floor, News Building, London SE1
> > 9SGUK
> > > > > > @shapeblue
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > > Regards,
> > > > > Hean Seng
> > > > >
> > > >
> > >
> > >
> > > --
> > > Regards,
> > > Hean Seng
> > >
> >
>
>
> --
> Regards,
> Hean Seng
>
Re: Cloudstack Advance with Security Group
Posted by Hean Seng <he...@gmail.com>.
Hi
Are you running on CentOS7 ?
I am running on CentOS7 , ACS 4.14 , and seem there is no log at of
security_group.log
# ls /var/log/cloudstack/agent/
agent.log resizevolume.log setup.log
I recheck back the Intall guide, seems no missing anything.
Older intallation guide, 4.11 mentioned need , allow
/usr/lib/sysctl.d/00-system.conf
# Enable netfilter on bridges. net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1 net.bridge.bridge-nf-call-arptables
= 1
And it has been done too.
On Mon, Sep 28, 2020 at 4:05 PM Ivan Kudryavtsev <iv...@bw-sw.com> wrote:
> Hi,
>
> No, this is not the issue.
> It's a normal state of the system, as KVM hooks are a new and optional
> feature of 4.14.
>
> You should find some sort of messages regarding security_groups at
> /var/log/cloudstack/agent/security_group.log
>
>
> On Mon, Sep 28, 2020 at 2:10 PM Hean Seng <he...@gmail.com> wrote:
>
> > I not sure where goes wrong, are you running on CentOS 7 ? I have this
> > error too, do you think is this contribute to the error as well:
> >
> > 2020-09-28 03:04:52,762 WARN [kvm.resource.LibvirtKvmAgentHook]
> > (agentRequest-Handler-5:null) (logid:4f23845b) Groovy script
> > '/etc/cloudstack/agent/hooks/libvirt-vm-xml-transformer.groovy' is not
> > available. Transformations will not be applied.
> >
> > 2020-09-28 03:04:52,762 WARN [kvm.resource.LibvirtKvmAgentHook]
> > (agentRequest-Handler-5:null) (logid:4f23845b) Groovy scripting engine is
> > not initialized. Data transformation skipped.
> >
> > 2020-09-28 03:04:53,083 WARN [kvm.resource.LibvirtKvmAgentHook]
> > (agentRequest-Handler-5:null) (logid:4f23845b) Groovy script
> > '/etc/cloudstack/agent/hooks/libvirt-vm-state-change.groovy' is not
> > available. Transformations will not be applied.
> >
> > On Mon, Sep 28, 2020 at 2:27 PM Ivan Kudryavtsev <iv...@bw-sw.com> wrote:
> >
> > > This just means you installed it in the wrong way. Ebtables and
> Iptables
> > > must be filled with rules like
> > >
> > > -A i-6242-10304-def -m state --state RELATED,ESTABLISHED -j ACCEPT
> > > -A i-6242-10304-def -p udp -m physdev --physdev-in vnet18
> > > --physdev-is-bridged -m udp --sport 68 --dport 67 -j ACCEPT
> > > -A i-6242-10304-def -p udp -m physdev --physdev-out vnet18
> > > --physdev-is-bridged -m udp --sport 67 --dport 68 -j ACCEPT
> > > -A i-6242-10304-def -m physdev --physdev-in vnet18 --physdev-is-bridged
> > -m
> > > set ! --match-set i-6242-10304-vm src -j DROP
> > > -A i-6242-10304-def -p udp -m physdev --physdev-in vnet18
> > > --physdev-is-bridged -m set --match-set i-6242-10304-vm src -m udp
> > --dport
> > > 53 -j RETURN
> > > -A i-6242-10304-def -p tcp -m physdev --physdev-in vnet18
> > > --physdev-is-bridged -m set --match-set i-6242-10304-vm src -m tcp
> > --dport
> > > 53 -j RETURN
> > > -A i-6242-10304-def -m physdev --physdev-in vnet18 --physdev-is-bridged
> > -m
> > > set --match-set i-6242-10304-vm src -j i-6242-10304-vm-eg
> > > -A i-6242-10304-def -m physdev --physdev-out vnet18
> --physdev-is-bridged
> > -j
> > > i-6242-10304-vm
> > > -A i-6242-10304-vm -p udp -m udp --dport 1:65535 -m state --state NEW
> -j
> > > ACCEPT
> > > -A i-6242-10304-vm -p tcp -m tcp --dport 1:65535 -m state --state NEW
> -j
> > > ACCEPT
> > > -A i-6242-10304-vm -p icmp -m icmp --icmp-type any -j ACCEPT
> > > -A i-6242-10304-vm -j DROP
> > >
> > >
> > > Bridge chain: i-4435-8929-vm-in, entries: 7, policy: ACCEPT
> > > -s ! 1e:0:32:0:2:2 -j DROP
> > > -p ARP -s ! 1e:0:32:0:2:2 -j DROP
> > > -p ARP --arp-mac-src ! 1e:0:32:0:2:2 -j DROP
> > > -p ARP -j i-4435-8929-vm-in-ips
> > > -p ARP --arp-op Request -j ACCEPT
> > > -p ARP --arp-op Reply -j ACCEPT
> > > -p ARP -j DROP
> > >
> > >
> > >
> > > On Mon, Sep 28, 2020 at 1:10 PM Hean Seng <he...@gmail.com> wrote:
> > >
> > > > I checked the hypervisor , it seems iptables is nothing inside ,
> this
> > is
> > > > centos7 , initially i turnoff firewalld , but even i turn on it now
> > and
> > > > try to update the security group rules, it seems empty iptable rules
> :
> > > >
> > > > [root@kvm03 ~]# iptables -L -v -n
> > > >
> > > > Chain INPUT (policy ACCEPT 82903 packets, 1170M bytes)
> > > >
> > > > pkts bytes target prot opt in out source
> > > > destination
> > > >
> > > >
> > > > Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
> > > >
> > > > pkts bytes target prot opt in out source
> > > > destination
> > > >
> > > >
> > > > Chain OUTPUT (policy ACCEPT 80505 packets, 25M bytes)
> > > >
> > > > pkts bytes target prot opt in out source
> > > > destination
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Mon, Sep 28, 2020 at 12:05 PM Pearl d'Silva <
> > > pearl.dsilva@shapeblue.com
> > > > >
> > > > wrote:
> > > >
> > > > > Hi Hean,
> > > > >
> > > > > In an Advanced Zone with Security Groups enabled, by default,
> egress
> > > > > traffic from the VM is allowed, while Ingress traffic is denied.
> > Hence,
> > > > as
> > > > > you rightly mentioned, security group rules are added accordingly.
> > > These
> > > > > rules get added on the hypervisor host, and you can verify them, by
> > > going
> > > > > into the host and searching for iptables rules corresponding to the
> > VM
> > > > > (internal name - i-x-y-VM).
> > > > > This blog maybe helpful in providing further details:
> > > > >
> > > > >
> > > >
> > >
> >
> https://shankerbalan.net/blog/cloudstack-advanced-zone-with-security-groups/
> > > > >
> > > > > Thanks,
> > > > > Pearl
> > > > > ________________________________
> > > > > From: Hean Seng <he...@gmail.com>
> > > > > Sent: Sunday, September 27, 2020 2:48 PM
> > > > > To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> > > > > Subject: Cloudstack Advance with Security Group
> > > > >
> > > > > Hi
> > > > >
> > > > > I created advance zone with security group, all working fine.
> > > > >
> > > > > But VMcreated ， seems the default security group that assigned to
> the
> > > VM.
> > > > > all accept policy , i understand is Default Deny, and once add in
> > the
> > > > port
> > > > > in Security Group Ingress and Egress, only is allowed
> > > > >
> > > > > Also, is this rules created at VirtualRouter of the SharedNetwork,
> or
> > > at
> > > > > the Hypervisor?
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Regards,
> > > > > Hean Seng
> > > > >
> > > > > pearl.dsilva@shapeblue.com
> > > > > www.shapeblue.com
> > > > > 3 London Bridge Street, 3rd floor, News Building, London SE1
> 9SGUK
> > > > > @shapeblue
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > > --
> > > > Regards,
> > > > Hean Seng
> > > >
> > >
> >
> >
> > --
> > Regards,
> > Hean Seng
> >
>
--
Regards,
Hean Seng
Re: Cloudstack Advance with Security Group
Posted by Ivan Kudryavtsev <iv...@bw-sw.com>.
Hi,
No, this is not the issue.
It's a normal state of the system, as KVM hooks are a new and optional
feature of 4.14.
You should find some sort of messages regarding security_groups at
/var/log/cloudstack/agent/security_group.log
On Mon, Sep 28, 2020 at 2:10 PM Hean Seng <he...@gmail.com> wrote:
> I not sure where goes wrong, are you running on CentOS 7 ? I have this
> error too, do you think is this contribute to the error as well:
>
> 2020-09-28 03:04:52,762 WARN [kvm.resource.LibvirtKvmAgentHook]
> (agentRequest-Handler-5:null) (logid:4f23845b) Groovy script
> '/etc/cloudstack/agent/hooks/libvirt-vm-xml-transformer.groovy' is not
> available. Transformations will not be applied.
>
> 2020-09-28 03:04:52,762 WARN [kvm.resource.LibvirtKvmAgentHook]
> (agentRequest-Handler-5:null) (logid:4f23845b) Groovy scripting engine is
> not initialized. Data transformation skipped.
>
> 2020-09-28 03:04:53,083 WARN [kvm.resource.LibvirtKvmAgentHook]
> (agentRequest-Handler-5:null) (logid:4f23845b) Groovy script
> '/etc/cloudstack/agent/hooks/libvirt-vm-state-change.groovy' is not
> available. Transformations will not be applied.
>
> On Mon, Sep 28, 2020 at 2:27 PM Ivan Kudryavtsev <iv...@bw-sw.com> wrote:
>
> > This just means you installed it in the wrong way. Ebtables and Iptables
> > must be filled with rules like
> >
> > -A i-6242-10304-def -m state --state RELATED,ESTABLISHED -j ACCEPT
> > -A i-6242-10304-def -p udp -m physdev --physdev-in vnet18
> > --physdev-is-bridged -m udp --sport 68 --dport 67 -j ACCEPT
> > -A i-6242-10304-def -p udp -m physdev --physdev-out vnet18
> > --physdev-is-bridged -m udp --sport 67 --dport 68 -j ACCEPT
> > -A i-6242-10304-def -m physdev --physdev-in vnet18 --physdev-is-bridged
> -m
> > set ! --match-set i-6242-10304-vm src -j DROP
> > -A i-6242-10304-def -p udp -m physdev --physdev-in vnet18
> > --physdev-is-bridged -m set --match-set i-6242-10304-vm src -m udp
> --dport
> > 53 -j RETURN
> > -A i-6242-10304-def -p tcp -m physdev --physdev-in vnet18
> > --physdev-is-bridged -m set --match-set i-6242-10304-vm src -m tcp
> --dport
> > 53 -j RETURN
> > -A i-6242-10304-def -m physdev --physdev-in vnet18 --physdev-is-bridged
> -m
> > set --match-set i-6242-10304-vm src -j i-6242-10304-vm-eg
> > -A i-6242-10304-def -m physdev --physdev-out vnet18 --physdev-is-bridged
> -j
> > i-6242-10304-vm
> > -A i-6242-10304-vm -p udp -m udp --dport 1:65535 -m state --state NEW -j
> > ACCEPT
> > -A i-6242-10304-vm -p tcp -m tcp --dport 1:65535 -m state --state NEW -j
> > ACCEPT
> > -A i-6242-10304-vm -p icmp -m icmp --icmp-type any -j ACCEPT
> > -A i-6242-10304-vm -j DROP
> >
> >
> > Bridge chain: i-4435-8929-vm-in, entries: 7, policy: ACCEPT
> > -s ! 1e:0:32:0:2:2 -j DROP
> > -p ARP -s ! 1e:0:32:0:2:2 -j DROP
> > -p ARP --arp-mac-src ! 1e:0:32:0:2:2 -j DROP
> > -p ARP -j i-4435-8929-vm-in-ips
> > -p ARP --arp-op Request -j ACCEPT
> > -p ARP --arp-op Reply -j ACCEPT
> > -p ARP -j DROP
> >
> >
> >
> > On Mon, Sep 28, 2020 at 1:10 PM Hean Seng <he...@gmail.com> wrote:
> >
> > > I checked the hypervisor , it seems iptables is nothing inside , this
> is
> > > centos7 , initially i turnoff firewalld , but even i turn on it now
> and
> > > try to update the security group rules, it seems empty iptable rules :
> > >
> > > [root@kvm03 ~]# iptables -L -v -n
> > >
> > > Chain INPUT (policy ACCEPT 82903 packets, 1170M bytes)
> > >
> > > pkts bytes target prot opt in out source
> > > destination
> > >
> > >
> > > Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
> > >
> > > pkts bytes target prot opt in out source
> > > destination
> > >
> > >
> > > Chain OUTPUT (policy ACCEPT 80505 packets, 25M bytes)
> > >
> > > pkts bytes target prot opt in out source
> > > destination
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Mon, Sep 28, 2020 at 12:05 PM Pearl d'Silva <
> > pearl.dsilva@shapeblue.com
> > > >
> > > wrote:
> > >
> > > > Hi Hean,
> > > >
> > > > In an Advanced Zone with Security Groups enabled, by default, egress
> > > > traffic from the VM is allowed, while Ingress traffic is denied.
> Hence,
> > > as
> > > > you rightly mentioned, security group rules are added accordingly.
> > These
> > > > rules get added on the hypervisor host, and you can verify them, by
> > going
> > > > into the host and searching for iptables rules corresponding to the
> VM
> > > > (internal name - i-x-y-VM).
> > > > This blog maybe helpful in providing further details:
> > > >
> > > >
> > >
> >
> https://shankerbalan.net/blog/cloudstack-advanced-zone-with-security-groups/
> > > >
> > > > Thanks,
> > > > Pearl
> > > > ________________________________
> > > > From: Hean Seng <he...@gmail.com>
> > > > Sent: Sunday, September 27, 2020 2:48 PM
> > > > To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> > > > Subject: Cloudstack Advance with Security Group
> > > >
> > > > Hi
> > > >
> > > > I created advance zone with security group, all working fine.
> > > >
> > > > But VMcreated ， seems the default security group that assigned to the
> > VM.
> > > > all accept policy , i understand is Default Deny, and once add in
> the
> > > port
> > > > in Security Group Ingress and Egress, only is allowed
> > > >
> > > > Also, is this rules created at VirtualRouter of the SharedNetwork, or
> > at
> > > > the Hypervisor?
> > > >
> > > >
> > > >
> > > > --
> > > > Regards,
> > > > Hean Seng
> > > >
> > > > pearl.dsilva@shapeblue.com
> > > > www.shapeblue.com
> > > > 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK
> > > > @shapeblue
> > > >
> > > >
> > > >
> > > >
> > >
> > > --
> > > Regards,
> > > Hean Seng
> > >
> >
>
>
> --
> Regards,
> Hean Seng
>
Re: Cloudstack Advance with Security Group
Posted by Hean Seng <he...@gmail.com>.
I not sure where goes wrong, are you running on CentOS 7 ? I have this
error too, do you think is this contribute to the error as well:
2020-09-28 03:04:52,762 WARN [kvm.resource.LibvirtKvmAgentHook]
(agentRequest-Handler-5:null) (logid:4f23845b) Groovy script
'/etc/cloudstack/agent/hooks/libvirt-vm-xml-transformer.groovy' is not
available. Transformations will not be applied.
2020-09-28 03:04:52,762 WARN [kvm.resource.LibvirtKvmAgentHook]
(agentRequest-Handler-5:null) (logid:4f23845b) Groovy scripting engine is
not initialized. Data transformation skipped.
2020-09-28 03:04:53,083 WARN [kvm.resource.LibvirtKvmAgentHook]
(agentRequest-Handler-5:null) (logid:4f23845b) Groovy script
'/etc/cloudstack/agent/hooks/libvirt-vm-state-change.groovy' is not
available. Transformations will not be applied.
On Mon, Sep 28, 2020 at 2:27 PM Ivan Kudryavtsev <iv...@bw-sw.com> wrote:
> This just means you installed it in the wrong way. Ebtables and Iptables
> must be filled with rules like
>
> -A i-6242-10304-def -m state --state RELATED,ESTABLISHED -j ACCEPT
> -A i-6242-10304-def -p udp -m physdev --physdev-in vnet18
> --physdev-is-bridged -m udp --sport 68 --dport 67 -j ACCEPT
> -A i-6242-10304-def -p udp -m physdev --physdev-out vnet18
> --physdev-is-bridged -m udp --sport 67 --dport 68 -j ACCEPT
> -A i-6242-10304-def -m physdev --physdev-in vnet18 --physdev-is-bridged -m
> set ! --match-set i-6242-10304-vm src -j DROP
> -A i-6242-10304-def -p udp -m physdev --physdev-in vnet18
> --physdev-is-bridged -m set --match-set i-6242-10304-vm src -m udp --dport
> 53 -j RETURN
> -A i-6242-10304-def -p tcp -m physdev --physdev-in vnet18
> --physdev-is-bridged -m set --match-set i-6242-10304-vm src -m tcp --dport
> 53 -j RETURN
> -A i-6242-10304-def -m physdev --physdev-in vnet18 --physdev-is-bridged -m
> set --match-set i-6242-10304-vm src -j i-6242-10304-vm-eg
> -A i-6242-10304-def -m physdev --physdev-out vnet18 --physdev-is-bridged -j
> i-6242-10304-vm
> -A i-6242-10304-vm -p udp -m udp --dport 1:65535 -m state --state NEW -j
> ACCEPT
> -A i-6242-10304-vm -p tcp -m tcp --dport 1:65535 -m state --state NEW -j
> ACCEPT
> -A i-6242-10304-vm -p icmp -m icmp --icmp-type any -j ACCEPT
> -A i-6242-10304-vm -j DROP
>
>
> Bridge chain: i-4435-8929-vm-in, entries: 7, policy: ACCEPT
> -s ! 1e:0:32:0:2:2 -j DROP
> -p ARP -s ! 1e:0:32:0:2:2 -j DROP
> -p ARP --arp-mac-src ! 1e:0:32:0:2:2 -j DROP
> -p ARP -j i-4435-8929-vm-in-ips
> -p ARP --arp-op Request -j ACCEPT
> -p ARP --arp-op Reply -j ACCEPT
> -p ARP -j DROP
>
>
>
> On Mon, Sep 28, 2020 at 1:10 PM Hean Seng <he...@gmail.com> wrote:
>
> > I checked the hypervisor , it seems iptables is nothing inside , this is
> > centos7 , initially i turnoff firewalld , but even i turn on it now and
> > try to update the security group rules, it seems empty iptable rules :
> >
> > [root@kvm03 ~]# iptables -L -v -n
> >
> > Chain INPUT (policy ACCEPT 82903 packets, 1170M bytes)
> >
> > pkts bytes target prot opt in out source
> > destination
> >
> >
> > Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
> >
> > pkts bytes target prot opt in out source
> > destination
> >
> >
> > Chain OUTPUT (policy ACCEPT 80505 packets, 25M bytes)
> >
> > pkts bytes target prot opt in out source
> > destination
> >
> >
> >
> >
> >
> >
> >
> > On Mon, Sep 28, 2020 at 12:05 PM Pearl d'Silva <
> pearl.dsilva@shapeblue.com
> > >
> > wrote:
> >
> > > Hi Hean,
> > >
> > > In an Advanced Zone with Security Groups enabled, by default, egress
> > > traffic from the VM is allowed, while Ingress traffic is denied. Hence,
> > as
> > > you rightly mentioned, security group rules are added accordingly.
> These
> > > rules get added on the hypervisor host, and you can verify them, by
> going
> > > into the host and searching for iptables rules corresponding to the VM
> > > (internal name - i-x-y-VM).
> > > This blog maybe helpful in providing further details:
> > >
> > >
> >
> https://shankerbalan.net/blog/cloudstack-advanced-zone-with-security-groups/
> > >
> > > Thanks,
> > > Pearl
> > > ________________________________
> > > From: Hean Seng <he...@gmail.com>
> > > Sent: Sunday, September 27, 2020 2:48 PM
> > > To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> > > Subject: Cloudstack Advance with Security Group
> > >
> > > Hi
> > >
> > > I created advance zone with security group, all working fine.
> > >
> > > But VMcreated ， seems the default security group that assigned to the
> VM.
> > > all accept policy , i understand is Default Deny, and once add in the
> > port
> > > in Security Group Ingress and Egress, only is allowed
> > >
> > > Also, is this rules created at VirtualRouter of the SharedNetwork, or
> at
> > > the Hypervisor?
> > >
> > >
> > >
> > > --
> > > Regards,
> > > Hean Seng
> > >
> > > pearl.dsilva@shapeblue.com
> > > www.shapeblue.com
> > > 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK
> > > @shapeblue
> > >
> > >
> > >
> > >
> >
> > --
> > Regards,
> > Hean Seng
> >
>
--
Regards,
Hean Seng
Re: Cloudstack Advance with Security Group
Posted by Ivan Kudryavtsev <iv...@bw-sw.com>.
This just means you installed it in the wrong way. Ebtables and Iptables
must be filled with rules like
-A i-6242-10304-def -m state --state RELATED,ESTABLISHED -j ACCEPT
-A i-6242-10304-def -p udp -m physdev --physdev-in vnet18
--physdev-is-bridged -m udp --sport 68 --dport 67 -j ACCEPT
-A i-6242-10304-def -p udp -m physdev --physdev-out vnet18
--physdev-is-bridged -m udp --sport 67 --dport 68 -j ACCEPT
-A i-6242-10304-def -m physdev --physdev-in vnet18 --physdev-is-bridged -m
set ! --match-set i-6242-10304-vm src -j DROP
-A i-6242-10304-def -p udp -m physdev --physdev-in vnet18
--physdev-is-bridged -m set --match-set i-6242-10304-vm src -m udp --dport
53 -j RETURN
-A i-6242-10304-def -p tcp -m physdev --physdev-in vnet18
--physdev-is-bridged -m set --match-set i-6242-10304-vm src -m tcp --dport
53 -j RETURN
-A i-6242-10304-def -m physdev --physdev-in vnet18 --physdev-is-bridged -m
set --match-set i-6242-10304-vm src -j i-6242-10304-vm-eg
-A i-6242-10304-def -m physdev --physdev-out vnet18 --physdev-is-bridged -j
i-6242-10304-vm
-A i-6242-10304-vm -p udp -m udp --dport 1:65535 -m state --state NEW -j
ACCEPT
-A i-6242-10304-vm -p tcp -m tcp --dport 1:65535 -m state --state NEW -j
ACCEPT
-A i-6242-10304-vm -p icmp -m icmp --icmp-type any -j ACCEPT
-A i-6242-10304-vm -j DROP
Bridge chain: i-4435-8929-vm-in, entries: 7, policy: ACCEPT
-s ! 1e:0:32:0:2:2 -j DROP
-p ARP -s ! 1e:0:32:0:2:2 -j DROP
-p ARP --arp-mac-src ! 1e:0:32:0:2:2 -j DROP
-p ARP -j i-4435-8929-vm-in-ips
-p ARP --arp-op Request -j ACCEPT
-p ARP --arp-op Reply -j ACCEPT
-p ARP -j DROP
On Mon, Sep 28, 2020 at 1:10 PM Hean Seng <he...@gmail.com> wrote:
> I checked the hypervisor , it seems iptables is nothing inside , this is
> centos7 , initially i turnoff firewalld , but even i turn on it now and
> try to update the security group rules, it seems empty iptable rules :
>
> [root@kvm03 ~]# iptables -L -v -n
>
> Chain INPUT (policy ACCEPT 82903 packets, 1170M bytes)
>
> pkts bytes target prot opt in out source
> destination
>
>
> Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
>
> pkts bytes target prot opt in out source
> destination
>
>
> Chain OUTPUT (policy ACCEPT 80505 packets, 25M bytes)
>
> pkts bytes target prot opt in out source
> destination
>
>
>
>
>
>
>
> On Mon, Sep 28, 2020 at 12:05 PM Pearl d'Silva <pearl.dsilva@shapeblue.com
> >
> wrote:
>
> > Hi Hean,
> >
> > In an Advanced Zone with Security Groups enabled, by default, egress
> > traffic from the VM is allowed, while Ingress traffic is denied. Hence,
> as
> > you rightly mentioned, security group rules are added accordingly. These
> > rules get added on the hypervisor host, and you can verify them, by going
> > into the host and searching for iptables rules corresponding to the VM
> > (internal name - i-x-y-VM).
> > This blog maybe helpful in providing further details:
> >
> >
> https://shankerbalan.net/blog/cloudstack-advanced-zone-with-security-groups/
> >
> > Thanks,
> > Pearl
> > ________________________________
> > From: Hean Seng <he...@gmail.com>
> > Sent: Sunday, September 27, 2020 2:48 PM
> > To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> > Subject: Cloudstack Advance with Security Group
> >
> > Hi
> >
> > I created advance zone with security group, all working fine.
> >
> > But VMcreated ， seems the default security group that assigned to the VM.
> > all accept policy , i understand is Default Deny, and once add in the
> port
> > in Security Group Ingress and Egress, only is allowed
> >
> > Also, is this rules created at VirtualRouter of the SharedNetwork, or at
> > the Hypervisor?
> >
> >
> >
> > --
> > Regards,
> > Hean Seng
> >
> > pearl.dsilva@shapeblue.com
> > www.shapeblue.com
> > 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK
> > @shapeblue
> >
> >
> >
> >
>
> --
> Regards,
> Hean Seng
>
Re: Cloudstack Advance with Security Group
Posted by Hean Seng <he...@gmail.com>.
I checked the hypervisor , it seems iptables is nothing inside , this is
centos7 , initially i turnoff firewalld , but even i turn on it now and
try to update the security group rules, it seems empty iptable rules :
[root@kvm03 ~]# iptables -L -v -n
Chain INPUT (policy ACCEPT 82903 packets, 1170M bytes)
pkts bytes target prot opt in out source
destination
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source
destination
Chain OUTPUT (policy ACCEPT 80505 packets, 25M bytes)
pkts bytes target prot opt in out source
destination
On Mon, Sep 28, 2020 at 12:05 PM Pearl d'Silva <pe...@shapeblue.com>
wrote:
> Hi Hean,
>
> In an Advanced Zone with Security Groups enabled, by default, egress
> traffic from the VM is allowed, while Ingress traffic is denied. Hence, as
> you rightly mentioned, security group rules are added accordingly. These
> rules get added on the hypervisor host, and you can verify them, by going
> into the host and searching for iptables rules corresponding to the VM
> (internal name - i-x-y-VM).
> This blog maybe helpful in providing further details:
>
> https://shankerbalan.net/blog/cloudstack-advanced-zone-with-security-groups/
>
> Thanks,
> Pearl
> ________________________________
> From: Hean Seng <he...@gmail.com>
> Sent: Sunday, September 27, 2020 2:48 PM
> To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> Subject: Cloudstack Advance with Security Group
>
> Hi
>
> I created advance zone with security group, all working fine.
>
> But VMcreated ， seems the default security group that assigned to the VM.
> all accept policy , i understand is Default Deny, and once add in the port
> in Security Group Ingress and Egress, only is allowed
>
> Also, is this rules created at VirtualRouter of the SharedNetwork, or at
> the Hypervisor?
>
>
>
> --
> Regards,
> Hean Seng
>
> pearl.dsilva@shapeblue.com
> www.shapeblue.com
> 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK
> @shapeblue
>
>
>
>
--
Regards,
Hean Seng
Re: Cloudstack Advance with Security Group
Posted by Pearl d'Silva <pe...@shapeblue.com>.
Hi Hean,
In an Advanced Zone with Security Groups enabled, by default, egress traffic from the VM is allowed, while Ingress traffic is denied. Hence, as you rightly mentioned, security group rules are added accordingly. These rules get added on the hypervisor host, and you can verify them, by going into the host and searching for iptables rules corresponding to the VM (internal name - i-x-y-VM).
This blog maybe helpful in providing further details:
https://shankerbalan.net/blog/cloudstack-advanced-zone-with-security-groups/
Thanks,
Pearl
________________________________
From: Hean Seng <he...@gmail.com>
Sent: Sunday, September 27, 2020 2:48 PM
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: Cloudstack Advance with Security Group
Hi
I created advance zone with security group, all working fine.
But VMcreated ， seems the default security group that assigned to the VM.
all accept policy , i understand is Default Deny, and once add in the port
in Security Group Ingress and Egress, only is allowed
Also, is this rules created at VirtualRouter of the SharedNetwork, or at
the Hypervisor?
--
Regards,
Hean Seng
pearl.dsilva@shapeblue.com
www.shapeblue.com
3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK
@shapeblue