You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by martin kolly <ma...@senselan.ch> on 2016/03/18 11:58:38 UTC
Issue: CLOUDSTACK-9255 Unable to start VM DomainRouter due to error
in finalizeStart
Hi All
We are facing the same issue as reported by Milamber (Ticket 9255)
https://issues.apache.org/jira/browse/CLOUDSTACK-9255. When deploying a
couple of VMs or Port Forwarding's the re-deployment of the router with
cleanup fails.
We found that iptables configuration takes a lot of time, this
eventually leads to a timeout on the management server "Unable to start
VM DomainRouter due to error in finalizeStart, not retrying"
Environment:
- Cloudstack 4.8
- KVM (local storage)
- hosts/mgr on Ubuntu 14.04
We tested with a simple set of four forwarding rules, here the setup:
root@r-96-VM:~# cat /etc/cloudstack/forwardingrules.json
{
"185.20.146.56": [
{
"internal_ip": "10.100.1.95",
"internal_ports": "22:22",
"protocol": "tcp",
"public_ip": "185.20.146.56",
"public_ports": "22:22",
"type": "forward"
}
],
"185.20.146.79": [
{
"internal_ip": "10.100.1.42",
"internal_ports": "22:22",
"protocol": "tcp",
"public_ip": "185.20.146.79",
"public_ports": "22:22",
"type": "forward"
},
{
"internal_ip": "10.100.1.42",
"internal_ports": "8443:8443",
"protocol": "tcp",
"public_ip": "185.20.146.79",
"public_ports": "8443:8443",
"type": "forward"
},
{
"internal_ip": "10.100.1.42",
"internal_ports": "53:53",
"protocol": "udp",
"public_ip": "185.20.146.79",
"public_ports": "53:53",
"type": "forward"
}
],
"id": "forwardingrules"
The definition for every port forwarding seems to take at ~1.5 seconds.
python /opt/cloud/bin/configure.py.timed
/etc/cloudstack/forwardingrules.json
-A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 22 -j
DNAT --to-destination 10.100.1.42:22
time : 0.000965118408203
-A PREROUTING -d 185.20.146.79/32 -i eth0 -p tcp -m tcp --dport 22 -j
DNAT --to-destination 10.100.1.42:22
time : 0.395485162735
-A OUTPUT -d 185.20.146.79/32 -p tcp -m tcp --dport 22 -j DNAT
--to-destination 10.100.1.42:22
time : 0.395533084869
-j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d
10.100.1.42/32 -o eth0 -p tcp -m tcp --dport 22
time : 1.16180706024
-A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 22 -j
MARK --set-xmark 0x2/0xffffffff
time : 1.16329216957
-A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 22 -m
state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask
0xffffffff
time : 1.16407108307
-A FORWARD -i eth2 -o eth0 -p tcp -m tcp --dport 22 -m state --state
NEW,ESTABLISHED -j ACCEPT
Total time for creating Policy : 1.53959512711
----------------------------------------------
-A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 8443 -j
DNAT --to-destination 10.100.1.42:8443
time : 0.000781059265137
-A PREROUTING -d 185.20.146.79/32 -i eth0 -p tcp -m tcp --dport 8443 -j
DNAT --to-destination 10.100.1.42:8443
time : 0.378201007843
-A OUTPUT -d 185.20.146.79/32 -p tcp -m tcp --dport 8443 -j DNAT
--to-destination 10.100.1.42:8443
time : 0.37822508812
-j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d
10.100.1.42/32 -o eth0 -p tcp -m tcp --dport 8443
time : 1.14627504349
-A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 8443 -j
MARK --set-xmark 0x2/0xffffffff
time : 1.1477329731
-A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 8443 -m
state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask
0xffffffff
time : 1.14850592613
-A FORWARD -i eth2 -o eth0 -p tcp -m tcp --dport 8443 -m state --state
NEW,ESTABLISHED -j ACCEPT
Total time for creating Policy : 1.52321791649
----------------------------------------------
-A PREROUTING -d 185.20.146.79/32 -i eth2 -p udp -m udp --dport 53 -j
DNAT --to-destination 10.100.1.42:53
time : 0.000754117965698
-A PREROUTING -d 185.20.146.79/32 -i eth0 -p udp -m udp --dport 53 -j
DNAT --to-destination 10.100.1.42:53
time : 0.383729934692
-A OUTPUT -d 185.20.146.79/32 -p udp -m udp --dport 53 -j DNAT
--to-destination 10.100.1.42:53
time : 0.383754968643
-j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d
10.100.1.42/32 -o eth0 -p udp -m udp --dport 53
time : 1.14376091957
-A PREROUTING -d 185.20.146.79/32 -i eth2 -p udp -m udp --dport 53 -j
MARK --set-xmark 0x2/0xffffffff
time : 1.14526605606
-A PREROUTING -d 185.20.146.79/32 -i eth2 -p udp -m udp --dport 53 -m
state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask
0xffffffff
time : 1.14599299431
-A FORWARD -i eth2 -o eth0 -p udp -m udp --dport 53 -m state --state
NEW,ESTABLISHED -j ACCEPT
Total time for creating Policy : 1.52742600441
----------------------------------------------
-A PREROUTING -d 185.20.146.56/32 -i eth2 -p tcp -m tcp --dport 22 -j
DNAT --to-destination 10.100.1.95:22
time : 0.000700950622559
-A PREROUTING -d 185.20.146.56/32 -i eth0 -p tcp -m tcp --dport 22 -j
DNAT --to-destination 10.100.1.95:22
time : 0.382349014282
-A OUTPUT -d 185.20.146.56/32 -p tcp -m tcp --dport 22 -j DNAT
--to-destination 10.100.1.95:22
time : 0.382384061813
-j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d
10.100.1.95/32 -o eth0 -p tcp -m tcp --dport 22
time : 1.1425909996
-A PREROUTING -d 185.20.146.56/32 -i eth2 -p tcp -m tcp --dport 22 -j
MARK --set-xmark 0x2/0xffffffff
time : 1.14400196075
-A PREROUTING -d 185.20.146.56/32 -i eth2 -p tcp -m tcp --dport 22 -m
state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask
0xffffffff
time : 1.14468812943
-A FORWARD -i eth2 -o eth0 -p tcp -m tcp --dport 22 -m state --state
NEW,ESTABLISHED -j ACCEPT
Total time for creating Policy : 1.52619600296
----------------------------------------------
Having a closer look at configure.py how the iptables rules are defined.
We think that it is not efficient to lookup these values for every policy:
def forward_vr(self, rule):
fw1 = "-A PREROUTING -d %s/32 -i %s -p %s -m %s --dport %s -j DNAT
--to-destination %s:%s" % \
(
rule['public_ip'],
*self.getDeviceByIp(rule['public_ip']),*
rule['protocol'],
rule['protocol'],
*self.portsToString(rule['public_ports'], ':'),*
rule['internal_ip'],
*self.portsToString(rule['internal_ports'], '-')*
)
fw2 = "-A PREROUTING -d %s/32 -i %s -p %s -m %s --dport %s -j DNAT
--to-destination %s:%s" % \
(
rule['public_ip'],
* self.getDeviceByIp(rule['internal_ip']),*
rule['protocol'],
rule['protocol'],
* self.portsToString(rule['public_ports'], ':'),*
rule['internal_ip'],
* self.portsToString(rule['internal_ports'], '-')
.....
*
Defining these values once at the beginning would be much more
efficient, no ?
def forward_vr(self, rule):
* pub_interface = self.getDeviceByIp(rule['public_ip'])**
** int_interface = self.getDeviceByIp(rule['internal_ip'])**
** pub_ports = self.portsToString(rule['public_ports'], ':')**
** int_ports = self.portsToString(rule['internal_ports'], '-')**
** int_network = self.getNetworkByIp(rule['internal_ip'])
* fw1 = "-A PREROUTING -d %s/32 -i %s -p %s -m %s --dport %s -j DNAT
--to-destination %s:%s" % \
(
rule['public_ip'],
pub_interface,
rule['protocol'],
rule['protocol'],
pub_ports,
rule['internal_ip'],
int_ports
)
fw2 = "-A PREROUTING -d %s/32 -i %s -p %s -m %s --dport %s -j DNAT
--to-destination %s:%s" % \
(
rule['public_ip'],
int_interface,
rule['protocol'],
rule['protocol'],
pub_ports,
rule['internal_ip'],
int_ports
)
.....
If we run the configure.py with these modifications we have the following:
root@r-96-VM:~# python /opt/cloud/bin/configure_modified.py
/etc/cloudstack/forwardingrules.json
-A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 22 -j
DNAT --to-destination 10.100.1.42:22
time : 0.000349044799805
-A PREROUTING -d 185.20.146.79/32 -i eth0 -p tcp -m tcp --dport 22 -j
DNAT --to-destination 10.100.1.42:22
time : 0.000686883926392
-A OUTPUT -d 185.20.146.79/32 -p tcp -m tcp --dport 22 -j DNAT
--to-destination 10.100.1.42:22
time : 0.000943899154663
-j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d
10.100.1.42/32 -o eth0 -p tcp -m tcp --dport 22
time : 0.00131487846375
-A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 22 -j
MARK --set-xmark 0x2/0xffffffff
time : 0.00161194801331
-A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 22 -m
state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask
0xffffffff
time : 0.00186896324158
-A FORWARD -i eth2 -o eth0 -p tcp -m tcp --dport 22 -m state --state
NEW,ESTABLISHED -j ACCEPT
Total time for creating Policy : 0.00216102600098
----------------------------------------------
-A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 8443 -j
DNAT --to-destination 10.100.1.42:8443
time : 0.000232934951782
-A PREROUTING -d 185.20.146.79/32 -i eth0 -p tcp -m tcp --dport 8443 -j
DNAT --to-destination 10.100.1.42:8443
time : 0.000478029251099
-A OUTPUT -d 185.20.146.79/32 -p tcp -m tcp --dport 8443 -j DNAT
--to-destination 10.100.1.42:8443
time : 0.00071907043457
-j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d
10.100.1.42/32 -o eth0 -p tcp -m tcp --dport 8443
time : 0.000991106033325
-A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 8443 -j
MARK --set-xmark 0x2/0xffffffff
time : 0.00136613845825
-A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 8443 -m
state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask
0xffffffff
time : 0.00174498558044
-A FORWARD -i eth2 -o eth0 -p tcp -m tcp --dport 8443 -m state --state
NEW,ESTABLISHED -j ACCEPT
Total time for creating Policy : 0.00219202041626
----------------------------------------------
-A PREROUTING -d 185.20.146.79/32 -i eth2 -p udp -m udp --dport 53 -j
DNAT --to-destination 10.100.1.42:53
time : 0.000226974487305
-A PREROUTING -d 185.20.146.79/32 -i eth0 -p udp -m udp --dport 53 -j
DNAT --to-destination 10.100.1.42:53
time : 0.000502824783325
-A OUTPUT -d 185.20.146.79/32 -p udp -m udp --dport 53 -j DNAT
--to-destination 10.100.1.42:53
time : 0.000762939453125
-j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d
10.100.1.42/32 -o eth0 -p udp -m udp --dport 53
time : 0.00103092193604
-A PREROUTING -d 185.20.146.79/32 -i eth2 -p udp -m udp --dport 53 -j
MARK --set-xmark 0x2/0xffffffff
time : 0.00134587287903
-A PREROUTING -d 185.20.146.79/32 -i eth2 -p udp -m udp --dport 53 -m
state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask
0xffffffff
time : 0.00158596038818
-A FORWARD -i eth2 -o eth0 -p udp -m udp --dport 53 -m state --state
NEW,ESTABLISHED -j ACCEPT
Total time for creating Policy : 0.00182485580444
----------------------------------------------
-A PREROUTING -d 185.20.146.56/32 -i eth2 -p tcp -m tcp --dport 22 -j
DNAT --to-destination 10.100.1.95:22
time : 0.000264167785645
-A PREROUTING -d 185.20.146.56/32 -i eth0 -p tcp -m tcp --dport 22 -j
DNAT --to-destination 10.100.1.95:22
time : 0.000508069992065
-A OUTPUT -d 185.20.146.56/32 -p tcp -m tcp --dport 22 -j DNAT
--to-destination 10.100.1.95:22
time : 0.000750064849854
-j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d
10.100.1.95/32 -o eth0 -p tcp -m tcp --dport 22
time : 0.00102114677429
-A PREROUTING -d 185.20.146.56/32 -i eth2 -p tcp -m tcp --dport 22 -j
MARK --set-xmark 0x2/0xffffffff
time : 0.00138115882874
-A PREROUTING -d 185.20.146.56/32 -i eth2 -p tcp -m tcp --dport 22 -m
state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask
0xffffffff
time : 0.00165915489197
-A FORWARD -i eth2 -o eth0 -p tcp -m tcp --dport 22 -m state --state
NEW,ESTABLISHED -j ACCEPT
Total time for creating Policy : 0.00196814537048
----------------------------------------------
Location of configure.py:
https://github.com/apache/cloudstack/blob/master/systemvm/patches/debian/config/opt/cloud/bin/configure.py
The modified scripts are attached. Thanks for your feedback.
regards
Martin
Re: Issue: CLOUDSTACK-9255 Unable to start VM DomainRouter due to
error in finalizeStart
Posted by Will Stevens <ws...@cloudops.com>.
To me, it looks like PR #1449 is still relevant since it is minimizing the
number of fetches required when the code is run and PR #1356 just made sure
code was run only when it needed to be.
*Will STEVENS*
Lead Developer
*CloudOps* *| *Cloud Solutions Experts
420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
w cloudops.com *|* tw @CloudOps_
On Sun, Mar 20, 2016 at 4:55 PM, Remi Bergsma <RB...@schubergphilis.com>
wrote:
> Hi Martin,
>
> Thanks, will have a look at it!
>
> These scripts are in the systemvm.iso in the CloudStack release, so not in
> the template. If you build a custom package or war to update your
> management servers you can use it already. Otherwise you need the next
> release.
>
> Regards, Remi
>
> Sent from my iPhone
>
> > On 20 Mar 2016, at 21:36, martin kolly <ma...@senselan.ch> wrote:
> >
> > Hi Remi
> >
> > PR #1449 created as requested. Tests in our environment showed that it
> speeds up the router configuration quite a bit.
> >
> > In the meantime https://github.com/apache/cloudstack/pull/1356 seems to
> be merged which resolves CLOUDSTACK-9255. So not sure if PR#1449 is still
> of interest..
> >
> > Do you know when the system vm with these fixes (PR1356) is available on
> http://cloudstack.apt-get.eu/systemvm/ ?
> >
> > regards
> > Martin
> >
> >
> >> On 03/18/2016 09:45 PM, Remi Bergsma wrote:
> >> Hi Martin,
> >>
> >> Thanks for the fix, didn’t catch you attachment first time.
> >>
> >> Would it be possible for you to send a Pull Request? Is this patch
> against master or a release branch? Generally speaking it’s best to make a
> PR against a release branch, 4.7 would be fine I guess in this case. Once
> it’s a PR we can test it.
> >>
> >> Regards,
> >> Remi
> >>
> >>
> >> From: martin kolly <martin.kolly@senselan.ch<mailto:
> martin.kolly@senselan.ch>>
> >> Reply-To: "dev@cloudstack.apache.org<ma...@cloudstack.apache.org>"
> <de...@cloudstack.apache.org>>
> >> Date: Friday 18 March 2016 at 11:58
> >> To: "dev@cloudstack.apache.org<ma...@cloudstack.apache.org>" <
> dev@cloudstack.apache.org<ma...@cloudstack.apache.org>>
> >> Subject: Issue: CLOUDSTACK-9255 Unable to start VM DomainRouter due to
> error in finalizeStart
> >>
> >> Hi All
> >>
> >> We are facing the same issue as reported by Milamber (Ticket 9255)
> https://issues.apache.org/jira/browse/CLOUDSTACK-9255. When deploying a
> couple of VMs or Port Forwarding's the re-deployment of the router with
> cleanup fails.
> >>
> >> We found that iptables configuration takes a lot of time, this
> eventually leads to a timeout on the management server "Unable to start VM
> DomainRouter due to error in finalizeStart, not retrying"
> >>
> >> Environment:
> >> - Cloudstack 4.8
> >> - KVM (local storage)
> >> - hosts/mgr on Ubuntu 14.04
> >>
> >> We tested with a simple set of four forwarding rules, here the setup:
> >>
> >> root@r-96-VM:~# cat /etc/cloudstack/forwardingrules.json
> >> {
> >> "185.20.146.56": [
> >> {
> >> "internal_ip": "10.100.1.95",
> >> "internal_ports": "22:22",
> >> "protocol": "tcp",
> >> "public_ip": "185.20.146.56",
> >> "public_ports": "22:22",
> >> "type": "forward"
> >> }
> >> ],
> >> "185.20.146.79": [
> >> {
> >> "internal_ip": "10.100.1.42",
> >> "internal_ports": "22:22",
> >> "protocol": "tcp",
> >> "public_ip": "185.20.146.79",
> >> "public_ports": "22:22",
> >> "type": "forward"
> >> },
> >> {
> >> "internal_ip": "10.100.1.42",
> >> "internal_ports": "8443:8443",
> >> "protocol": "tcp",
> >> "public_ip": "185.20.146.79",
> >> "public_ports": "8443:8443",
> >> "type": "forward"
> >> },
> >> {
> >> "internal_ip": "10.100.1.42",
> >> "internal_ports": "53:53",
> >> "protocol": "udp",
> >> "public_ip": "185.20.146.79",
> >> "public_ports": "53:53",
> >> "type": "forward"
> >> }
> >> ],
> >> "id": "forwardingrules"
> >>
> >> The definition for every port forwarding seems to take at ~1.5 seconds.
> >>
> >> python /opt/cloud/bin/configure.py.timed
> /etc/cloudstack/forwardingrules.json
> >>
> >> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 22 -j
> DNAT --to-destination 10.100.1.42:22
> >> time : 0.000965118408203
> >> -A PREROUTING -d 185.20.146.79/32 -i eth0 -p tcp -m tcp --dport 22 -j
> DNAT --to-destination 10.100.1.42:22
> >> time : 0.395485162735
> >> -A OUTPUT -d 185.20.146.79/32 -p tcp -m tcp --dport 22 -j DNAT
> --to-destination 10.100.1.42:22
> >> time : 0.395533084869
> >> -j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d
> 10.100.1.42/32 -o eth0 -p tcp -m tcp --dport 22
> >> time : 1.16180706024
> >> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 22 -j
> MARK --set-xmark 0x2/0xffffffff
> >> time : 1.16329216957
> >> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 22 -m
> state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask
> 0xffffffff
> >> time : 1.16407108307
> >> -A FORWARD -i eth2 -o eth0 -p tcp -m tcp --dport 22 -m state --state
> NEW,ESTABLISHED -j ACCEPT
> >> Total time for creating Policy : 1.53959512711
> >> ----------------------------------------------
> >> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 8443
> -j DNAT --to-destination 10.100.1.42:8443
> >> time : 0.000781059265137
> >> -A PREROUTING -d 185.20.146.79/32 -i eth0 -p tcp -m tcp --dport 8443
> -j DNAT --to-destination 10.100.1.42:8443
> >> time : 0.378201007843
> >> -A OUTPUT -d 185.20.146.79/32 -p tcp -m tcp --dport 8443 -j DNAT
> --to-destination 10.100.1.42:8443
> >> time : 0.37822508812
> >> -j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d
> 10.100.1.42/32 -o eth0 -p tcp -m tcp --dport 8443
> >> time : 1.14627504349
> >> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 8443
> -j MARK --set-xmark 0x2/0xffffffff
> >> time : 1.1477329731
> >> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 8443
> -m state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask
> 0xffffffff
> >> time : 1.14850592613
> >> -A FORWARD -i eth2 -o eth0 -p tcp -m tcp --dport 8443 -m state --state
> NEW,ESTABLISHED -j ACCEPT
> >> Total time for creating Policy : 1.52321791649
> >> ----------------------------------------------
> >> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p udp -m udp --dport 53 -j
> DNAT --to-destination 10.100.1.42:53
> >> time : 0.000754117965698
> >> -A PREROUTING -d 185.20.146.79/32 -i eth0 -p udp -m udp --dport 53 -j
> DNAT --to-destination 10.100.1.42:53
> >> time : 0.383729934692
> >> -A OUTPUT -d 185.20.146.79/32 -p udp -m udp --dport 53 -j DNAT
> --to-destination 10.100.1.42:53
> >> time : 0.383754968643
> >> -j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d
> 10.100.1.42/32 -o eth0 -p udp -m udp --dport 53
> >> time : 1.14376091957
> >> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p udp -m udp --dport 53 -j
> MARK --set-xmark 0x2/0xffffffff
> >> time : 1.14526605606
> >> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p udp -m udp --dport 53 -m
> state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask
> 0xffffffff
> >> time : 1.14599299431
> >> -A FORWARD -i eth2 -o eth0 -p udp -m udp --dport 53 -m state --state
> NEW,ESTABLISHED -j ACCEPT
> >> Total time for creating Policy : 1.52742600441
> >> ----------------------------------------------
> >> -A PREROUTING -d 185.20.146.56/32 -i eth2 -p tcp -m tcp --dport 22 -j
> DNAT --to-destination 10.100.1.95:22
> >> time : 0.000700950622559
> >> -A PREROUTING -d 185.20.146.56/32 -i eth0 -p tcp -m tcp --dport 22 -j
> DNAT --to-destination 10.100.1.95:22
> >> time : 0.382349014282
> >> -A OUTPUT -d 185.20.146.56/32 -p tcp -m tcp --dport 22 -j DNAT
> --to-destination 10.100.1.95:22
> >> time : 0.382384061813
> >> -j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d
> 10.100.1.95/32 -o eth0 -p tcp -m tcp --dport 22
> >> time : 1.1425909996
> >> -A PREROUTING -d 185.20.146.56/32 -i eth2 -p tcp -m tcp --dport 22 -j
> MARK --set-xmark 0x2/0xffffffff
> >> time : 1.14400196075
> >> -A PREROUTING -d 185.20.146.56/32 -i eth2 -p tcp -m tcp --dport 22 -m
> state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask
> 0xffffffff
> >> time : 1.14468812943
> >> -A FORWARD -i eth2 -o eth0 -p tcp -m tcp --dport 22 -m state --state
> NEW,ESTABLISHED -j ACCEPT
> >> Total time for creating Policy : 1.52619600296
> >> ----------------------------------------------
> >>
> >> Having a closer look at configure.py how the iptables rules are
> defined. We think that it is not efficient to lookup these values for every
> policy:
> >>
> >> def forward_vr(self, rule):
> >>
> >> fw1 = "-A PREROUTING -d %s/32 -i %s -p %s -m %s --dport %s -j DNAT
> --to-destination %s:%s" % \
> >> (
> >> rule['public_ip'],
> >> self.getDeviceByIp(rule['public_ip']),
> >> rule['protocol'],
> >> rule['protocol'],
> >> self.portsToString(rule['public_ports'], ':'),
> >> rule['internal_ip'],
> >> self.portsToString(rule['internal_ports'], '-')
> >> )
> >> fw2 = "-A PREROUTING -d %s/32 -i %s -p %s -m %s --dport %s -j DNAT
> --to-destination %s:%s" % \
> >> (
> >> rule['public_ip'],
> >> self.getDeviceByIp(rule['internal_ip']),
> >> rule['protocol'],
> >> rule['protocol'],
> >> self.portsToString(rule['public_ports'], ':'),
> >> rule['internal_ip'],
> >> self.portsToString(rule['internal_ports'], '-')
> >> .....
> >>
> >>
> >> Defining these values once at the beginning would be much more
> efficient, no ?
> >>
> >> def forward_vr(self, rule):
> >>
> >> pub_interface = self.getDeviceByIp(rule['public_ip'])
> >> int_interface = self.getDeviceByIp(rule['internal_ip'])
> >> pub_ports = self.portsToString(rule['public_ports'], ':')
> >> int_ports = self.portsToString(rule['internal_ports'], '-')
> >> int_network = self.getNetworkByIp(rule['internal_ip'])
> >>
> >> fw1 = "-A PREROUTING -d %s/32 -i %s -p %s -m %s --dport %s -j DNAT
> --to-destination %s:%s" % \
> >> (
> >> rule['public_ip'],
> >> pub_interface,
> >> rule['protocol'],
> >> rule['protocol'],
> >> pub_ports,
> >> rule['internal_ip'],
> >> int_ports
> >> )
> >>
> >> fw2 = "-A PREROUTING -d %s/32 -i %s -p %s -m %s --dport %s -j DNAT
> --to-destination %s:%s" % \
> >> (
> >> rule['public_ip'],
> >> int_interface,
> >> rule['protocol'],
> >> rule['protocol'],
> >> pub_ports,
> >> rule['internal_ip'],
> >> int_ports
> >> )
> >> .....
> >>
> >> If we run the configure.py with these modifications we have the
> following:
> >>
> >> root@r-96-VM:~# python /opt/cloud/bin/configure_modified.py
> /etc/cloudstack/forwardingrules.json
> >> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 22 -j
> DNAT --to-destination 10.100.1.42:22
> >> time : 0.000349044799805
> >> -A PREROUTING -d 185.20.146.79/32 -i eth0 -p tcp -m tcp --dport 22 -j
> DNAT --to-destination 10.100.1.42:22
> >> time : 0.000686883926392
> >> -A OUTPUT -d 185.20.146.79/32 -p tcp -m tcp --dport 22 -j DNAT
> --to-destination 10.100.1.42:22
> >> time : 0.000943899154663
> >> -j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d
> 10.100.1.42/32 -o eth0 -p tcp -m tcp --dport 22
> >> time : 0.00131487846375
> >> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 22 -j
> MARK --set-xmark 0x2/0xffffffff
> >> time : 0.00161194801331
> >> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 22 -m
> state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask
> 0xffffffff
> >> time : 0.00186896324158
> >> -A FORWARD -i eth2 -o eth0 -p tcp -m tcp --dport 22 -m state --state
> NEW,ESTABLISHED -j ACCEPT
> >> Total time for creating Policy : 0.00216102600098
> >> ----------------------------------------------
> >> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 8443
> -j DNAT --to-destination 10.100.1.42:8443
> >> time : 0.000232934951782
> >> -A PREROUTING -d 185.20.146.79/32 -i eth0 -p tcp -m tcp --dport 8443
> -j DNAT --to-destination 10.100.1.42:8443
> >> time : 0.000478029251099
> >> -A OUTPUT -d 185.20.146.79/32 -p tcp -m tcp --dport 8443 -j DNAT
> --to-destination 10.100.1.42:8443
> >> time : 0.00071907043457
> >> -j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d
> 10.100.1.42/32 -o eth0 -p tcp -m tcp --dport 8443
> >> time : 0.000991106033325
> >> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 8443
> -j MARK --set-xmark 0x2/0xffffffff
> >> time : 0.00136613845825
> >> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 8443
> -m state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask
> 0xffffffff
> >> time : 0.00174498558044
> >> -A FORWARD -i eth2 -o eth0 -p tcp -m tcp --dport 8443 -m state --state
> NEW,ESTABLISHED -j ACCEPT
> >> Total time for creating Policy : 0.00219202041626
> >> ----------------------------------------------
> >> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p udp -m udp --dport 53 -j
> DNAT --to-destination 10.100.1.42:53
> >> time : 0.000226974487305
> >> -A PREROUTING -d 185.20.146.79/32 -i eth0 -p udp -m udp --dport 53 -j
> DNAT --to-destination 10.100.1.42:53
> >> time : 0.000502824783325
> >> -A OUTPUT -d 185.20.146.79/32 -p udp -m udp --dport 53 -j DNAT
> --to-destination 10.100.1.42:53
> >> time : 0.000762939453125
> >> -j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d
> 10.100.1.42/32 -o eth0 -p udp -m udp --dport 53
> >> time : 0.00103092193604
> >> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p udp -m udp --dport 53 -j
> MARK --set-xmark 0x2/0xffffffff
> >> time : 0.00134587287903
> >> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p udp -m udp --dport 53 -m
> state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask
> 0xffffffff
> >> time : 0.00158596038818
> >> -A FORWARD -i eth2 -o eth0 -p udp -m udp --dport 53 -m state --state
> NEW,ESTABLISHED -j ACCEPT
> >> Total time for creating Policy : 0.00182485580444
> >> ----------------------------------------------
> >> -A PREROUTING -d 185.20.146.56/32 -i eth2 -p tcp -m tcp --dport 22 -j
> DNAT --to-destination 10.100.1.95:22
> >> time : 0.000264167785645
> >> -A PREROUTING -d 185.20.146.56/32 -i eth0 -p tcp -m tcp --dport 22 -j
> DNAT --to-destination 10.100.1.95:22
> >> time : 0.000508069992065
> >> -A OUTPUT -d 185.20.146.56/32 -p tcp -m tcp --dport 22 -j DNAT
> --to-destination 10.100.1.95:22
> >> time : 0.000750064849854
> >> -j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d
> 10.100.1.95/32 -o eth0 -p tcp -m tcp --dport 22
> >> time : 0.00102114677429
> >> -A PREROUTING -d 185.20.146.56/32 -i eth2 -p tcp -m tcp --dport 22 -j
> MARK --set-xmark 0x2/0xffffffff
> >> time : 0.00138115882874
> >> -A PREROUTING -d 185.20.146.56/32 -i eth2 -p tcp -m tcp --dport 22 -m
> state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask
> 0xffffffff
> >> time : 0.00165915489197
> >> -A FORWARD -i eth2 -o eth0 -p tcp -m tcp --dport 22 -m state --state
> NEW,ESTABLISHED -j ACCEPT
> >> Total time for creating Policy : 0.00196814537048
> >> ----------------------------------------------
> >>
> >> Location of configure.py:
> >>
> https://github.com/apache/cloudstack/blob/master/systemvm/patches/debian/config/opt/cloud/bin/configure.py
> >>
> >> The modified scripts are attached. Thanks for your feedback.
> >>
> >> regards
> >> Martin
> >
>
Re: Issue: CLOUDSTACK-9255 Unable to start VM DomainRouter due to
error in finalizeStart
Posted by Remi Bergsma <RB...@schubergphilis.com>.
Hi Martin,
Thanks, will have a look at it!
These scripts are in the systemvm.iso in the CloudStack release, so not in the template. If you build a custom package or war to update your management servers you can use it already. Otherwise you need the next release.
Regards, Remi
Sent from my iPhone
> On 20 Mar 2016, at 21:36, martin kolly <ma...@senselan.ch> wrote:
>
> Hi Remi
>
> PR #1449 created as requested. Tests in our environment showed that it speeds up the router configuration quite a bit.
>
> In the meantime https://github.com/apache/cloudstack/pull/1356 seems to be merged which resolves CLOUDSTACK-9255. So not sure if PR#1449 is still of interest..
>
> Do you know when the system vm with these fixes (PR1356) is available on http://cloudstack.apt-get.eu/systemvm/ ?
>
> regards
> Martin
>
>
>> On 03/18/2016 09:45 PM, Remi Bergsma wrote:
>> Hi Martin,
>>
>> Thanks for the fix, didn’t catch you attachment first time.
>>
>> Would it be possible for you to send a Pull Request? Is this patch against master or a release branch? Generally speaking it’s best to make a PR against a release branch, 4.7 would be fine I guess in this case. Once it’s a PR we can test it.
>>
>> Regards,
>> Remi
>>
>>
>> From: martin kolly <ma...@senselan.ch>>
>> Reply-To: "dev@cloudstack.apache.org<ma...@cloudstack.apache.org>" <de...@cloudstack.apache.org>>
>> Date: Friday 18 March 2016 at 11:58
>> To: "dev@cloudstack.apache.org<ma...@cloudstack.apache.org>" <de...@cloudstack.apache.org>>
>> Subject: Issue: CLOUDSTACK-9255 Unable to start VM DomainRouter due to error in finalizeStart
>>
>> Hi All
>>
>> We are facing the same issue as reported by Milamber (Ticket 9255) https://issues.apache.org/jira/browse/CLOUDSTACK-9255. When deploying a couple of VMs or Port Forwarding's the re-deployment of the router with cleanup fails.
>>
>> We found that iptables configuration takes a lot of time, this eventually leads to a timeout on the management server "Unable to start VM DomainRouter due to error in finalizeStart, not retrying"
>>
>> Environment:
>> - Cloudstack 4.8
>> - KVM (local storage)
>> - hosts/mgr on Ubuntu 14.04
>>
>> We tested with a simple set of four forwarding rules, here the setup:
>>
>> root@r-96-VM:~# cat /etc/cloudstack/forwardingrules.json
>> {
>> "185.20.146.56": [
>> {
>> "internal_ip": "10.100.1.95",
>> "internal_ports": "22:22",
>> "protocol": "tcp",
>> "public_ip": "185.20.146.56",
>> "public_ports": "22:22",
>> "type": "forward"
>> }
>> ],
>> "185.20.146.79": [
>> {
>> "internal_ip": "10.100.1.42",
>> "internal_ports": "22:22",
>> "protocol": "tcp",
>> "public_ip": "185.20.146.79",
>> "public_ports": "22:22",
>> "type": "forward"
>> },
>> {
>> "internal_ip": "10.100.1.42",
>> "internal_ports": "8443:8443",
>> "protocol": "tcp",
>> "public_ip": "185.20.146.79",
>> "public_ports": "8443:8443",
>> "type": "forward"
>> },
>> {
>> "internal_ip": "10.100.1.42",
>> "internal_ports": "53:53",
>> "protocol": "udp",
>> "public_ip": "185.20.146.79",
>> "public_ports": "53:53",
>> "type": "forward"
>> }
>> ],
>> "id": "forwardingrules"
>>
>> The definition for every port forwarding seems to take at ~1.5 seconds.
>>
>> python /opt/cloud/bin/configure.py.timed /etc/cloudstack/forwardingrules.json
>>
>> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 22 -j DNAT --to-destination 10.100.1.42:22
>> time : 0.000965118408203
>> -A PREROUTING -d 185.20.146.79/32 -i eth0 -p tcp -m tcp --dport 22 -j DNAT --to-destination 10.100.1.42:22
>> time : 0.395485162735
>> -A OUTPUT -d 185.20.146.79/32 -p tcp -m tcp --dport 22 -j DNAT --to-destination 10.100.1.42:22
>> time : 0.395533084869
>> -j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d 10.100.1.42/32 -o eth0 -p tcp -m tcp --dport 22
>> time : 1.16180706024
>> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 22 -j MARK --set-xmark 0x2/0xffffffff
>> time : 1.16329216957
>> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 22 -m state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask 0xffffffff
>> time : 1.16407108307
>> -A FORWARD -i eth2 -o eth0 -p tcp -m tcp --dport 22 -m state --state NEW,ESTABLISHED -j ACCEPT
>> Total time for creating Policy : 1.53959512711
>> ----------------------------------------------
>> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 8443 -j DNAT --to-destination 10.100.1.42:8443
>> time : 0.000781059265137
>> -A PREROUTING -d 185.20.146.79/32 -i eth0 -p tcp -m tcp --dport 8443 -j DNAT --to-destination 10.100.1.42:8443
>> time : 0.378201007843
>> -A OUTPUT -d 185.20.146.79/32 -p tcp -m tcp --dport 8443 -j DNAT --to-destination 10.100.1.42:8443
>> time : 0.37822508812
>> -j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d 10.100.1.42/32 -o eth0 -p tcp -m tcp --dport 8443
>> time : 1.14627504349
>> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 8443 -j MARK --set-xmark 0x2/0xffffffff
>> time : 1.1477329731
>> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 8443 -m state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask 0xffffffff
>> time : 1.14850592613
>> -A FORWARD -i eth2 -o eth0 -p tcp -m tcp --dport 8443 -m state --state NEW,ESTABLISHED -j ACCEPT
>> Total time for creating Policy : 1.52321791649
>> ----------------------------------------------
>> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p udp -m udp --dport 53 -j DNAT --to-destination 10.100.1.42:53
>> time : 0.000754117965698
>> -A PREROUTING -d 185.20.146.79/32 -i eth0 -p udp -m udp --dport 53 -j DNAT --to-destination 10.100.1.42:53
>> time : 0.383729934692
>> -A OUTPUT -d 185.20.146.79/32 -p udp -m udp --dport 53 -j DNAT --to-destination 10.100.1.42:53
>> time : 0.383754968643
>> -j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d 10.100.1.42/32 -o eth0 -p udp -m udp --dport 53
>> time : 1.14376091957
>> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p udp -m udp --dport 53 -j MARK --set-xmark 0x2/0xffffffff
>> time : 1.14526605606
>> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p udp -m udp --dport 53 -m state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask 0xffffffff
>> time : 1.14599299431
>> -A FORWARD -i eth2 -o eth0 -p udp -m udp --dport 53 -m state --state NEW,ESTABLISHED -j ACCEPT
>> Total time for creating Policy : 1.52742600441
>> ----------------------------------------------
>> -A PREROUTING -d 185.20.146.56/32 -i eth2 -p tcp -m tcp --dport 22 -j DNAT --to-destination 10.100.1.95:22
>> time : 0.000700950622559
>> -A PREROUTING -d 185.20.146.56/32 -i eth0 -p tcp -m tcp --dport 22 -j DNAT --to-destination 10.100.1.95:22
>> time : 0.382349014282
>> -A OUTPUT -d 185.20.146.56/32 -p tcp -m tcp --dport 22 -j DNAT --to-destination 10.100.1.95:22
>> time : 0.382384061813
>> -j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d 10.100.1.95/32 -o eth0 -p tcp -m tcp --dport 22
>> time : 1.1425909996
>> -A PREROUTING -d 185.20.146.56/32 -i eth2 -p tcp -m tcp --dport 22 -j MARK --set-xmark 0x2/0xffffffff
>> time : 1.14400196075
>> -A PREROUTING -d 185.20.146.56/32 -i eth2 -p tcp -m tcp --dport 22 -m state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask 0xffffffff
>> time : 1.14468812943
>> -A FORWARD -i eth2 -o eth0 -p tcp -m tcp --dport 22 -m state --state NEW,ESTABLISHED -j ACCEPT
>> Total time for creating Policy : 1.52619600296
>> ----------------------------------------------
>>
>> Having a closer look at configure.py how the iptables rules are defined. We think that it is not efficient to lookup these values for every policy:
>>
>> def forward_vr(self, rule):
>>
>> fw1 = "-A PREROUTING -d %s/32 -i %s -p %s -m %s --dport %s -j DNAT --to-destination %s:%s" % \
>> (
>> rule['public_ip'],
>> self.getDeviceByIp(rule['public_ip']),
>> rule['protocol'],
>> rule['protocol'],
>> self.portsToString(rule['public_ports'], ':'),
>> rule['internal_ip'],
>> self.portsToString(rule['internal_ports'], '-')
>> )
>> fw2 = "-A PREROUTING -d %s/32 -i %s -p %s -m %s --dport %s -j DNAT --to-destination %s:%s" % \
>> (
>> rule['public_ip'],
>> self.getDeviceByIp(rule['internal_ip']),
>> rule['protocol'],
>> rule['protocol'],
>> self.portsToString(rule['public_ports'], ':'),
>> rule['internal_ip'],
>> self.portsToString(rule['internal_ports'], '-')
>> .....
>>
>>
>> Defining these values once at the beginning would be much more efficient, no ?
>>
>> def forward_vr(self, rule):
>>
>> pub_interface = self.getDeviceByIp(rule['public_ip'])
>> int_interface = self.getDeviceByIp(rule['internal_ip'])
>> pub_ports = self.portsToString(rule['public_ports'], ':')
>> int_ports = self.portsToString(rule['internal_ports'], '-')
>> int_network = self.getNetworkByIp(rule['internal_ip'])
>>
>> fw1 = "-A PREROUTING -d %s/32 -i %s -p %s -m %s --dport %s -j DNAT --to-destination %s:%s" % \
>> (
>> rule['public_ip'],
>> pub_interface,
>> rule['protocol'],
>> rule['protocol'],
>> pub_ports,
>> rule['internal_ip'],
>> int_ports
>> )
>>
>> fw2 = "-A PREROUTING -d %s/32 -i %s -p %s -m %s --dport %s -j DNAT --to-destination %s:%s" % \
>> (
>> rule['public_ip'],
>> int_interface,
>> rule['protocol'],
>> rule['protocol'],
>> pub_ports,
>> rule['internal_ip'],
>> int_ports
>> )
>> .....
>>
>> If we run the configure.py with these modifications we have the following:
>>
>> root@r-96-VM:~# python /opt/cloud/bin/configure_modified.py /etc/cloudstack/forwardingrules.json
>> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 22 -j DNAT --to-destination 10.100.1.42:22
>> time : 0.000349044799805
>> -A PREROUTING -d 185.20.146.79/32 -i eth0 -p tcp -m tcp --dport 22 -j DNAT --to-destination 10.100.1.42:22
>> time : 0.000686883926392
>> -A OUTPUT -d 185.20.146.79/32 -p tcp -m tcp --dport 22 -j DNAT --to-destination 10.100.1.42:22
>> time : 0.000943899154663
>> -j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d 10.100.1.42/32 -o eth0 -p tcp -m tcp --dport 22
>> time : 0.00131487846375
>> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 22 -j MARK --set-xmark 0x2/0xffffffff
>> time : 0.00161194801331
>> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 22 -m state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask 0xffffffff
>> time : 0.00186896324158
>> -A FORWARD -i eth2 -o eth0 -p tcp -m tcp --dport 22 -m state --state NEW,ESTABLISHED -j ACCEPT
>> Total time for creating Policy : 0.00216102600098
>> ----------------------------------------------
>> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 8443 -j DNAT --to-destination 10.100.1.42:8443
>> time : 0.000232934951782
>> -A PREROUTING -d 185.20.146.79/32 -i eth0 -p tcp -m tcp --dport 8443 -j DNAT --to-destination 10.100.1.42:8443
>> time : 0.000478029251099
>> -A OUTPUT -d 185.20.146.79/32 -p tcp -m tcp --dport 8443 -j DNAT --to-destination 10.100.1.42:8443
>> time : 0.00071907043457
>> -j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d 10.100.1.42/32 -o eth0 -p tcp -m tcp --dport 8443
>> time : 0.000991106033325
>> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 8443 -j MARK --set-xmark 0x2/0xffffffff
>> time : 0.00136613845825
>> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 8443 -m state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask 0xffffffff
>> time : 0.00174498558044
>> -A FORWARD -i eth2 -o eth0 -p tcp -m tcp --dport 8443 -m state --state NEW,ESTABLISHED -j ACCEPT
>> Total time for creating Policy : 0.00219202041626
>> ----------------------------------------------
>> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p udp -m udp --dport 53 -j DNAT --to-destination 10.100.1.42:53
>> time : 0.000226974487305
>> -A PREROUTING -d 185.20.146.79/32 -i eth0 -p udp -m udp --dport 53 -j DNAT --to-destination 10.100.1.42:53
>> time : 0.000502824783325
>> -A OUTPUT -d 185.20.146.79/32 -p udp -m udp --dport 53 -j DNAT --to-destination 10.100.1.42:53
>> time : 0.000762939453125
>> -j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d 10.100.1.42/32 -o eth0 -p udp -m udp --dport 53
>> time : 0.00103092193604
>> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p udp -m udp --dport 53 -j MARK --set-xmark 0x2/0xffffffff
>> time : 0.00134587287903
>> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p udp -m udp --dport 53 -m state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask 0xffffffff
>> time : 0.00158596038818
>> -A FORWARD -i eth2 -o eth0 -p udp -m udp --dport 53 -m state --state NEW,ESTABLISHED -j ACCEPT
>> Total time for creating Policy : 0.00182485580444
>> ----------------------------------------------
>> -A PREROUTING -d 185.20.146.56/32 -i eth2 -p tcp -m tcp --dport 22 -j DNAT --to-destination 10.100.1.95:22
>> time : 0.000264167785645
>> -A PREROUTING -d 185.20.146.56/32 -i eth0 -p tcp -m tcp --dport 22 -j DNAT --to-destination 10.100.1.95:22
>> time : 0.000508069992065
>> -A OUTPUT -d 185.20.146.56/32 -p tcp -m tcp --dport 22 -j DNAT --to-destination 10.100.1.95:22
>> time : 0.000750064849854
>> -j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d 10.100.1.95/32 -o eth0 -p tcp -m tcp --dport 22
>> time : 0.00102114677429
>> -A PREROUTING -d 185.20.146.56/32 -i eth2 -p tcp -m tcp --dport 22 -j MARK --set-xmark 0x2/0xffffffff
>> time : 0.00138115882874
>> -A PREROUTING -d 185.20.146.56/32 -i eth2 -p tcp -m tcp --dport 22 -m state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask 0xffffffff
>> time : 0.00165915489197
>> -A FORWARD -i eth2 -o eth0 -p tcp -m tcp --dport 22 -m state --state NEW,ESTABLISHED -j ACCEPT
>> Total time for creating Policy : 0.00196814537048
>> ----------------------------------------------
>>
>> Location of configure.py:
>> https://github.com/apache/cloudstack/blob/master/systemvm/patches/debian/config/opt/cloud/bin/configure.py
>>
>> The modified scripts are attached. Thanks for your feedback.
>>
>> regards
>> Martin
>
Re: Issue: CLOUDSTACK-9255 Unable to start VM DomainRouter due to
error in finalizeStart
Posted by martin kolly <ma...@senselan.ch>.
Hi Remi
PR #1449 created as requested. Tests in our environment showed that it
speeds up the router configuration quite a bit.
In the meantime https://github.com/apache/cloudstack/pull/1356 seems to
be merged which resolves CLOUDSTACK-9255. So not sure if PR#1449 is
still of interest..
Do you know when the system vm with these fixes (PR1356) is available on
http://cloudstack.apt-get.eu/systemvm/ ?
regards
Martin
On 03/18/2016 09:45 PM, Remi Bergsma wrote:
> Hi Martin,
>
> Thanks for the fix, didn’t catch you attachment first time.
>
> Would it be possible for you to send a Pull Request? Is this patch against master or a release branch? Generally speaking it’s best to make a PR against a release branch, 4.7 would be fine I guess in this case. Once it’s a PR we can test it.
>
> Regards,
> Remi
>
>
> From: martin kolly <ma...@senselan.ch>>
> Reply-To: "dev@cloudstack.apache.org<ma...@cloudstack.apache.org>" <de...@cloudstack.apache.org>>
> Date: Friday 18 March 2016 at 11:58
> To: "dev@cloudstack.apache.org<ma...@cloudstack.apache.org>" <de...@cloudstack.apache.org>>
> Subject: Issue: CLOUDSTACK-9255 Unable to start VM DomainRouter due to error in finalizeStart
>
> Hi All
>
> We are facing the same issue as reported by Milamber (Ticket 9255) https://issues.apache.org/jira/browse/CLOUDSTACK-9255. When deploying a couple of VMs or Port Forwarding's the re-deployment of the router with cleanup fails.
>
> We found that iptables configuration takes a lot of time, this eventually leads to a timeout on the management server "Unable to start VM DomainRouter due to error in finalizeStart, not retrying"
>
> Environment:
> - Cloudstack 4.8
> - KVM (local storage)
> - hosts/mgr on Ubuntu 14.04
>
> We tested with a simple set of four forwarding rules, here the setup:
>
> root@r-96-VM:~# cat /etc/cloudstack/forwardingrules.json
> {
> "185.20.146.56": [
> {
> "internal_ip": "10.100.1.95",
> "internal_ports": "22:22",
> "protocol": "tcp",
> "public_ip": "185.20.146.56",
> "public_ports": "22:22",
> "type": "forward"
> }
> ],
> "185.20.146.79": [
> {
> "internal_ip": "10.100.1.42",
> "internal_ports": "22:22",
> "protocol": "tcp",
> "public_ip": "185.20.146.79",
> "public_ports": "22:22",
> "type": "forward"
> },
> {
> "internal_ip": "10.100.1.42",
> "internal_ports": "8443:8443",
> "protocol": "tcp",
> "public_ip": "185.20.146.79",
> "public_ports": "8443:8443",
> "type": "forward"
> },
> {
> "internal_ip": "10.100.1.42",
> "internal_ports": "53:53",
> "protocol": "udp",
> "public_ip": "185.20.146.79",
> "public_ports": "53:53",
> "type": "forward"
> }
> ],
> "id": "forwardingrules"
>
> The definition for every port forwarding seems to take at ~1.5 seconds.
>
> python /opt/cloud/bin/configure.py.timed /etc/cloudstack/forwardingrules.json
>
> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 22 -j DNAT --to-destination 10.100.1.42:22
> time : 0.000965118408203
> -A PREROUTING -d 185.20.146.79/32 -i eth0 -p tcp -m tcp --dport 22 -j DNAT --to-destination 10.100.1.42:22
> time : 0.395485162735
> -A OUTPUT -d 185.20.146.79/32 -p tcp -m tcp --dport 22 -j DNAT --to-destination 10.100.1.42:22
> time : 0.395533084869
> -j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d 10.100.1.42/32 -o eth0 -p tcp -m tcp --dport 22
> time : 1.16180706024
> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 22 -j MARK --set-xmark 0x2/0xffffffff
> time : 1.16329216957
> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 22 -m state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask 0xffffffff
> time : 1.16407108307
> -A FORWARD -i eth2 -o eth0 -p tcp -m tcp --dport 22 -m state --state NEW,ESTABLISHED -j ACCEPT
> Total time for creating Policy : 1.53959512711
> ----------------------------------------------
> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 8443 -j DNAT --to-destination 10.100.1.42:8443
> time : 0.000781059265137
> -A PREROUTING -d 185.20.146.79/32 -i eth0 -p tcp -m tcp --dport 8443 -j DNAT --to-destination 10.100.1.42:8443
> time : 0.378201007843
> -A OUTPUT -d 185.20.146.79/32 -p tcp -m tcp --dport 8443 -j DNAT --to-destination 10.100.1.42:8443
> time : 0.37822508812
> -j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d 10.100.1.42/32 -o eth0 -p tcp -m tcp --dport 8443
> time : 1.14627504349
> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 8443 -j MARK --set-xmark 0x2/0xffffffff
> time : 1.1477329731
> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 8443 -m state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask 0xffffffff
> time : 1.14850592613
> -A FORWARD -i eth2 -o eth0 -p tcp -m tcp --dport 8443 -m state --state NEW,ESTABLISHED -j ACCEPT
> Total time for creating Policy : 1.52321791649
> ----------------------------------------------
> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p udp -m udp --dport 53 -j DNAT --to-destination 10.100.1.42:53
> time : 0.000754117965698
> -A PREROUTING -d 185.20.146.79/32 -i eth0 -p udp -m udp --dport 53 -j DNAT --to-destination 10.100.1.42:53
> time : 0.383729934692
> -A OUTPUT -d 185.20.146.79/32 -p udp -m udp --dport 53 -j DNAT --to-destination 10.100.1.42:53
> time : 0.383754968643
> -j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d 10.100.1.42/32 -o eth0 -p udp -m udp --dport 53
> time : 1.14376091957
> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p udp -m udp --dport 53 -j MARK --set-xmark 0x2/0xffffffff
> time : 1.14526605606
> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p udp -m udp --dport 53 -m state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask 0xffffffff
> time : 1.14599299431
> -A FORWARD -i eth2 -o eth0 -p udp -m udp --dport 53 -m state --state NEW,ESTABLISHED -j ACCEPT
> Total time for creating Policy : 1.52742600441
> ----------------------------------------------
> -A PREROUTING -d 185.20.146.56/32 -i eth2 -p tcp -m tcp --dport 22 -j DNAT --to-destination 10.100.1.95:22
> time : 0.000700950622559
> -A PREROUTING -d 185.20.146.56/32 -i eth0 -p tcp -m tcp --dport 22 -j DNAT --to-destination 10.100.1.95:22
> time : 0.382349014282
> -A OUTPUT -d 185.20.146.56/32 -p tcp -m tcp --dport 22 -j DNAT --to-destination 10.100.1.95:22
> time : 0.382384061813
> -j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d 10.100.1.95/32 -o eth0 -p tcp -m tcp --dport 22
> time : 1.1425909996
> -A PREROUTING -d 185.20.146.56/32 -i eth2 -p tcp -m tcp --dport 22 -j MARK --set-xmark 0x2/0xffffffff
> time : 1.14400196075
> -A PREROUTING -d 185.20.146.56/32 -i eth2 -p tcp -m tcp --dport 22 -m state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask 0xffffffff
> time : 1.14468812943
> -A FORWARD -i eth2 -o eth0 -p tcp -m tcp --dport 22 -m state --state NEW,ESTABLISHED -j ACCEPT
> Total time for creating Policy : 1.52619600296
> ----------------------------------------------
>
> Having a closer look at configure.py how the iptables rules are defined. We think that it is not efficient to lookup these values for every policy:
>
> def forward_vr(self, rule):
>
> fw1 = "-A PREROUTING -d %s/32 -i %s -p %s -m %s --dport %s -j DNAT --to-destination %s:%s" % \
> (
> rule['public_ip'],
> self.getDeviceByIp(rule['public_ip']),
> rule['protocol'],
> rule['protocol'],
> self.portsToString(rule['public_ports'], ':'),
> rule['internal_ip'],
> self.portsToString(rule['internal_ports'], '-')
> )
> fw2 = "-A PREROUTING -d %s/32 -i %s -p %s -m %s --dport %s -j DNAT --to-destination %s:%s" % \
> (
> rule['public_ip'],
> self.getDeviceByIp(rule['internal_ip']),
> rule['protocol'],
> rule['protocol'],
> self.portsToString(rule['public_ports'], ':'),
> rule['internal_ip'],
> self.portsToString(rule['internal_ports'], '-')
> .....
>
>
> Defining these values once at the beginning would be much more efficient, no ?
>
> def forward_vr(self, rule):
>
> pub_interface = self.getDeviceByIp(rule['public_ip'])
> int_interface = self.getDeviceByIp(rule['internal_ip'])
> pub_ports = self.portsToString(rule['public_ports'], ':')
> int_ports = self.portsToString(rule['internal_ports'], '-')
> int_network = self.getNetworkByIp(rule['internal_ip'])
>
> fw1 = "-A PREROUTING -d %s/32 -i %s -p %s -m %s --dport %s -j DNAT --to-destination %s:%s" % \
> (
> rule['public_ip'],
> pub_interface,
> rule['protocol'],
> rule['protocol'],
> pub_ports,
> rule['internal_ip'],
> int_ports
> )
>
> fw2 = "-A PREROUTING -d %s/32 -i %s -p %s -m %s --dport %s -j DNAT --to-destination %s:%s" % \
> (
> rule['public_ip'],
> int_interface,
> rule['protocol'],
> rule['protocol'],
> pub_ports,
> rule['internal_ip'],
> int_ports
> )
> .....
>
> If we run the configure.py with these modifications we have the following:
>
> root@r-96-VM:~# python /opt/cloud/bin/configure_modified.py /etc/cloudstack/forwardingrules.json
> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 22 -j DNAT --to-destination 10.100.1.42:22
> time : 0.000349044799805
> -A PREROUTING -d 185.20.146.79/32 -i eth0 -p tcp -m tcp --dport 22 -j DNAT --to-destination 10.100.1.42:22
> time : 0.000686883926392
> -A OUTPUT -d 185.20.146.79/32 -p tcp -m tcp --dport 22 -j DNAT --to-destination 10.100.1.42:22
> time : 0.000943899154663
> -j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d 10.100.1.42/32 -o eth0 -p tcp -m tcp --dport 22
> time : 0.00131487846375
> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 22 -j MARK --set-xmark 0x2/0xffffffff
> time : 0.00161194801331
> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 22 -m state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask 0xffffffff
> time : 0.00186896324158
> -A FORWARD -i eth2 -o eth0 -p tcp -m tcp --dport 22 -m state --state NEW,ESTABLISHED -j ACCEPT
> Total time for creating Policy : 0.00216102600098
> ----------------------------------------------
> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 8443 -j DNAT --to-destination 10.100.1.42:8443
> time : 0.000232934951782
> -A PREROUTING -d 185.20.146.79/32 -i eth0 -p tcp -m tcp --dport 8443 -j DNAT --to-destination 10.100.1.42:8443
> time : 0.000478029251099
> -A OUTPUT -d 185.20.146.79/32 -p tcp -m tcp --dport 8443 -j DNAT --to-destination 10.100.1.42:8443
> time : 0.00071907043457
> -j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d 10.100.1.42/32 -o eth0 -p tcp -m tcp --dport 8443
> time : 0.000991106033325
> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 8443 -j MARK --set-xmark 0x2/0xffffffff
> time : 0.00136613845825
> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 8443 -m state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask 0xffffffff
> time : 0.00174498558044
> -A FORWARD -i eth2 -o eth0 -p tcp -m tcp --dport 8443 -m state --state NEW,ESTABLISHED -j ACCEPT
> Total time for creating Policy : 0.00219202041626
> ----------------------------------------------
> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p udp -m udp --dport 53 -j DNAT --to-destination 10.100.1.42:53
> time : 0.000226974487305
> -A PREROUTING -d 185.20.146.79/32 -i eth0 -p udp -m udp --dport 53 -j DNAT --to-destination 10.100.1.42:53
> time : 0.000502824783325
> -A OUTPUT -d 185.20.146.79/32 -p udp -m udp --dport 53 -j DNAT --to-destination 10.100.1.42:53
> time : 0.000762939453125
> -j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d 10.100.1.42/32 -o eth0 -p udp -m udp --dport 53
> time : 0.00103092193604
> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p udp -m udp --dport 53 -j MARK --set-xmark 0x2/0xffffffff
> time : 0.00134587287903
> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p udp -m udp --dport 53 -m state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask 0xffffffff
> time : 0.00158596038818
> -A FORWARD -i eth2 -o eth0 -p udp -m udp --dport 53 -m state --state NEW,ESTABLISHED -j ACCEPT
> Total time for creating Policy : 0.00182485580444
> ----------------------------------------------
> -A PREROUTING -d 185.20.146.56/32 -i eth2 -p tcp -m tcp --dport 22 -j DNAT --to-destination 10.100.1.95:22
> time : 0.000264167785645
> -A PREROUTING -d 185.20.146.56/32 -i eth0 -p tcp -m tcp --dport 22 -j DNAT --to-destination 10.100.1.95:22
> time : 0.000508069992065
> -A OUTPUT -d 185.20.146.56/32 -p tcp -m tcp --dport 22 -j DNAT --to-destination 10.100.1.95:22
> time : 0.000750064849854
> -j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d 10.100.1.95/32 -o eth0 -p tcp -m tcp --dport 22
> time : 0.00102114677429
> -A PREROUTING -d 185.20.146.56/32 -i eth2 -p tcp -m tcp --dport 22 -j MARK --set-xmark 0x2/0xffffffff
> time : 0.00138115882874
> -A PREROUTING -d 185.20.146.56/32 -i eth2 -p tcp -m tcp --dport 22 -m state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask 0xffffffff
> time : 0.00165915489197
> -A FORWARD -i eth2 -o eth0 -p tcp -m tcp --dport 22 -m state --state NEW,ESTABLISHED -j ACCEPT
> Total time for creating Policy : 0.00196814537048
> ----------------------------------------------
>
> Location of configure.py:
> https://github.com/apache/cloudstack/blob/master/systemvm/patches/debian/config/opt/cloud/bin/configure.py
>
> The modified scripts are attached. Thanks for your feedback.
>
> regards
> Martin
>
Re: Issue: CLOUDSTACK-9255 Unable to start VM DomainRouter due to
error in finalizeStart
Posted by Remi Bergsma <RB...@schubergphilis.com>.
Hi Martin,
Thanks for the fix, didn’t catch you attachment first time.
Would it be possible for you to send a Pull Request? Is this patch against master or a release branch? Generally speaking it’s best to make a PR against a release branch, 4.7 would be fine I guess in this case. Once it’s a PR we can test it.
Regards,
Remi
From: martin kolly <ma...@senselan.ch>>
Reply-To: "dev@cloudstack.apache.org<ma...@cloudstack.apache.org>" <de...@cloudstack.apache.org>>
Date: Friday 18 March 2016 at 11:58
To: "dev@cloudstack.apache.org<ma...@cloudstack.apache.org>" <de...@cloudstack.apache.org>>
Subject: Issue: CLOUDSTACK-9255 Unable to start VM DomainRouter due to error in finalizeStart
Hi All
We are facing the same issue as reported by Milamber (Ticket 9255) https://issues.apache.org/jira/browse/CLOUDSTACK-9255. When deploying a couple of VMs or Port Forwarding's the re-deployment of the router with cleanup fails.
We found that iptables configuration takes a lot of time, this eventually leads to a timeout on the management server "Unable to start VM DomainRouter due to error in finalizeStart, not retrying"
Environment:
- Cloudstack 4.8
- KVM (local storage)
- hosts/mgr on Ubuntu 14.04
We tested with a simple set of four forwarding rules, here the setup:
root@r-96-VM:~# cat /etc/cloudstack/forwardingrules.json
{
"185.20.146.56": [
{
"internal_ip": "10.100.1.95",
"internal_ports": "22:22",
"protocol": "tcp",
"public_ip": "185.20.146.56",
"public_ports": "22:22",
"type": "forward"
}
],
"185.20.146.79": [
{
"internal_ip": "10.100.1.42",
"internal_ports": "22:22",
"protocol": "tcp",
"public_ip": "185.20.146.79",
"public_ports": "22:22",
"type": "forward"
},
{
"internal_ip": "10.100.1.42",
"internal_ports": "8443:8443",
"protocol": "tcp",
"public_ip": "185.20.146.79",
"public_ports": "8443:8443",
"type": "forward"
},
{
"internal_ip": "10.100.1.42",
"internal_ports": "53:53",
"protocol": "udp",
"public_ip": "185.20.146.79",
"public_ports": "53:53",
"type": "forward"
}
],
"id": "forwardingrules"
The definition for every port forwarding seems to take at ~1.5 seconds.
python /opt/cloud/bin/configure.py.timed /etc/cloudstack/forwardingrules.json
-A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 22 -j DNAT --to-destination 10.100.1.42:22
time : 0.000965118408203
-A PREROUTING -d 185.20.146.79/32 -i eth0 -p tcp -m tcp --dport 22 -j DNAT --to-destination 10.100.1.42:22
time : 0.395485162735
-A OUTPUT -d 185.20.146.79/32 -p tcp -m tcp --dport 22 -j DNAT --to-destination 10.100.1.42:22
time : 0.395533084869
-j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d 10.100.1.42/32 -o eth0 -p tcp -m tcp --dport 22
time : 1.16180706024
-A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 22 -j MARK --set-xmark 0x2/0xffffffff
time : 1.16329216957
-A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 22 -m state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask 0xffffffff
time : 1.16407108307
-A FORWARD -i eth2 -o eth0 -p tcp -m tcp --dport 22 -m state --state NEW,ESTABLISHED -j ACCEPT
Total time for creating Policy : 1.53959512711
----------------------------------------------
-A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 8443 -j DNAT --to-destination 10.100.1.42:8443
time : 0.000781059265137
-A PREROUTING -d 185.20.146.79/32 -i eth0 -p tcp -m tcp --dport 8443 -j DNAT --to-destination 10.100.1.42:8443
time : 0.378201007843
-A OUTPUT -d 185.20.146.79/32 -p tcp -m tcp --dport 8443 -j DNAT --to-destination 10.100.1.42:8443
time : 0.37822508812
-j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d 10.100.1.42/32 -o eth0 -p tcp -m tcp --dport 8443
time : 1.14627504349
-A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 8443 -j MARK --set-xmark 0x2/0xffffffff
time : 1.1477329731
-A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 8443 -m state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask 0xffffffff
time : 1.14850592613
-A FORWARD -i eth2 -o eth0 -p tcp -m tcp --dport 8443 -m state --state NEW,ESTABLISHED -j ACCEPT
Total time for creating Policy : 1.52321791649
----------------------------------------------
-A PREROUTING -d 185.20.146.79/32 -i eth2 -p udp -m udp --dport 53 -j DNAT --to-destination 10.100.1.42:53
time : 0.000754117965698
-A PREROUTING -d 185.20.146.79/32 -i eth0 -p udp -m udp --dport 53 -j DNAT --to-destination 10.100.1.42:53
time : 0.383729934692
-A OUTPUT -d 185.20.146.79/32 -p udp -m udp --dport 53 -j DNAT --to-destination 10.100.1.42:53
time : 0.383754968643
-j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d 10.100.1.42/32 -o eth0 -p udp -m udp --dport 53
time : 1.14376091957
-A PREROUTING -d 185.20.146.79/32 -i eth2 -p udp -m udp --dport 53 -j MARK --set-xmark 0x2/0xffffffff
time : 1.14526605606
-A PREROUTING -d 185.20.146.79/32 -i eth2 -p udp -m udp --dport 53 -m state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask 0xffffffff
time : 1.14599299431
-A FORWARD -i eth2 -o eth0 -p udp -m udp --dport 53 -m state --state NEW,ESTABLISHED -j ACCEPT
Total time for creating Policy : 1.52742600441
----------------------------------------------
-A PREROUTING -d 185.20.146.56/32 -i eth2 -p tcp -m tcp --dport 22 -j DNAT --to-destination 10.100.1.95:22
time : 0.000700950622559
-A PREROUTING -d 185.20.146.56/32 -i eth0 -p tcp -m tcp --dport 22 -j DNAT --to-destination 10.100.1.95:22
time : 0.382349014282
-A OUTPUT -d 185.20.146.56/32 -p tcp -m tcp --dport 22 -j DNAT --to-destination 10.100.1.95:22
time : 0.382384061813
-j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d 10.100.1.95/32 -o eth0 -p tcp -m tcp --dport 22
time : 1.1425909996
-A PREROUTING -d 185.20.146.56/32 -i eth2 -p tcp -m tcp --dport 22 -j MARK --set-xmark 0x2/0xffffffff
time : 1.14400196075
-A PREROUTING -d 185.20.146.56/32 -i eth2 -p tcp -m tcp --dport 22 -m state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask 0xffffffff
time : 1.14468812943
-A FORWARD -i eth2 -o eth0 -p tcp -m tcp --dport 22 -m state --state NEW,ESTABLISHED -j ACCEPT
Total time for creating Policy : 1.52619600296
----------------------------------------------
Having a closer look at configure.py how the iptables rules are defined. We think that it is not efficient to lookup these values for every policy:
def forward_vr(self, rule):
fw1 = "-A PREROUTING -d %s/32 -i %s -p %s -m %s --dport %s -j DNAT --to-destination %s:%s" % \
(
rule['public_ip'],
self.getDeviceByIp(rule['public_ip']),
rule['protocol'],
rule['protocol'],
self.portsToString(rule['public_ports'], ':'),
rule['internal_ip'],
self.portsToString(rule['internal_ports'], '-')
)
fw2 = "-A PREROUTING -d %s/32 -i %s -p %s -m %s --dport %s -j DNAT --to-destination %s:%s" % \
(
rule['public_ip'],
self.getDeviceByIp(rule['internal_ip']),
rule['protocol'],
rule['protocol'],
self.portsToString(rule['public_ports'], ':'),
rule['internal_ip'],
self.portsToString(rule['internal_ports'], '-')
.....
Defining these values once at the beginning would be much more efficient, no ?
def forward_vr(self, rule):
pub_interface = self.getDeviceByIp(rule['public_ip'])
int_interface = self.getDeviceByIp(rule['internal_ip'])
pub_ports = self.portsToString(rule['public_ports'], ':')
int_ports = self.portsToString(rule['internal_ports'], '-')
int_network = self.getNetworkByIp(rule['internal_ip'])
fw1 = "-A PREROUTING -d %s/32 -i %s -p %s -m %s --dport %s -j DNAT --to-destination %s:%s" % \
(
rule['public_ip'],
pub_interface,
rule['protocol'],
rule['protocol'],
pub_ports,
rule['internal_ip'],
int_ports
)
fw2 = "-A PREROUTING -d %s/32 -i %s -p %s -m %s --dport %s -j DNAT --to-destination %s:%s" % \
(
rule['public_ip'],
int_interface,
rule['protocol'],
rule['protocol'],
pub_ports,
rule['internal_ip'],
int_ports
)
.....
If we run the configure.py with these modifications we have the following:
root@r-96-VM:~# python /opt/cloud/bin/configure_modified.py /etc/cloudstack/forwardingrules.json
-A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 22 -j DNAT --to-destination 10.100.1.42:22
time : 0.000349044799805
-A PREROUTING -d 185.20.146.79/32 -i eth0 -p tcp -m tcp --dport 22 -j DNAT --to-destination 10.100.1.42:22
time : 0.000686883926392
-A OUTPUT -d 185.20.146.79/32 -p tcp -m tcp --dport 22 -j DNAT --to-destination 10.100.1.42:22
time : 0.000943899154663
-j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d 10.100.1.42/32 -o eth0 -p tcp -m tcp --dport 22
time : 0.00131487846375
-A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 22 -j MARK --set-xmark 0x2/0xffffffff
time : 0.00161194801331
-A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 22 -m state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask 0xffffffff
time : 0.00186896324158
-A FORWARD -i eth2 -o eth0 -p tcp -m tcp --dport 22 -m state --state NEW,ESTABLISHED -j ACCEPT
Total time for creating Policy : 0.00216102600098
----------------------------------------------
-A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 8443 -j DNAT --to-destination 10.100.1.42:8443
time : 0.000232934951782
-A PREROUTING -d 185.20.146.79/32 -i eth0 -p tcp -m tcp --dport 8443 -j DNAT --to-destination 10.100.1.42:8443
time : 0.000478029251099
-A OUTPUT -d 185.20.146.79/32 -p tcp -m tcp --dport 8443 -j DNAT --to-destination 10.100.1.42:8443
time : 0.00071907043457
-j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d 10.100.1.42/32 -o eth0 -p tcp -m tcp --dport 8443
time : 0.000991106033325
-A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 8443 -j MARK --set-xmark 0x2/0xffffffff
time : 0.00136613845825
-A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 8443 -m state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask 0xffffffff
time : 0.00174498558044
-A FORWARD -i eth2 -o eth0 -p tcp -m tcp --dport 8443 -m state --state NEW,ESTABLISHED -j ACCEPT
Total time for creating Policy : 0.00219202041626
----------------------------------------------
-A PREROUTING -d 185.20.146.79/32 -i eth2 -p udp -m udp --dport 53 -j DNAT --to-destination 10.100.1.42:53
time : 0.000226974487305
-A PREROUTING -d 185.20.146.79/32 -i eth0 -p udp -m udp --dport 53 -j DNAT --to-destination 10.100.1.42:53
time : 0.000502824783325
-A OUTPUT -d 185.20.146.79/32 -p udp -m udp --dport 53 -j DNAT --to-destination 10.100.1.42:53
time : 0.000762939453125
-j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d 10.100.1.42/32 -o eth0 -p udp -m udp --dport 53
time : 0.00103092193604
-A PREROUTING -d 185.20.146.79/32 -i eth2 -p udp -m udp --dport 53 -j MARK --set-xmark 0x2/0xffffffff
time : 0.00134587287903
-A PREROUTING -d 185.20.146.79/32 -i eth2 -p udp -m udp --dport 53 -m state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask 0xffffffff
time : 0.00158596038818
-A FORWARD -i eth2 -o eth0 -p udp -m udp --dport 53 -m state --state NEW,ESTABLISHED -j ACCEPT
Total time for creating Policy : 0.00182485580444
----------------------------------------------
-A PREROUTING -d 185.20.146.56/32 -i eth2 -p tcp -m tcp --dport 22 -j DNAT --to-destination 10.100.1.95:22
time : 0.000264167785645
-A PREROUTING -d 185.20.146.56/32 -i eth0 -p tcp -m tcp --dport 22 -j DNAT --to-destination 10.100.1.95:22
time : 0.000508069992065
-A OUTPUT -d 185.20.146.56/32 -p tcp -m tcp --dport 22 -j DNAT --to-destination 10.100.1.95:22
time : 0.000750064849854
-j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d 10.100.1.95/32 -o eth0 -p tcp -m tcp --dport 22
time : 0.00102114677429
-A PREROUTING -d 185.20.146.56/32 -i eth2 -p tcp -m tcp --dport 22 -j MARK --set-xmark 0x2/0xffffffff
time : 0.00138115882874
-A PREROUTING -d 185.20.146.56/32 -i eth2 -p tcp -m tcp --dport 22 -m state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask 0xffffffff
time : 0.00165915489197
-A FORWARD -i eth2 -o eth0 -p tcp -m tcp --dport 22 -m state --state NEW,ESTABLISHED -j ACCEPT
Total time for creating Policy : 0.00196814537048
----------------------------------------------
Location of configure.py:
https://github.com/apache/cloudstack/blob/master/systemvm/patches/debian/config/opt/cloud/bin/configure.py
The modified scripts are attached. Thanks for your feedback.
regards
Martin
Re: Issue: CLOUDSTACK-9255 Unable to start VM DomainRouter due to
error in finalizeStart
Posted by Wei ZHOU <us...@gmail.com>.
nice !!!!!!
2016-03-18 11:58 GMT+01:00 martin kolly <ma...@senselan.ch>:
> Hi All
>
> We are facing the same issue as reported by Milamber (Ticket 9255)
> https://issues.apache.org/jira/browse/CLOUDSTACK-9255. When deploying a
> couple of VMs or Port Forwarding's the re-deployment of the router with
> cleanup fails.
>
> We found that iptables configuration takes a lot of time, this eventually
> leads to a timeout on the management server "Unable to start VM
> DomainRouter due to error in finalizeStart, not retrying"
>
> Environment:
> - Cloudstack 4.8
> - KVM (local storage)
> - hosts/mgr on Ubuntu 14.04
>
> We tested with a simple set of four forwarding rules, here the setup:
>
> root@r-96-VM:~# cat /etc/cloudstack/forwardingrules.json
> {
> "185.20.146.56": [
> {
> "internal_ip": "10.100.1.95",
> "internal_ports": "22:22",
> "protocol": "tcp",
> "public_ip": "185.20.146.56",
> "public_ports": "22:22",
> "type": "forward"
> }
> ],
> "185.20.146.79": [
> {
> "internal_ip": "10.100.1.42",
> "internal_ports": "22:22",
> "protocol": "tcp",
> "public_ip": "185.20.146.79",
> "public_ports": "22:22",
> "type": "forward"
> },
> {
> "internal_ip": "10.100.1.42",
> "internal_ports": "8443:8443",
> "protocol": "tcp",
> "public_ip": "185.20.146.79",
> "public_ports": "8443:8443",
> "type": "forward"
> },
> {
> "internal_ip": "10.100.1.42",
> "internal_ports": "53:53",
> "protocol": "udp",
> "public_ip": "185.20.146.79",
> "public_ports": "53:53",
> "type": "forward"
> }
> ],
> "id": "forwardingrules"
>
> The definition for every port forwarding seems to take at ~1.5 seconds.
>
> python /opt/cloud/bin/configure.py.timed
> /etc/cloudstack/forwardingrules.json
>
> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 22 -j
> DNAT --to-destination 10.100.1.42:22
> time : 0.000965118408203
> -A PREROUTING -d 185.20.146.79/32 -i eth0 -p tcp -m tcp --dport 22 -j
> DNAT --to-destination 10.100.1.42:22
> time : 0.395485162735
> -A OUTPUT -d 185.20.146.79/32 -p tcp -m tcp --dport 22 -j DNAT
> --to-destination 10.100.1.42:22
> time : 0.395533084869
> -j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d
> 10.100.1.42/32 -o eth0 -p tcp -m tcp --dport 22
> time : 1.16180706024
> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 22 -j
> MARK --set-xmark 0x2/0xffffffff
> time : 1.16329216957
> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 22 -m
> state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask
> 0xffffffff
> time : 1.16407108307
> -A FORWARD -i eth2 -o eth0 -p tcp -m tcp --dport 22 -m state --state
> NEW,ESTABLISHED -j ACCEPT
> Total time for creating Policy : 1.53959512711
> ----------------------------------------------
> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 8443 -j
> DNAT --to-destination 10.100.1.42:8443
> time : 0.000781059265137
> -A PREROUTING -d 185.20.146.79/32 -i eth0 -p tcp -m tcp --dport 8443 -j
> DNAT --to-destination 10.100.1.42:8443
> time : 0.378201007843
> -A OUTPUT -d 185.20.146.79/32 -p tcp -m tcp --dport 8443 -j DNAT
> --to-destination 10.100.1.42:8443
> time : 0.37822508812
> -j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d
> 10.100.1.42/32 -o eth0 -p tcp -m tcp --dport 8443
> time : 1.14627504349
> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 8443 -j
> MARK --set-xmark 0x2/0xffffffff
> time : 1.1477329731
> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 8443 -m
> state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask
> 0xffffffff
> time : 1.14850592613
> -A FORWARD -i eth2 -o eth0 -p tcp -m tcp --dport 8443 -m state --state
> NEW,ESTABLISHED -j ACCEPT
> Total time for creating Policy : 1.52321791649
> ----------------------------------------------
> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p udp -m udp --dport 53 -j
> DNAT --to-destination 10.100.1.42:53
> time : 0.000754117965698
> -A PREROUTING -d 185.20.146.79/32 -i eth0 -p udp -m udp --dport 53 -j
> DNAT --to-destination 10.100.1.42:53
> time : 0.383729934692
> -A OUTPUT -d 185.20.146.79/32 -p udp -m udp --dport 53 -j DNAT
> --to-destination 10.100.1.42:53
> time : 0.383754968643
> -j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d
> 10.100.1.42/32 -o eth0 -p udp -m udp --dport 53
> time : 1.14376091957
> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p udp -m udp --dport 53 -j
> MARK --set-xmark 0x2/0xffffffff
> time : 1.14526605606
> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p udp -m udp --dport 53 -m
> state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask
> 0xffffffff
> time : 1.14599299431
> -A FORWARD -i eth2 -o eth0 -p udp -m udp --dport 53 -m state --state
> NEW,ESTABLISHED -j ACCEPT
> Total time for creating Policy : 1.52742600441
> ----------------------------------------------
> -A PREROUTING -d 185.20.146.56/32 -i eth2 -p tcp -m tcp --dport 22 -j
> DNAT --to-destination 10.100.1.95:22
> time : 0.000700950622559
> -A PREROUTING -d 185.20.146.56/32 -i eth0 -p tcp -m tcp --dport 22 -j
> DNAT --to-destination 10.100.1.95:22
> time : 0.382349014282
> -A OUTPUT -d 185.20.146.56/32 -p tcp -m tcp --dport 22 -j DNAT
> --to-destination 10.100.1.95:22
> time : 0.382384061813
> -j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d
> 10.100.1.95/32 -o eth0 -p tcp -m tcp --dport 22
> time : 1.1425909996
> -A PREROUTING -d 185.20.146.56/32 -i eth2 -p tcp -m tcp --dport 22 -j
> MARK --set-xmark 0x2/0xffffffff
> time : 1.14400196075
> -A PREROUTING -d 185.20.146.56/32 -i eth2 -p tcp -m tcp --dport 22 -m
> state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask
> 0xffffffff
> time : 1.14468812943
> -A FORWARD -i eth2 -o eth0 -p tcp -m tcp --dport 22 -m state --state
> NEW,ESTABLISHED -j ACCEPT
> Total time for creating Policy : 1.52619600296
> ----------------------------------------------
>
> Having a closer look at configure.py how the iptables rules are defined.
> We think that it is not efficient to lookup these values for every policy:
>
> def forward_vr(self, rule):
>
> fw1 = "-A PREROUTING -d %s/32 -i %s -p %s -m %s --dport %s -j DNAT
> --to-destination %s:%s" % \
> (
> rule['public_ip'],
> *self.getDeviceByIp(rule['public_ip']),*
> rule['protocol'],
> rule['protocol'],
> *self.portsToString(rule['public_ports'], ':'),*
> rule['internal_ip'],
> *self.portsToString(rule['internal_ports'], '-')*
> )
> fw2 = "-A PREROUTING -d %s/32 -i %s -p %s -m %s --dport %s -j DNAT
> --to-destination %s:%s" % \
> (
> rule['public_ip'],
> * self.getDeviceByIp(rule['internal_ip']),*
> rule['protocol'],
> rule['protocol'],
> * self.portsToString(rule['public_ports'], ':'),*
> rule['internal_ip'],
>
>
> * self.portsToString(rule['internal_ports'], '-') ..... *
>
> Defining these values once at the beginning would be much more efficient,
> no ?
>
> def forward_vr(self, rule):
>
> * pub_interface = self.getDeviceByIp(rule['public_ip'])*
> * int_interface = self.getDeviceByIp(rule['internal_ip'])*
> * pub_ports = self.portsToString(rule['public_ports'], ':')*
> * int_ports = self.portsToString(rule['internal_ports'], '-')*
>
>
> * int_network = self.getNetworkByIp(rule['internal_ip']) * fw1 =
> "-A PREROUTING -d %s/32 -i %s -p %s -m %s --dport %s -j DNAT
> --to-destination %s:%s" % \
> (
> rule['public_ip'],
> pub_interface,
> rule['protocol'],
> rule['protocol'],
> pub_ports,
> rule['internal_ip'],
> int_ports
> )
>
> fw2 = "-A PREROUTING -d %s/32 -i %s -p %s -m %s --dport %s -j DNAT
> --to-destination %s:%s" % \
> (
> rule['public_ip'],
> int_interface,
> rule['protocol'],
> rule['protocol'],
> pub_ports,
> rule['internal_ip'],
> int_ports
> )
> .....
>
> If we run the configure.py with these modifications we have the following:
>
> root@r-96-VM:~# python /opt/cloud/bin/configure_modified.py
> /etc/cloudstack/forwardingrules.json
> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 22 -j
> DNAT --to-destination 10.100.1.42:22
> time : 0.000349044799805
> -A PREROUTING -d 185.20.146.79/32 -i eth0 -p tcp -m tcp --dport 22 -j
> DNAT --to-destination 10.100.1.42:22
> time : 0.000686883926392
> -A OUTPUT -d 185.20.146.79/32 -p tcp -m tcp --dport 22 -j DNAT
> --to-destination 10.100.1.42:22
> time : 0.000943899154663
> -j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d
> 10.100.1.42/32 -o eth0 -p tcp -m tcp --dport 22
> time : 0.00131487846375
> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 22 -j
> MARK --set-xmark 0x2/0xffffffff
> time : 0.00161194801331
> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 22 -m
> state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask
> 0xffffffff
> time : 0.00186896324158
> -A FORWARD -i eth2 -o eth0 -p tcp -m tcp --dport 22 -m state --state
> NEW,ESTABLISHED -j ACCEPT
> Total time for creating Policy : 0.00216102600098
> ----------------------------------------------
> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 8443 -j
> DNAT --to-destination 10.100.1.42:8443
> time : 0.000232934951782
> -A PREROUTING -d 185.20.146.79/32 -i eth0 -p tcp -m tcp --dport 8443 -j
> DNAT --to-destination 10.100.1.42:8443
> time : 0.000478029251099
> -A OUTPUT -d 185.20.146.79/32 -p tcp -m tcp --dport 8443 -j DNAT
> --to-destination 10.100.1.42:8443
> time : 0.00071907043457
> -j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d
> 10.100.1.42/32 -o eth0 -p tcp -m tcp --dport 8443
> time : 0.000991106033325
> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 8443 -j
> MARK --set-xmark 0x2/0xffffffff
> time : 0.00136613845825
> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 8443 -m
> state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask
> 0xffffffff
> time : 0.00174498558044
> -A FORWARD -i eth2 -o eth0 -p tcp -m tcp --dport 8443 -m state --state
> NEW,ESTABLISHED -j ACCEPT
> Total time for creating Policy : 0.00219202041626
> ----------------------------------------------
> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p udp -m udp --dport 53 -j
> DNAT --to-destination 10.100.1.42:53
> time : 0.000226974487305
> -A PREROUTING -d 185.20.146.79/32 -i eth0 -p udp -m udp --dport 53 -j
> DNAT --to-destination 10.100.1.42:53
> time : 0.000502824783325
> -A OUTPUT -d 185.20.146.79/32 -p udp -m udp --dport 53 -j DNAT
> --to-destination 10.100.1.42:53
> time : 0.000762939453125
> -j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d
> 10.100.1.42/32 -o eth0 -p udp -m udp --dport 53
> time : 0.00103092193604
> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p udp -m udp --dport 53 -j
> MARK --set-xmark 0x2/0xffffffff
> time : 0.00134587287903
> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p udp -m udp --dport 53 -m
> state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask
> 0xffffffff
> time : 0.00158596038818
> -A FORWARD -i eth2 -o eth0 -p udp -m udp --dport 53 -m state --state
> NEW,ESTABLISHED -j ACCEPT
> Total time for creating Policy : 0.00182485580444
> ----------------------------------------------
> -A PREROUTING -d 185.20.146.56/32 -i eth2 -p tcp -m tcp --dport 22 -j
> DNAT --to-destination 10.100.1.95:22
> time : 0.000264167785645
> -A PREROUTING -d 185.20.146.56/32 -i eth0 -p tcp -m tcp --dport 22 -j
> DNAT --to-destination 10.100.1.95:22
> time : 0.000508069992065
> -A OUTPUT -d 185.20.146.56/32 -p tcp -m tcp --dport 22 -j DNAT
> --to-destination 10.100.1.95:22
> time : 0.000750064849854
> -j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d
> 10.100.1.95/32 -o eth0 -p tcp -m tcp --dport 22
> time : 0.00102114677429
> -A PREROUTING -d 185.20.146.56/32 -i eth2 -p tcp -m tcp --dport 22 -j
> MARK --set-xmark 0x2/0xffffffff
> time : 0.00138115882874
> -A PREROUTING -d 185.20.146.56/32 -i eth2 -p tcp -m tcp --dport 22 -m
> state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask
> 0xffffffff
> time : 0.00165915489197
> -A FORWARD -i eth2 -o eth0 -p tcp -m tcp --dport 22 -m state --state
> NEW,ESTABLISHED -j ACCEPT
> Total time for creating Policy : 0.00196814537048
> ----------------------------------------------
>
> Location of configure.py:
>
> https://github.com/apache/cloudstack/blob/master/systemvm/patches/debian/config/opt/cloud/bin/configure.py
>
> The modified scripts are attached. Thanks for your feedback.
>
> regards
> Martin
>
>
Re: Issue: CLOUDSTACK-9255 Unable to start VM DomainRouter due to
error in finalizeStart
Posted by Daan Hoogland <da...@gmail.com>.
Boris closed it,
@Boris why?
On Fri, Mar 18, 2016 at 12:14 PM, Remi Bergsma <RB...@schubergphilis.com>
wrote:
> Hi,
>
> This issue has been resolved some time ago but unfortunately the PR hasn’t
> been merged nor tested yet.
>
> https://github.com/apache/cloudstack/pull/1400
>
> This PR makes it like 50-60 times faster, because it first generates all
> iptables commands and then loads them once.
>
> We run this in production for weeks already. Not sure why the PR is
> closed, it simply works.
>
> Regards,
> Remi
>
>
> From: martin kolly <martin.kolly@senselan.ch<mailto:
> martin.kolly@senselan.ch>>
> Reply-To: "dev@cloudstack.apache.org<ma...@cloudstack.apache.org>" <
> dev@cloudstack.apache.org<ma...@cloudstack.apache.org>>
> Date: Friday 18 March 2016 at 11:58
> To: "dev@cloudstack.apache.org<ma...@cloudstack.apache.org>" <
> dev@cloudstack.apache.org<ma...@cloudstack.apache.org>>
> Subject: Issue: CLOUDSTACK-9255 Unable to start VM DomainRouter due to
> error in finalizeStart
>
> Hi All
>
> We are facing the same issue as reported by Milamber (Ticket 9255)
> https://issues.apache.org/jira/browse/CLOUDSTACK-9255. When deploying a
> couple of VMs or Port Forwarding's the re-deployment of the router with
> cleanup fails.
>
> We found that iptables configuration takes a lot of time, this eventually
> leads to a timeout on the management server "Unable to start VM
> DomainRouter due to error in finalizeStart, not retrying"
>
> Environment:
> - Cloudstack 4.8
> - KVM (local storage)
> - hosts/mgr on Ubuntu 14.04
>
> We tested with a simple set of four forwarding rules, here the setup:
>
> root@r-96-VM:~# cat /etc/cloudstack/forwardingrules.json
> {
> "185.20.146.56": [
> {
> "internal_ip": "10.100.1.95",
> "internal_ports": "22:22",
> "protocol": "tcp",
> "public_ip": "185.20.146.56",
> "public_ports": "22:22",
> "type": "forward"
> }
> ],
> "185.20.146.79": [
> {
> "internal_ip": "10.100.1.42",
> "internal_ports": "22:22",
> "protocol": "tcp",
> "public_ip": "185.20.146.79",
> "public_ports": "22:22",
> "type": "forward"
> },
> {
> "internal_ip": "10.100.1.42",
> "internal_ports": "8443:8443",
> "protocol": "tcp",
> "public_ip": "185.20.146.79",
> "public_ports": "8443:8443",
> "type": "forward"
> },
> {
> "internal_ip": "10.100.1.42",
> "internal_ports": "53:53",
> "protocol": "udp",
> "public_ip": "185.20.146.79",
> "public_ports": "53:53",
> "type": "forward"
> }
> ],
> "id": "forwardingrules"
>
> The definition for every port forwarding seems to take at ~1.5 seconds.
>
> python /opt/cloud/bin/configure.py.timed
> /etc/cloudstack/forwardingrules.json
>
> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 22 -j
> DNAT --to-destination 10.100.1.42:22
> time : 0.000965118408203
> -A PREROUTING -d 185.20.146.79/32 -i eth0 -p tcp -m tcp --dport 22 -j
> DNAT --to-destination 10.100.1.42:22
> time : 0.395485162735
> -A OUTPUT -d 185.20.146.79/32 -p tcp -m tcp --dport 22 -j DNAT
> --to-destination 10.100.1.42:22
> time : 0.395533084869
> -j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d
> 10.100.1.42/32 -o eth0 -p tcp -m tcp --dport 22
> time : 1.16180706024
> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 22 -j
> MARK --set-xmark 0x2/0xffffffff
> time : 1.16329216957
> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 22 -m
> state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask
> 0xffffffff
> time : 1.16407108307
> -A FORWARD -i eth2 -o eth0 -p tcp -m tcp --dport 22 -m state --state
> NEW,ESTABLISHED -j ACCEPT
> Total time for creating Policy : 1.53959512711
> ----------------------------------------------
> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 8443 -j
> DNAT --to-destination 10.100.1.42:8443
> time : 0.000781059265137
> -A PREROUTING -d 185.20.146.79/32 -i eth0 -p tcp -m tcp --dport 8443 -j
> DNAT --to-destination 10.100.1.42:8443
> time : 0.378201007843
> -A OUTPUT -d 185.20.146.79/32 -p tcp -m tcp --dport 8443 -j DNAT
> --to-destination 10.100.1.42:8443
> time : 0.37822508812
> -j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d
> 10.100.1.42/32 -o eth0 -p tcp -m tcp --dport 8443
> time : 1.14627504349
> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 8443 -j
> MARK --set-xmark 0x2/0xffffffff
> time : 1.1477329731
> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 8443 -m
> state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask
> 0xffffffff
> time : 1.14850592613
> -A FORWARD -i eth2 -o eth0 -p tcp -m tcp --dport 8443 -m state --state
> NEW,ESTABLISHED -j ACCEPT
> Total time for creating Policy : 1.52321791649
> ----------------------------------------------
> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p udp -m udp --dport 53 -j
> DNAT --to-destination 10.100.1.42:53
> time : 0.000754117965698
> -A PREROUTING -d 185.20.146.79/32 -i eth0 -p udp -m udp --dport 53 -j
> DNAT --to-destination 10.100.1.42:53
> time : 0.383729934692
> -A OUTPUT -d 185.20.146.79/32 -p udp -m udp --dport 53 -j DNAT
> --to-destination 10.100.1.42:53
> time : 0.383754968643
> -j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d
> 10.100.1.42/32 -o eth0 -p udp -m udp --dport 53
> time : 1.14376091957
> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p udp -m udp --dport 53 -j
> MARK --set-xmark 0x2/0xffffffff
> time : 1.14526605606
> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p udp -m udp --dport 53 -m
> state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask
> 0xffffffff
> time : 1.14599299431
> -A FORWARD -i eth2 -o eth0 -p udp -m udp --dport 53 -m state --state
> NEW,ESTABLISHED -j ACCEPT
> Total time for creating Policy : 1.52742600441
> ----------------------------------------------
> -A PREROUTING -d 185.20.146.56/32 -i eth2 -p tcp -m tcp --dport 22 -j
> DNAT --to-destination 10.100.1.95:22
> time : 0.000700950622559
> -A PREROUTING -d 185.20.146.56/32 -i eth0 -p tcp -m tcp --dport 22 -j
> DNAT --to-destination 10.100.1.95:22
> time : 0.382349014282
> -A OUTPUT -d 185.20.146.56/32 -p tcp -m tcp --dport 22 -j DNAT
> --to-destination 10.100.1.95:22
> time : 0.382384061813
> -j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d
> 10.100.1.95/32 -o eth0 -p tcp -m tcp --dport 22
> time : 1.1425909996
> -A PREROUTING -d 185.20.146.56/32 -i eth2 -p tcp -m tcp --dport 22 -j
> MARK --set-xmark 0x2/0xffffffff
> time : 1.14400196075
> -A PREROUTING -d 185.20.146.56/32 -i eth2 -p tcp -m tcp --dport 22 -m
> state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask
> 0xffffffff
> time : 1.14468812943
> -A FORWARD -i eth2 -o eth0 -p tcp -m tcp --dport 22 -m state --state
> NEW,ESTABLISHED -j ACCEPT
> Total time for creating Policy : 1.52619600296
> ----------------------------------------------
>
> Having a closer look at configure.py how the iptables rules are defined.
> We think that it is not efficient to lookup these values for every policy:
>
> def forward_vr(self, rule):
>
> fw1 = "-A PREROUTING -d %s/32 -i %s -p %s -m %s --dport %s -j DNAT
> --to-destination %s:%s" % \
> (
> rule['public_ip'],
> self.getDeviceByIp(rule['public_ip']),
> rule['protocol'],
> rule['protocol'],
> self.portsToString(rule['public_ports'], ':'),
> rule['internal_ip'],
> self.portsToString(rule['internal_ports'], '-')
> )
> fw2 = "-A PREROUTING -d %s/32 -i %s -p %s -m %s --dport %s -j DNAT
> --to-destination %s:%s" % \
> (
> rule['public_ip'],
> self.getDeviceByIp(rule['internal_ip']),
> rule['protocol'],
> rule['protocol'],
> self.portsToString(rule['public_ports'], ':'),
> rule['internal_ip'],
> self.portsToString(rule['internal_ports'], '-')
> .....
>
>
> Defining these values once at the beginning would be much more efficient,
> no ?
>
> def forward_vr(self, rule):
>
> pub_interface = self.getDeviceByIp(rule['public_ip'])
> int_interface = self.getDeviceByIp(rule['internal_ip'])
> pub_ports = self.portsToString(rule['public_ports'], ':')
> int_ports = self.portsToString(rule['internal_ports'], '-')
> int_network = self.getNetworkByIp(rule['internal_ip'])
>
> fw1 = "-A PREROUTING -d %s/32 -i %s -p %s -m %s --dport %s -j DNAT
> --to-destination %s:%s" % \
> (
> rule['public_ip'],
> pub_interface,
> rule['protocol'],
> rule['protocol'],
> pub_ports,
> rule['internal_ip'],
> int_ports
> )
>
> fw2 = "-A PREROUTING -d %s/32 -i %s -p %s -m %s --dport %s -j DNAT
> --to-destination %s:%s" % \
> (
> rule['public_ip'],
> int_interface,
> rule['protocol'],
> rule['protocol'],
> pub_ports,
> rule['internal_ip'],
> int_ports
> )
> .....
>
> If we run the configure.py with these modifications we have the following:
>
> root@r-96-VM:~# python /opt/cloud/bin/configure_modified.py
> /etc/cloudstack/forwardingrules.json
> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 22 -j
> DNAT --to-destination 10.100.1.42:22
> time : 0.000349044799805
> -A PREROUTING -d 185.20.146.79/32 -i eth0 -p tcp -m tcp --dport 22 -j
> DNAT --to-destination 10.100.1.42:22
> time : 0.000686883926392
> -A OUTPUT -d 185.20.146.79/32 -p tcp -m tcp --dport 22 -j DNAT
> --to-destination 10.100.1.42:22
> time : 0.000943899154663
> -j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d
> 10.100.1.42/32 -o eth0 -p tcp -m tcp --dport 22
> time : 0.00131487846375
> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 22 -j
> MARK --set-xmark 0x2/0xffffffff
> time : 0.00161194801331
> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 22 -m
> state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask
> 0xffffffff
> time : 0.00186896324158
> -A FORWARD -i eth2 -o eth0 -p tcp -m tcp --dport 22 -m state --state
> NEW,ESTABLISHED -j ACCEPT
> Total time for creating Policy : 0.00216102600098
> ----------------------------------------------
> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 8443 -j
> DNAT --to-destination 10.100.1.42:8443
> time : 0.000232934951782
> -A PREROUTING -d 185.20.146.79/32 -i eth0 -p tcp -m tcp --dport 8443 -j
> DNAT --to-destination 10.100.1.42:8443
> time : 0.000478029251099
> -A OUTPUT -d 185.20.146.79/32 -p tcp -m tcp --dport 8443 -j DNAT
> --to-destination 10.100.1.42:8443
> time : 0.00071907043457
> -j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d
> 10.100.1.42/32 -o eth0 -p tcp -m tcp --dport 8443
> time : 0.000991106033325
> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 8443 -j
> MARK --set-xmark 0x2/0xffffffff
> time : 0.00136613845825
> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 8443 -m
> state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask
> 0xffffffff
> time : 0.00174498558044
> -A FORWARD -i eth2 -o eth0 -p tcp -m tcp --dport 8443 -m state --state
> NEW,ESTABLISHED -j ACCEPT
> Total time for creating Policy : 0.00219202041626
> ----------------------------------------------
> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p udp -m udp --dport 53 -j
> DNAT --to-destination 10.100.1.42:53
> time : 0.000226974487305
> -A PREROUTING -d 185.20.146.79/32 -i eth0 -p udp -m udp --dport 53 -j
> DNAT --to-destination 10.100.1.42:53
> time : 0.000502824783325
> -A OUTPUT -d 185.20.146.79/32 -p udp -m udp --dport 53 -j DNAT
> --to-destination 10.100.1.42:53
> time : 0.000762939453125
> -j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d
> 10.100.1.42/32 -o eth0 -p udp -m udp --dport 53
> time : 0.00103092193604
> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p udp -m udp --dport 53 -j
> MARK --set-xmark 0x2/0xffffffff
> time : 0.00134587287903
> -A PREROUTING -d 185.20.146.79/32 -i eth2 -p udp -m udp --dport 53 -m
> state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask
> 0xffffffff
> time : 0.00158596038818
> -A FORWARD -i eth2 -o eth0 -p udp -m udp --dport 53 -m state --state
> NEW,ESTABLISHED -j ACCEPT
> Total time for creating Policy : 0.00182485580444
> ----------------------------------------------
> -A PREROUTING -d 185.20.146.56/32 -i eth2 -p tcp -m tcp --dport 22 -j
> DNAT --to-destination 10.100.1.95:22
> time : 0.000264167785645
> -A PREROUTING -d 185.20.146.56/32 -i eth0 -p tcp -m tcp --dport 22 -j
> DNAT --to-destination 10.100.1.95:22
> time : 0.000508069992065
> -A OUTPUT -d 185.20.146.56/32 -p tcp -m tcp --dport 22 -j DNAT
> --to-destination 10.100.1.95:22
> time : 0.000750064849854
> -j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d
> 10.100.1.95/32 -o eth0 -p tcp -m tcp --dport 22
> time : 0.00102114677429
> -A PREROUTING -d 185.20.146.56/32 -i eth2 -p tcp -m tcp --dport 22 -j
> MARK --set-xmark 0x2/0xffffffff
> time : 0.00138115882874
> -A PREROUTING -d 185.20.146.56/32 -i eth2 -p tcp -m tcp --dport 22 -m
> state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask
> 0xffffffff
> time : 0.00165915489197
> -A FORWARD -i eth2 -o eth0 -p tcp -m tcp --dport 22 -m state --state
> NEW,ESTABLISHED -j ACCEPT
> Total time for creating Policy : 0.00196814537048
> ----------------------------------------------
>
> Location of configure.py:
>
> https://github.com/apache/cloudstack/blob/master/systemvm/patches/debian/config/opt/cloud/bin/configure.py
>
> The modified scripts are attached. Thanks for your feedback.
>
> regards
> Martin
>
>
--
Daan
Re: Issue: CLOUDSTACK-9255 Unable to start VM DomainRouter due to
error in finalizeStart
Posted by Remi Bergsma <RB...@schubergphilis.com>.
Hi,
This issue has been resolved some time ago but unfortunately the PR hasn’t been merged nor tested yet.
https://github.com/apache/cloudstack/pull/1400
This PR makes it like 50-60 times faster, because it first generates all iptables commands and then loads them once.
We run this in production for weeks already. Not sure why the PR is closed, it simply works.
Regards,
Remi
From: martin kolly <ma...@senselan.ch>>
Reply-To: "dev@cloudstack.apache.org<ma...@cloudstack.apache.org>" <de...@cloudstack.apache.org>>
Date: Friday 18 March 2016 at 11:58
To: "dev@cloudstack.apache.org<ma...@cloudstack.apache.org>" <de...@cloudstack.apache.org>>
Subject: Issue: CLOUDSTACK-9255 Unable to start VM DomainRouter due to error in finalizeStart
Hi All
We are facing the same issue as reported by Milamber (Ticket 9255) https://issues.apache.org/jira/browse/CLOUDSTACK-9255. When deploying a couple of VMs or Port Forwarding's the re-deployment of the router with cleanup fails.
We found that iptables configuration takes a lot of time, this eventually leads to a timeout on the management server "Unable to start VM DomainRouter due to error in finalizeStart, not retrying"
Environment:
- Cloudstack 4.8
- KVM (local storage)
- hosts/mgr on Ubuntu 14.04
We tested with a simple set of four forwarding rules, here the setup:
root@r-96-VM:~# cat /etc/cloudstack/forwardingrules.json
{
"185.20.146.56": [
{
"internal_ip": "10.100.1.95",
"internal_ports": "22:22",
"protocol": "tcp",
"public_ip": "185.20.146.56",
"public_ports": "22:22",
"type": "forward"
}
],
"185.20.146.79": [
{
"internal_ip": "10.100.1.42",
"internal_ports": "22:22",
"protocol": "tcp",
"public_ip": "185.20.146.79",
"public_ports": "22:22",
"type": "forward"
},
{
"internal_ip": "10.100.1.42",
"internal_ports": "8443:8443",
"protocol": "tcp",
"public_ip": "185.20.146.79",
"public_ports": "8443:8443",
"type": "forward"
},
{
"internal_ip": "10.100.1.42",
"internal_ports": "53:53",
"protocol": "udp",
"public_ip": "185.20.146.79",
"public_ports": "53:53",
"type": "forward"
}
],
"id": "forwardingrules"
The definition for every port forwarding seems to take at ~1.5 seconds.
python /opt/cloud/bin/configure.py.timed /etc/cloudstack/forwardingrules.json
-A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 22 -j DNAT --to-destination 10.100.1.42:22
time : 0.000965118408203
-A PREROUTING -d 185.20.146.79/32 -i eth0 -p tcp -m tcp --dport 22 -j DNAT --to-destination 10.100.1.42:22
time : 0.395485162735
-A OUTPUT -d 185.20.146.79/32 -p tcp -m tcp --dport 22 -j DNAT --to-destination 10.100.1.42:22
time : 0.395533084869
-j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d 10.100.1.42/32 -o eth0 -p tcp -m tcp --dport 22
time : 1.16180706024
-A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 22 -j MARK --set-xmark 0x2/0xffffffff
time : 1.16329216957
-A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 22 -m state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask 0xffffffff
time : 1.16407108307
-A FORWARD -i eth2 -o eth0 -p tcp -m tcp --dport 22 -m state --state NEW,ESTABLISHED -j ACCEPT
Total time for creating Policy : 1.53959512711
----------------------------------------------
-A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 8443 -j DNAT --to-destination 10.100.1.42:8443
time : 0.000781059265137
-A PREROUTING -d 185.20.146.79/32 -i eth0 -p tcp -m tcp --dport 8443 -j DNAT --to-destination 10.100.1.42:8443
time : 0.378201007843
-A OUTPUT -d 185.20.146.79/32 -p tcp -m tcp --dport 8443 -j DNAT --to-destination 10.100.1.42:8443
time : 0.37822508812
-j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d 10.100.1.42/32 -o eth0 -p tcp -m tcp --dport 8443
time : 1.14627504349
-A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 8443 -j MARK --set-xmark 0x2/0xffffffff
time : 1.1477329731
-A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 8443 -m state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask 0xffffffff
time : 1.14850592613
-A FORWARD -i eth2 -o eth0 -p tcp -m tcp --dport 8443 -m state --state NEW,ESTABLISHED -j ACCEPT
Total time for creating Policy : 1.52321791649
----------------------------------------------
-A PREROUTING -d 185.20.146.79/32 -i eth2 -p udp -m udp --dport 53 -j DNAT --to-destination 10.100.1.42:53
time : 0.000754117965698
-A PREROUTING -d 185.20.146.79/32 -i eth0 -p udp -m udp --dport 53 -j DNAT --to-destination 10.100.1.42:53
time : 0.383729934692
-A OUTPUT -d 185.20.146.79/32 -p udp -m udp --dport 53 -j DNAT --to-destination 10.100.1.42:53
time : 0.383754968643
-j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d 10.100.1.42/32 -o eth0 -p udp -m udp --dport 53
time : 1.14376091957
-A PREROUTING -d 185.20.146.79/32 -i eth2 -p udp -m udp --dport 53 -j MARK --set-xmark 0x2/0xffffffff
time : 1.14526605606
-A PREROUTING -d 185.20.146.79/32 -i eth2 -p udp -m udp --dport 53 -m state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask 0xffffffff
time : 1.14599299431
-A FORWARD -i eth2 -o eth0 -p udp -m udp --dport 53 -m state --state NEW,ESTABLISHED -j ACCEPT
Total time for creating Policy : 1.52742600441
----------------------------------------------
-A PREROUTING -d 185.20.146.56/32 -i eth2 -p tcp -m tcp --dport 22 -j DNAT --to-destination 10.100.1.95:22
time : 0.000700950622559
-A PREROUTING -d 185.20.146.56/32 -i eth0 -p tcp -m tcp --dport 22 -j DNAT --to-destination 10.100.1.95:22
time : 0.382349014282
-A OUTPUT -d 185.20.146.56/32 -p tcp -m tcp --dport 22 -j DNAT --to-destination 10.100.1.95:22
time : 0.382384061813
-j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d 10.100.1.95/32 -o eth0 -p tcp -m tcp --dport 22
time : 1.1425909996
-A PREROUTING -d 185.20.146.56/32 -i eth2 -p tcp -m tcp --dport 22 -j MARK --set-xmark 0x2/0xffffffff
time : 1.14400196075
-A PREROUTING -d 185.20.146.56/32 -i eth2 -p tcp -m tcp --dport 22 -m state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask 0xffffffff
time : 1.14468812943
-A FORWARD -i eth2 -o eth0 -p tcp -m tcp --dport 22 -m state --state NEW,ESTABLISHED -j ACCEPT
Total time for creating Policy : 1.52619600296
----------------------------------------------
Having a closer look at configure.py how the iptables rules are defined. We think that it is not efficient to lookup these values for every policy:
def forward_vr(self, rule):
fw1 = "-A PREROUTING -d %s/32 -i %s -p %s -m %s --dport %s -j DNAT --to-destination %s:%s" % \
(
rule['public_ip'],
self.getDeviceByIp(rule['public_ip']),
rule['protocol'],
rule['protocol'],
self.portsToString(rule['public_ports'], ':'),
rule['internal_ip'],
self.portsToString(rule['internal_ports'], '-')
)
fw2 = "-A PREROUTING -d %s/32 -i %s -p %s -m %s --dport %s -j DNAT --to-destination %s:%s" % \
(
rule['public_ip'],
self.getDeviceByIp(rule['internal_ip']),
rule['protocol'],
rule['protocol'],
self.portsToString(rule['public_ports'], ':'),
rule['internal_ip'],
self.portsToString(rule['internal_ports'], '-')
.....
Defining these values once at the beginning would be much more efficient, no ?
def forward_vr(self, rule):
pub_interface = self.getDeviceByIp(rule['public_ip'])
int_interface = self.getDeviceByIp(rule['internal_ip'])
pub_ports = self.portsToString(rule['public_ports'], ':')
int_ports = self.portsToString(rule['internal_ports'], '-')
int_network = self.getNetworkByIp(rule['internal_ip'])
fw1 = "-A PREROUTING -d %s/32 -i %s -p %s -m %s --dport %s -j DNAT --to-destination %s:%s" % \
(
rule['public_ip'],
pub_interface,
rule['protocol'],
rule['protocol'],
pub_ports,
rule['internal_ip'],
int_ports
)
fw2 = "-A PREROUTING -d %s/32 -i %s -p %s -m %s --dport %s -j DNAT --to-destination %s:%s" % \
(
rule['public_ip'],
int_interface,
rule['protocol'],
rule['protocol'],
pub_ports,
rule['internal_ip'],
int_ports
)
.....
If we run the configure.py with these modifications we have the following:
root@r-96-VM:~# python /opt/cloud/bin/configure_modified.py /etc/cloudstack/forwardingrules.json
-A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 22 -j DNAT --to-destination 10.100.1.42:22
time : 0.000349044799805
-A PREROUTING -d 185.20.146.79/32 -i eth0 -p tcp -m tcp --dport 22 -j DNAT --to-destination 10.100.1.42:22
time : 0.000686883926392
-A OUTPUT -d 185.20.146.79/32 -p tcp -m tcp --dport 22 -j DNAT --to-destination 10.100.1.42:22
time : 0.000943899154663
-j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d 10.100.1.42/32 -o eth0 -p tcp -m tcp --dport 22
time : 0.00131487846375
-A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 22 -j MARK --set-xmark 0x2/0xffffffff
time : 0.00161194801331
-A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 22 -m state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask 0xffffffff
time : 0.00186896324158
-A FORWARD -i eth2 -o eth0 -p tcp -m tcp --dport 22 -m state --state NEW,ESTABLISHED -j ACCEPT
Total time for creating Policy : 0.00216102600098
----------------------------------------------
-A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 8443 -j DNAT --to-destination 10.100.1.42:8443
time : 0.000232934951782
-A PREROUTING -d 185.20.146.79/32 -i eth0 -p tcp -m tcp --dport 8443 -j DNAT --to-destination 10.100.1.42:8443
time : 0.000478029251099
-A OUTPUT -d 185.20.146.79/32 -p tcp -m tcp --dport 8443 -j DNAT --to-destination 10.100.1.42:8443
time : 0.00071907043457
-j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d 10.100.1.42/32 -o eth0 -p tcp -m tcp --dport 8443
time : 0.000991106033325
-A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 8443 -j MARK --set-xmark 0x2/0xffffffff
time : 0.00136613845825
-A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 8443 -m state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask 0xffffffff
time : 0.00174498558044
-A FORWARD -i eth2 -o eth0 -p tcp -m tcp --dport 8443 -m state --state NEW,ESTABLISHED -j ACCEPT
Total time for creating Policy : 0.00219202041626
----------------------------------------------
-A PREROUTING -d 185.20.146.79/32 -i eth2 -p udp -m udp --dport 53 -j DNAT --to-destination 10.100.1.42:53
time : 0.000226974487305
-A PREROUTING -d 185.20.146.79/32 -i eth0 -p udp -m udp --dport 53 -j DNAT --to-destination 10.100.1.42:53
time : 0.000502824783325
-A OUTPUT -d 185.20.146.79/32 -p udp -m udp --dport 53 -j DNAT --to-destination 10.100.1.42:53
time : 0.000762939453125
-j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d 10.100.1.42/32 -o eth0 -p udp -m udp --dport 53
time : 0.00103092193604
-A PREROUTING -d 185.20.146.79/32 -i eth2 -p udp -m udp --dport 53 -j MARK --set-xmark 0x2/0xffffffff
time : 0.00134587287903
-A PREROUTING -d 185.20.146.79/32 -i eth2 -p udp -m udp --dport 53 -m state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask 0xffffffff
time : 0.00158596038818
-A FORWARD -i eth2 -o eth0 -p udp -m udp --dport 53 -m state --state NEW,ESTABLISHED -j ACCEPT
Total time for creating Policy : 0.00182485580444
----------------------------------------------
-A PREROUTING -d 185.20.146.56/32 -i eth2 -p tcp -m tcp --dport 22 -j DNAT --to-destination 10.100.1.95:22
time : 0.000264167785645
-A PREROUTING -d 185.20.146.56/32 -i eth0 -p tcp -m tcp --dport 22 -j DNAT --to-destination 10.100.1.95:22
time : 0.000508069992065
-A OUTPUT -d 185.20.146.56/32 -p tcp -m tcp --dport 22 -j DNAT --to-destination 10.100.1.95:22
time : 0.000750064849854
-j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d 10.100.1.95/32 -o eth0 -p tcp -m tcp --dport 22
time : 0.00102114677429
-A PREROUTING -d 185.20.146.56/32 -i eth2 -p tcp -m tcp --dport 22 -j MARK --set-xmark 0x2/0xffffffff
time : 0.00138115882874
-A PREROUTING -d 185.20.146.56/32 -i eth2 -p tcp -m tcp --dport 22 -m state --state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask 0xffffffff
time : 0.00165915489197
-A FORWARD -i eth2 -o eth0 -p tcp -m tcp --dport 22 -m state --state NEW,ESTABLISHED -j ACCEPT
Total time for creating Policy : 0.00196814537048
----------------------------------------------
Location of configure.py:
https://github.com/apache/cloudstack/blob/master/systemvm/patches/debian/config/opt/cloud/bin/configure.py
The modified scripts are attached. Thanks for your feedback.
regards
Martin