You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Rohit Yadav <ro...@citrix.com> on 2012/08/16 11:41:37 UTC
Ideas for virtual Redundant Router
Hi,
At present the redundancy of the virtual redundant routers, VRRP [3], operates outside the scope of CloudStack [2]. And any event that happens outside of CS, like rebooting by ssh-ing into the router vms or a crash/start, is not know by the management server (Sheng's comment[1]).
Therefore, these bugs are resolved by design :
http://bugs.cloudstack.org/browse/CS-15907
http://bugs.cloudstack.org/browse/CS-15942
Request for comments to handle events outside of CS's scope for redundant routers: [2]
1. Async Callback:
With bug http://bugs.cloudstack.org/browse/CS-15970 fixed, the management server knows when router's redundant changed and CS re-applies iptables rules when state changes from UNKNOWN (possible crash/reboot) to MASTER/BACKUP.
2. Pull mechanism:
On reboot, a script/daemon on the router pulls any updates or iptables rules, but it won't work if management server is down.
3, Push mechanism:
CS periodically checks/resets/updates iptables rules etc. Cons: it's a bad design.
4. Modifying VRRP [3] [4] such that when one of the routers goes away, the other one will be responsible to re-apply iptables rules. Cons: when both/all go down.
Best Regards,
Rohit
Refs:
[1] http://bugs.cloudstack.org/browse/CS-15907?focusedCommentId=139006&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-139006
[2] http://bugs.cloudstack.org/browse/CS-15907?focusedCommentId=138910&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-138910
[3] http://docs.cloudstack.org/CloudStack_Documentation/Design_Documents/Redundant_Virtual_Router
[4] http://tools.ietf.org/html/rfc3768
Re: Ideas for virtual Redundant Router
Posted by Rohit Yadav <ro...@citrix.com>.
Hi,
I've opened request based on our discussion and comments by Chiradeep: http://bugs.cloudstack.org/browse/CS-16131
And in a slightly related topic on VRR; VRR in basic zone: http://bugs.cloudstack.org/browse/CS-16132
Please post your comments/suggestion/design/behaviour. Thanks.
Regards,
Rohit
On 21-Aug-2012, at 10:23 PM, Chiradeep Vittal <Ch...@citrix.com> wrote:
> Yes, conntrackd, vrrp are being used. They don't handle the actual
> provisioning of the iptables rules though.
> There is also non-connection tracking-related configuration (state) that
> needs to be beamed down to a freshly started router.
> That happens if the router is started by CloudStack (via the API or HA),
> but there are other ways (log in and type reboot) that the configuration
> state on the VR gets out of sync with the CloudStack database.
>
> On 8/21/12 8:56 AM, "Kelceydamage@bbits" <ke...@bbits.ca> wrote:
>
>> We are already using an up tables solution I thought: contrackt
>>
>> Sent from my iPhone
>>
>> On Aug 21, 2012, at 2:05 AM, Matthew Patton <mp...@inforelay.com> wrote:
>>
>>> Please let's not reinvent the wheel. See pfsense, vrrp/carp, and pfsync.
>>>
>>> A redundant iptables solution doesn't spring to mind but it already
>>> exists no doubt.
>
Re: Ideas for virtual Redundant Router
Posted by Chiradeep Vittal <Ch...@citrix.com>.
Yes, conntrackd, vrrp are being used. They don't handle the actual
provisioning of the iptables rules though.
There is also non-connection tracking-related configuration (state) that
needs to be beamed down to a freshly started router.
That happens if the router is started by CloudStack (via the API or HA),
but there are other ways (log in and type reboot) that the configuration
state on the VR gets out of sync with the CloudStack database.
On 8/21/12 8:56 AM, "Kelceydamage@bbits" <ke...@bbits.ca> wrote:
>We are already using an up tables solution I thought: contrackt
>
>Sent from my iPhone
>
>On Aug 21, 2012, at 2:05 AM, Matthew Patton <mp...@inforelay.com> wrote:
>
>> Please let's not reinvent the wheel. See pfsense, vrrp/carp, and pfsync.
>>
>> A redundant iptables solution doesn't spring to mind but it already
>>exists no doubt.
Re: Ideas for virtual Redundant Router
Posted by "Kelceydamage@bbits" <ke...@bbits.ca>.
We are already using an up tables solution I thought: contrackt
Sent from my iPhone
On Aug 21, 2012, at 2:05 AM, Matthew Patton <mp...@inforelay.com> wrote:
> Please let's not reinvent the wheel. See pfsense, vrrp/carp, and pfsync.
>
> A redundant iptables solution doesn't spring to mind but it already exists no doubt.
Re: Ideas for virtual Redundant Router
Posted by Matthew Patton <mp...@inforelay.com>.
Please let's not reinvent the wheel. See pfsense, vrrp/carp, and pfsync.
A redundant iptables solution doesn't spring to mind but it already exists no doubt.
Re: Ideas for virtual Redundant Router
Posted by Rohit Yadav <ro...@citrix.com>.
Hi Chiradeep,
On 21-Aug-2012, at 2:50 AM, Chiradeep Vittal <Ch...@citrix.com> wrote:
> On 8/16/12 2:41 AM, "Rohit Yadav" <ro...@citrix.com> wrote:
>>
>>
>> 1. Async Callback:
>> With bug http://bugs.cloudstack.org/browse/CS-15970 fixed, the
>> management server knows when router's redundant changed and CS re-applies
>> iptables rules when state changes from UNKNOWN (possible crash/reboot) to
>> MASTER/BACKUP.
>
> This may fix the issue at hand, but out-of-band operations on the VR is a
> generic problem.
These were related to the bug http://bugs.cloudstack.org/browse/CS-15907 which is resolved by design now, but I want to fix this in later versions. I agree with your comments on push mechanisms, see below.
>
>>
>> 2. Pull mechanism:
>> On reboot, a script/daemon on the router pulls any updates or iptables
>> rules, but it won't work if management server is down.
>
> Also doesn't account for several other failure scenarios.
>
>>
>> 3, Push mechanism:
>> CS periodically checks/resets/updates iptables rules etc. Cons: it's a
>> bad design.
>
> Not sure why you think this is a bad design.
I prefer event driven programming, but you're right it should work well.
> Every update to the virtual
> router gets a transaction id. You can design a background task to push
> updates to the VR whenever the txid is different from the database. This
> is similar to the design of the SecurityGroups feature. It isn't unlike a
> DB master-slave sync or in some respects to eventual consistency. You
> could even checksum the configuration to detect other inconsistencies.
We already have a background thread that periodically checks for redundant router status and gets values (Master, Backup or Unknown) every 30s (which may be reduced to a lower number for frequent polling, by admin).
We use state changes to push and reapply iptables rules. This becomes an event driven polling/push mechanism which can use txids/checksum as you suggest. Will work on it soon.
>>
>> 4. Modifying VRRP [3] [4] such that when one of the routers goes away,
>> the other one will be responsible to re-apply iptables rules. Cons: when
>> both/all go down.
>
> Modifying GPL software is a non-starter in most cases.
Thanks,
Rohit
Re: Ideas for virtual Redundant Router
Posted by Chiradeep Vittal <Ch...@citrix.com>.
On 8/16/12 2:41 AM, "Rohit Yadav" <ro...@citrix.com> wrote:
>Hi,
>
>At present the redundancy of the virtual redundant routers, VRRP [3],
>operates outside the scope of CloudStack [2]. And any event that happens
>outside of CS, like rebooting by ssh-ing into the router vms or a
>crash/start, is not know by the management server (Sheng's comment[1]).
>
>Therefore, these bugs are resolved by design :
>http://bugs.cloudstack.org/browse/CS-15907
>http://bugs.cloudstack.org/browse/CS-15942
>
>Request for comments to handle events outside of CS's scope for redundant
>routers: [2]
>
>1. Async Callback:
> With bug http://bugs.cloudstack.org/browse/CS-15970 fixed, the
>management server knows when router's redundant changed and CS re-applies
>iptables rules when state changes from UNKNOWN (possible crash/reboot) to
>MASTER/BACKUP.
This may fix the issue at hand, but out-of-band operations on the VR is a
generic problem.
>
>2. Pull mechanism:
> On reboot, a script/daemon on the router pulls any updates or iptables
>rules, but it won't work if management server is down.
Also doesn't account for several other failure scenarios.
>
>3, Push mechanism:
> CS periodically checks/resets/updates iptables rules etc. Cons: it's a
>bad design.
Not sure why you think this is a bad design. Every update to the virtual
router gets a transaction id. You can design a background task to push
updates to the VR whenever the txid is different from the database. This
is similar to the design of the SecurityGroups feature. It isn't unlike a
DB master-slave sync or in some respects to eventual consistency. You
could even checksum the configuration to detect other inconsistencies.
>
>4. Modifying VRRP [3] [4] such that when one of the routers goes away,
>the other one will be responsible to re-apply iptables rules. Cons: when
>both/all go down.
Modifying GPL software is a non-starter in most cases.