You are viewing a plain text version of this content. The canonical link for it is here.
Posted to proton@qpid.apache.org by Michael Goulish <mg...@redhat.com> on 2013/04/09 17:13:00 UTC
messenger routing suggestion
The idea I put forward for a change to pn_messenger_route()
may have seemed not very well motivated.
Here is a more complete example, and a more complete
suggestion.
Example
===================================================
All of these nodes are Messenger nodes.
1. Sender
You have a node that's a sender.
It is sending to only abstract addresses.
It uses its routing table to map all its abstract
addresses to the same receiver -- because that receiver
is where the system centralizes knowledge about
changing network conditions.
Sender's routing table
----------------------------------
COLOSSUS --> 1.2.3.4:6666
GUARDIAN --> 1.2.3.4:6666
HAL9000 --> 1.2.3.4:6666
SKYNET --> 1.2.3.4:6666
2. Router
You have a "router" node that is listening on 1.2.3.4:6666 .
It receives and then forwards messages.
It can do this because the messages it receives from
Sender still have their untranslated addresses.
It has this routing table
-----------------------------------
COLOSSUS --> 5.6.7.8:1234
COLOSSUS --> 5.6.7.9:1234
GUARDIAN --> 5.6.7.23:3456
HAL9000 --> null
SKYNET --> 5.6.7.99:6969
3. Adapting to changing conditions
The 'router' node can change its address translation
table based on messages that it receives from other parts
of the network. For example, maybe it does load-balancing
this way.
But in general -- it needs to change its translation table
at run time because conditions in the network are changing.
It is the node that encapsulates knowledge about those changes
so that the rest of our nodes do not need to worry about it.
They just send to "COLOSSUS" or whatever.
This implies that we should be able to change routing
dynamically.
4. Fanout
Note that there are two translations for COLOSSUS.
We send to both of them. ( This is a change. )
This is how we implement fanout with the address translation.
5. Load and Store
Since the translation table can change due to changing
network conditions, the Router node should be able to
store its table to a file, and load from that file.
The information that it has "learned" during operation
is not lost. Or it can use this facility to fork off
another copy of itself.
6. API changes
It seems to me that the address translation functionality
is potentially very powerful, with a few teensy changes.
Here they are:
/*=========================================================
At send time, the messenger examines its translation
table, and sends a copy of the message to each matching
address. ( this is a change )
The address stored in the message is not changed.
( this is true now. )
=========================================================*/
/*-------------------------------------------
Append the given translation to the list.
-------------------------------------------*/
pn_messenger_route_add ( pn_messenger_t *messenger,
const char *pattern,
const char *address );
/*-----------------------------------------------------
If the given pattern already exists in the list,
replace its first occurrence with this translation.
Otherwise add this translation to the list.
-----------------------------------------------------*/
pn_messenger_route_replace ( pn_messenger_t *messenger,
const char *pattern,
const char *address );
/*-----------------------------------------------------
Delete the given translation from the list.
( Else, NOOP. )
-----------------------------------------------------*/
pn_messenger_route_delete ( pn_messenger_t *messenger,
const char *pattern,
const char *address );
/*-----------------------------------------------------
Delete from the list all translations with this
pattern.
-----------------------------------------------------*/
pn_messenger_route_delete_pattern ( pn_messenger_t *messenger,
const char *pattern );
/*-----------------------------------------------------
Clear the translation table.
-----------------------------------------------------*/
pn_messenger_route_clear_table ( pn_messenger_t *messenger );
/*-----------------------------------------------------
Load from the given fp
-----------------------------------------------------*/
pn_messenger_route_load_table ( pn_messenger_t *messenger,
FILE * fp );
/*-----------------------------------------------------
Store to the given fp
-----------------------------------------------------*/
pn_messenger_route_load_table ( pn_messenger_t *messenger,
FILE * fp );
Re: messenger routing suggestion
Posted by Rafael Schloming <rh...@alum.mit.edu>.
On Tue, Apr 9, 2013 at 11:13 AM, Michael Goulish <mg...@redhat.com>wrote:
> The idea I put forward for a change to pn_messenger_route()
> may have seemed not very well motivated.
>
I don't think it's unmotivated, but there are a lot of different options in
this area and I think we need some careful exploration here. I'm not sure I
would recommend a deployment exactly like you describe below for a number
of reasons, but regardless I still buy the dynamic routing changes as a
useful scenario. I'll comment more inline, but I'm going to reference
detail in my response to your previous email which I unfortunately posted
before reading this one, so please have a look at that one first. ;-)
>
> Here is a more complete example, and a more complete
> suggestion.
>
>
> Example
> ===================================================
>
> All of these nodes are Messenger nodes.
>
>
> 1. Sender
> You have a node that's a sender.
> It is sending to only abstract addresses.
> It uses its routing table to map all its abstract
> addresses to the same receiver -- because that receiver
> is where the system centralizes knowledge about
> changing network conditions.
>
> Sender's routing table
> ----------------------------------
>
> COLOSSUS --> 1.2.3.4:6666
> GUARDIAN --> 1.2.3.4:6666
> HAL9000 --> 1.2.3.4:6666
> SKYNET --> 1.2.3.4:6666
>
One comment about this particular deployment is that you have a single
point of failure between your Sender and your Router. If you lose
connectivity or your Router goes down, then your whole network is down.
Borrowing the notation from my previous post, this is a good example of
where you might want to use a "failover" rule, e.g. "* --> 1.2.3.4:6666,
4.3.2.1:6666". This would provide you redundant routes, and is a good
illustration of why you'd want to be able to specify that behaviour without
recoding your application.
>
>
>
> 2. Router
> You have a "router" node that is listening on 1.2.3.4:6666 .
> It receives and then forwards messages.
> It can do this because the messages it receives from
> Sender still have their untranslated addresses.
>
> It has this routing table
> -----------------------------------
> COLOSSUS --> 5.6.7.8:1234
> COLOSSUS --> 5.6.7.9:1234
> GUARDIAN --> 5.6.7.23:3456
> HAL9000 --> null
> SKYNET --> 5.6.7.99:6969
>
>
>
> 3. Adapting to changing conditions
> The 'router' node can change its address translation
> table based on messages that it receives from other parts
> of the network. For example, maybe it does load-balancing
> this way.
>
> But in general -- it needs to change its translation table
> at run time because conditions in the network are changing.
> It is the node that encapsulates knowledge about those changes
> so that the rest of our nodes do not need to worry about it.
> They just send to "COLOSSUS" or whatever.
>
> This implies that we should be able to change routing
> dynamically.
>
This part I buy, although I think your particular example of load balancing
is actually a cross cutting behaviour that we want to be able to handle
throughout the network. I also think your scenario would get more
compelling if the Router were actually doing some kind of application level
message processing, e.g. maybe transforming the messages or something. I
believe we (qpid) expect to be building self contained components that you
can drop in that will perform this kind of pure routing function for you
and these components will most certainly have dynamically updated routing
tables, however they won't necessarily be built on top of messenger
(although that possibility is admittedly intriguing).
>
> 4. Fanout
> Note that there are two translations for COLOSSUS.
> We send to both of them. ( This is a change. )
> This is how we implement fanout with the address translation.
>
On board here, however as I mentioned in my previous email fanout is only
one option if you have multiple next hops. I don't know if it makes sense
to think about multiple rules with a given pattern or simply think about
one rule per pattern where the right hand side of the rule can reference
multiple next hops.
>
>
>
> 5. Load and Store
> Since the translation table can change due to changing
> network conditions, the Router node should be able to
> store its table to a file, and load from that file.
> The information that it has "learned" during operation
> is not lost. Or it can use this facility to fork off
> another copy of itself.
>
For portability reasons I don't think messenger should do direct file
access for anything except pure convenience, so I would translate this into
an API that let you query the existing set of routes and access enough
information to externalize them. I think this would be more flexible
anyways as you might not want to shove a big routing table into a flat
file, e.g. you might want to stick it into a sqlite database or something
like that.
>
>
> 6. API changes
> It seems to me that the address translation functionality
> is potentially very powerful, with a few teensy changes.
>
> Here they are:
>
>
>
> /*=========================================================
> At send time, the messenger examines its translation
> table, and sends a copy of the message to each matching
> address. ( this is a change )
>
> The address stored in the message is not changed.
> ( this is true now. )
> =========================================================*/
>
> /*-------------------------------------------
> Append the given translation to the list.
> -------------------------------------------*/
> pn_messenger_route_add ( pn_messenger_t *messenger,
> const char *pattern,
> const char *address );
>
>
>
> /*-----------------------------------------------------
> If the given pattern already exists in the list,
> replace its first occurrence with this translation.
> Otherwise add this translation to the list.
> -----------------------------------------------------*/
> pn_messenger_route_replace ( pn_messenger_t *messenger,
> const char *pattern,
> const char *address );
>
>
> /*-----------------------------------------------------
> Delete the given translation from the list.
> ( Else, NOOP. )
> -----------------------------------------------------*/
> pn_messenger_route_delete ( pn_messenger_t *messenger,
> const char *pattern,
> const char *address );
>
>
> /*-----------------------------------------------------
> Delete from the list all translations with this
> pattern.
> -----------------------------------------------------*/
> pn_messenger_route_delete_pattern ( pn_messenger_t *messenger,
> const char *pattern );
>
>
> /*-----------------------------------------------------
> Clear the translation table.
> -----------------------------------------------------*/
> pn_messenger_route_clear_table ( pn_messenger_t *messenger );
>
>
>
> /*-----------------------------------------------------
> Load from the given fp
> -----------------------------------------------------*/
> pn_messenger_route_load_table ( pn_messenger_t *messenger,
> FILE * fp );
>
>
>
> /*-----------------------------------------------------
> Store to the given fp
> -----------------------------------------------------*/
> pn_messenger_route_load_table ( pn_messenger_t *messenger,
> FILE * fp );
>
As I mentioned in my prior email I think this area might benefit from an
explicit notion of priority. I'd also suggest exposing a public pn_route_t
type and making pn_messenger_route(messenger, priority, pattern, rule) into
a constructor. You could then hang the appropriate parts of the above API
off of the pn_route_t type, e.g. have blah_route_delete -> pn_route_free,
and pn_route_update(route, priority, pattern, rule), accessors for
pn_route_pattern, pn_route_priority, pn_route_rule, have a
pn_route_next(...) etc. I think that would separate out the routing portion
from the rest of the messenger API and therefore be a bit more
comprehensible.
--Rafael