You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tomee.apache.org by David Blevins <da...@visi.com> on 2008/08/30 01:38:05 UTC
Client/Server Multicast Discovery
Been looking into how to do service discovery between client/server
and server/server.
We've got some code in the client to do failover on a server list but
by default it really isn't wired up. It'd be nice to get something in
so even if we don't support full clustered replication, at the very
least we could support a bunch of servers that "work together" in a
stateless fashion.
Looking at adding a new "multicast" server service that just
advertises the URIs of the other networks services available in the
system. We'd probably want it off by default (maybe), but then a
client could just sort of boot without being pointed to a specific
server address and theoretically find a server to talk to.
-David
Re: Client/Server Multicast Discovery and Failover
Posted by David Blevins <da...@visi.com>.
Created a doc for this functionality finally and worked in the new
"Multipoint" (i.e. TCP based discovery) stuff I've been working on.
If people could review the doc and give some feedback, that would be
great!
https://cwiki.apache.org/confluence/display/OPENEJBx30/Failover
-David
Re: Client/Server Multicast Discovery and Failover
Posted by Dain Sundstrom <da...@iq80.com>.
On Sep 12, 2008, at 12:45 PM, David Blevins wrote:
> I originally had the version be a simple "increment by one"
> strategy, but eventually went with the value of
> System.currentTimeMillis(). It's possible more than one server is
> reachable via the ServerMetaData (i.e. multicast://) and each server
> has it's own list and version number. Secondly, if a server is
> restarted, the version number will go back to zero and the client
> could be stuck thinking it has a more current list than the server.
Time sometimes moves backwards on servers with connected to a time
server. How about something slightly more unique like a 16 bit rand +
the most significant 48 bits of the system time? 48 bits of
milliseconds is like 9000 years.
> When a server shuts down, more connections are refused, existing
> connections not in mid-request are closed, any remaining connections
> are closed immediately after completion of the request in progress
> and clients can failover gracefully to the next server in the list.
> If a server crashes requests are retried on the next server in the
> list. This failover pattern is followed until there are no more
> servers in the list at which point the client attempts a final
> multicast search (if it was created with a multicast PROVIDER_URL)
> before abandoning the request and throwing an exception to the
> caller. Currently, the failover is ordered but could very easily be
> made random. The multicast discovery aspect of the client adds a
> nice randomness to the selection of the first server that is perhaps
> somewhat "just". Theoretically, servers that are under more load
> will send out less heart beats than servers with no load. This may
> not happen as theory dictates, but certainly as we get more ejb
> statistic data wired into the server functionality we can pursue
> deliberate heartbeat throttling techniques that might make that
> theory really sing in practice.
Very cool.
-dain
Re: Client/Server Multicast Discovery and Failover
Posted by David Blevins <da...@visi.com>.
Think I got all this functionality buttoned up. Here's a high level
view of it in its current form. As usual nothing is in stone and any
and all things can be changed based on group feedback.
DISCOVERY
What we have going on from a tech perspective is each server sends and
receives a multicast heartbeat. Each multicast packet contains a
single URI that advertises a service, its group, and its location.
Say for example "cluster1:ejb:ejbd://thehost:4201". We can definitely
explore the SLP format as Alan suggests.
There are other advantages of the simple, unchanging, URI style. The
URI is essentially stateless as there is no "i'm alive" URI or an "i'm
dead" URI, there is simply a URI for each service a server offers and
its presence on the network indicates its availability and its absence
indicates the service is no longer available. In this way the issues
with UDP being unordered and unreliable melt away as state is no
longer a concern and packet sizes are always small. Complicated
libraries that ride atop UDP and attempt to offer reliability
(retransmission) and ordering on UDP can be avoided. UDP/Multicast is
only used for discovery and from there on out critical information is
transmitted over TCP/IP which is obviously going to do a better job at
ensuring reliability and ordering.
On the client side of things, a special "multicast://" URL can be used
in the InitialContext properties to signify that multicast should be
used to seed the connection process. Such as:
Properties properties = new Properties();
properties.setProperty(Context.INITIAL_CONTEXT_FACTORY,
"org.apache.openejb.client.RemoteInitialContextFactory");
properties.setProperty(Context.PROVIDER_URL, "multicast://
239.255.2.3:6142");
InitialContext remoteContext = new InitialContext(properties);
The URL has optional query parameters such as "schemes" and "group"
and "timeout" which allow you to zero in on a particular type of
service of a particular cluster group as well as set how long you are
willing to wait in the discovery process till finally giving up. The
first matching service that it sees "flowing" around on the UDP stream
is the one it picks and sticks to for that and subsequent requests,
ensuring UDP is only used when there are no other servers to talk to.
FAILOVER
On each request the server, the client will send the version number
associated with the list of servers in the cluster it is aware of.
Initially this version will be zero and the list will be empty. Only
when the server sees the client has an old list will the server send
the updated list. This is an important distinction as the list
(ClusterMetaData) is not transmitted back and forth on every request,
only on change. If the membership of the cluster is stable there is
essentially no clustering overhead to the protocol -- 8 byte overhead
to each request and 1 byte on each response -- so you will *not* see
an exponential slowdown in response times the more members are added
to the cluster. This new list takes affect for all proxies that share
the same ServerMetaData data. Internally we key the ClusterMetaData
by ServerMetaData. I originally had the version be a simple
"increment by one" strategy, but eventually went with the value of
System.currentTimeMillis(). It's possible more than one server is
reachable via the ServerMetaData (i.e. multicast://) and each server
has it's own list and version number. Secondly, if a server is
restarted, the version number will go back to zero and the client
could be stuck thinking it has a more current list than the server.
When a server shuts down, more connections are refused, existing
connections not in mid-request are closed, any remaining connections
are closed immediately after completion of the request in progress and
clients can failover gracefully to the next server in the list. If a
server crashes requests are retried on the next server in the list.
This failover pattern is followed until there are no more servers in
the list at which point the client attempts a final multicast search
(if it was created with a multicast PROVIDER_URL) before abandoning
the request and throwing an exception to the caller. Currently, the
failover is ordered but could very easily be made random. The
multicast discovery aspect of the client adds a nice randomness to the
selection of the first server that is perhaps somewhat "just".
Theoretically, servers that are under more load will send out less
heart beats than servers with no load. This may not happen as theory
dictates, but certainly as we get more ejb statistic data wired into
the server functionality we can pursue deliberate heartbeat throttling
techniques that might make that theory really sing in practice.
-David
Re: Client/Server Multicast Discovery
Posted by David Blevins <da...@visi.com>.
On Aug 29, 2008, at 4:38 PM, David Blevins wrote:
> Been looking into how to do service discovery between client/server
> and server/server.
>
> We've got some code in the client to do failover on a server list
> but by default it really isn't wired up. It'd be nice to get
> something in so even if we don't support full clustered replication,
> at the very least we could support a bunch of servers that "work
> together" in a stateless fashion.
>
> Looking at adding a new "multicast" server service that just
> advertises the URIs of the other networks services available in the
> system. We'd probably want it off by default (maybe), but then a
> client could just sort of boot without being pointed to a specific
> server address and theoretically find a server to talk to.
More progress on the multicast discovery with grouping and failover.
On the area of tracking and communicating a new server list for the
cluster I added a ClusterMetaData which is similar to what the
ServerMetaData was aiming to be. The issue with the ServerMetaData is
that it's tracked on a per-ejb-proxy basis and any updates to the list
of servers in the cluster are only reflected in the proxy immediately
used. All other proxies will still hold onto the outdated list.
Second, not all request types could be clustered and have failover,
essentially only ejb requests could failover, jndi and authentication
could not. Now the ClusterMetaData version associated with the
ServerMetaData is sent to the server *before* the main request and
then the server can send back a new list regardless of which type of
request it is.
On the failover side, I essentially had to rewrite all of the
synchronization related to connection management and shutting down the
server. The code simply was not written so that it could be stopped
cleanly from a client's perspective. It's now capable of a graceful
shutdown which is of course critical for failover. Active requests
will be finished, new requests will be refused, inactive connections
with the client will be cleanly and immediately closed on the server
side. There is a bit of a trick in this regard as even if one side
perfectly and cleanly closes it may not be visible to the other side
for quite a while. Still hacking on that. Chatted with Hiram about
it a bit and it doesn't really sound like there is a "cure", only
things you can do to overcome the symptoms.
-David
Re: Client/Server Multicast Discovery
Posted by "Daniel S. Haischt" <da...@googlemail.com>.
regarding SLP I would suggest to look at Zeroconf as well. Goals of
SLP and Zeroconf
aka multicast DNS are quiet similar except the fact that (knowing you
are on a Mac)
Zeroconf is more common on Macs and Windoze machines.
Another way to advertise a service is a standard mostly driven by M$
which is named
UPnP.
I did quiet a bunch of stuff regarding Zeroconf but in C/C++ and
Python. Simply let me
know if this sounds like something that could be of use for the
client/server multicast
discovery.
Open Source SLP impl:
* OpenSLP
Open Source Zeroconf (Multicast DNS) impl:
* Avahi
* FreeBSD/NetBSD Zeroconf stack
* Apple's Bonjour
* there was a Java impl but I don't remember the name at the moment
Open Source UPnP impl:
* UPnP Port Works
* GUPnP
* Barracuda UPnP
Related stuff:
* Wide area DNS service discovery
* NAT-PMP
On Tue, Sep 9, 2008 at 10:42 PM, Alan D. Cabrera <li...@toolazydogs.com> wrote:
>
> On Sep 9, 2008, at 12:58 PM, David Blevins wrote:
>
>>
>> On Sep 9, 2008, at 12:24 PM, Alan D. Cabrera wrote:
>>
>>>
>>> On Aug 29, 2008, at 4:38 PM, David Blevins wrote:
>>>
>>>> Been looking into how to do service discovery between client/server and
>>>> server/server.
>>>>
>>>> We've got some code in the client to do failover on a server list but by
>>>> default it really isn't wired up. It'd be nice to get something in so even
>>>> if we don't support full clustered replication, at the very least we could
>>>> support a bunch of servers that "work together" in a stateless fashion.
>>>>
>>>> Looking at adding a new "multicast" server service that just advertises
>>>> the URIs of the other networks services available in the system. We'd
>>>> probably want it off by default (maybe), but then a client could just sort
>>>> of boot without being pointed to a specific server address and theoretically
>>>> find a server to talk to.
>>>
>>> Would you be using something like SLP to advertise the services?
>>
>> Wow that looks almost identical to what I cooked up. What I've got the
>> URIs are formatted like this:
>>
>> foogroup:ejb:ejbd://somehost:4201/?name=value
>> foogroup:ejb:http://somehost:80/openejb/ejb
>>
>> And so so on. Only things in the foogroup can see services of the
>> foogroup. The second scheme, "ejb", denotes the type of service. And
>> finally the "ejbd://somehost:4201/?name=value" denotes the actual location
>> of the service.
>>
>> At first glance it looks quite SLP compatible.
>
> Great minds...
>
> Yeah, for SLP, your URL could be
>
> service:ejb:ejbd://somehost:4201
>
> The attributes would be
>
> name=value
>
> and your SLP scope would be foogroup. The thing I like about it is that you
> can use LDAP query language to select services based on their attributes.
>
>
> Regards,
> Alan
>
>
Re: Client/Server Multicast Discovery
Posted by "Alan D. Cabrera" <li...@toolazydogs.com>.
On Sep 9, 2008, at 12:58 PM, David Blevins wrote:
>
> On Sep 9, 2008, at 12:24 PM, Alan D. Cabrera wrote:
>
>>
>> On Aug 29, 2008, at 4:38 PM, David Blevins wrote:
>>
>>> Been looking into how to do service discovery between client/
>>> server and server/server.
>>>
>>> We've got some code in the client to do failover on a server list
>>> but by default it really isn't wired up. It'd be nice to get
>>> something in so even if we don't support full clustered
>>> replication, at the very least we could support a bunch of servers
>>> that "work together" in a stateless fashion.
>>>
>>> Looking at adding a new "multicast" server service that just
>>> advertises the URIs of the other networks services available in
>>> the system. We'd probably want it off by default (maybe), but
>>> then a client could just sort of boot without being pointed to a
>>> specific server address and theoretically find a server to talk to.
>>
>> Would you be using something like SLP to advertise the services?
>
> Wow that looks almost identical to what I cooked up. What I've got
> the URIs are formatted like this:
>
> foogroup:ejb:ejbd://somehost:4201/?name=value
> foogroup:ejb:http://somehost:80/openejb/ejb
>
> And so so on. Only things in the foogroup can see services of the
> foogroup. The second scheme, "ejb", denotes the type of service.
> And finally the "ejbd://somehost:4201/?name=value" denotes the
> actual location of the service.
>
> At first glance it looks quite SLP compatible.
Great minds...
Yeah, for SLP, your URL could be
service:ejb:ejbd://somehost:4201
The attributes would be
name=value
and your SLP scope would be foogroup. The thing I like about it is
that you can use LDAP query language to select services based on their
attributes.
Regards,
Alan
Re: Client/Server Multicast Discovery
Posted by David Blevins <da...@visi.com>.
On Sep 9, 2008, at 12:24 PM, Alan D. Cabrera wrote:
>
> On Aug 29, 2008, at 4:38 PM, David Blevins wrote:
>
>> Been looking into how to do service discovery between client/server
>> and server/server.
>>
>> We've got some code in the client to do failover on a server list
>> but by default it really isn't wired up. It'd be nice to get
>> something in so even if we don't support full clustered
>> replication, at the very least we could support a bunch of servers
>> that "work together" in a stateless fashion.
>>
>> Looking at adding a new "multicast" server service that just
>> advertises the URIs of the other networks services available in the
>> system. We'd probably want it off by default (maybe), but then a
>> client could just sort of boot without being pointed to a specific
>> server address and theoretically find a server to talk to.
>
> Would you be using something like SLP to advertise the services?
Wow that looks almost identical to what I cooked up. What I've got
the URIs are formatted like this:
foogroup:ejb:ejbd://somehost:4201/?name=value
foogroup:ejb:http://somehost:80/openejb/ejb
And so so on. Only things in the foogroup can see services of the
foogroup. The second scheme, "ejb", denotes the type of service. And
finally the "ejbd://somehost:4201/?name=value" denotes the actual
location of the service.
At first glance it looks quite SLP compatable.
-David
Re: Client/Server Multicast Discovery
Posted by "Alan D. Cabrera" <li...@toolazydogs.com>.
On Aug 29, 2008, at 4:38 PM, David Blevins wrote:
> Been looking into how to do service discovery between client/server
> and server/server.
>
> We've got some code in the client to do failover on a server list
> but by default it really isn't wired up. It'd be nice to get
> something in so even if we don't support full clustered replication,
> at the very least we could support a bunch of servers that "work
> together" in a stateless fashion.
>
> Looking at adding a new "multicast" server service that just
> advertises the URIs of the other networks services available in the
> system. We'd probably want it off by default (maybe), but then a
> client could just sort of boot without being pointed to a specific
> server address and theoretically find a server to talk to.
Would you be using something like SLP to advertise the services?
Regards,
Alan