You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@river.apache.org by Tom Hobbs <tv...@googlemail.com> on 2010/02/09 11:21:07 UTC
Service Wrapper Example
Hi,
I mentioned in another thread that I had come across code which provides
service fail over and auto-rediscovery. I've posted details of the kind of
code that was used to (note this has been reinvented in my head just now and
only loosely tested);
http://wiki.apache.org/river/AutomaticServiceReplacement
I hope that someone finds it useful and/or interesting.
It's important to note that in the interests of simplicity the
ServiceWrapper class in the article is explicitly linked to the dummy
service described. Obviously using more reflection magic it's possible to
remove this linkage so that ServiceWrapper can wrap any service you like and
it's invoke method would invoke the method supplied to it, rather than the
only method available in the dummy article.
Thanks to Jukka for his Wiki account suggestion.
Enjoy,
Tom
Re: Service Wrapper Example
Posted by Peter Firmstone <ji...@zeus.net.au>.
Hey, thanks Tom, there's still some life in River yet old mate, off to
bed now so I'll have a look in the morning.
Cheers,
Peter.
Tom Hobbs wrote:
> Hi,
>
> I mentioned in another thread that I had come across code which provides
> service fail over and auto-rediscovery. I've posted details of the kind of
> code that was used to (note this has been reinvented in my head just now and
> only loosely tested);
>
> http://wiki.apache.org/river/AutomaticServiceReplacement
>
> I hope that someone finds it useful and/or interesting.
>
> It's important to note that in the interests of simplicity the
> ServiceWrapper class in the article is explicitly linked to the dummy
> service described. Obviously using more reflection magic it's possible to
> remove this linkage so that ServiceWrapper can wrap any service you like and
> it's invoke method would invoke the method supplied to it, rather than the
> only method available in the dummy article.
>
> Thanks to Jukka for his Wiki account suggestion.
>
> Enjoy,
>
> Tom
>
>
Re: Service Wrapper Example
Posted by Peter Firmstone <ji...@zeus.net.au>.
Sim IJskes - QCG wrote:
> Tom Hobbs wrote:
>
>> Certainly in my experience, detecting errors and recovering is always
>> the
>> job of the client. To use a daft example; why would a web page
>> detect that
>> a browser has unexpectedly disappeared and try to find a new browser to
>> display itself on? But in the event of a web server going down, it's
>> always
>> the browser/etc that needs to go any find another copy of the page
>> somewhere.
>
> This is not always the case. For instance in the transport layer. A
> server can detect that an ack/nack is overdue and start a retransmission.
>
> But thats not what i tried to express. In that specific email i meant
> a client of the service. I haven't seen any self healing behaviour in
> the jeri transports, or the layers between the actual java.lang.Proxy
> of the service and its transport, so any hickup there will lead to a
> RemoteException. So i guess, with the current state of affairs, the
> only place for selfhealing (with keeping the RemoteReference the same)
> is for a SmartProxy.
>
> What you have done, is created a ServiceWrapper which does the
> wrapping/proxying on the clients initiative, and retrieves a new
> RemoteReference for every transport error. This is also a perfectly
> valid approach.
>
> The only problem i see, is (in both scenarios) that when an anonymous
> (not registered, bu exported) remote reference gets serialized as for
> instance a return value from a call to the service, and this reference
> is passed through the system, it will still experience transport
> errors. So this remote reference needs to be wrapped also, either at
> the server or the client side.
>
> While writing this, i'm thinking this might be also fixed in the
> invocation layer. Altough it still only guards against transport
> errors, and not against dropping a member of a server cluster.
>
>> This style of service wrapping worked very well in a complex trading
>> platform that I was previously involved in. It enabled us, with the
>> provision of some additional business rules - especially regarding
>> state, to
>> take down services at random and have the system automatically recover
>> without interrupting the client. It truly was a self-healing system.
>
> Indeed, i can see this. And very practical for dynamic cluster scaling
> issues, for instance during a deployment of a new version. (i'm
> thinking about: during the change reducing the number of cluster
> members, upgrading the freed cluster members, and a hot switchover for
> the 2 groups).
>
> Gr. Sim
>
Indeed, this does look very useful, thanks for the contribution &
example. It's a different problem domain than firewall traversal, I'm
looking into to how to handle possible security issues with dynamic
address & port changes on firewalls at a much lower level, in my case I
want to ensure I have the same service and if not throw a
RemoteException or a Security Exception, otherwise the opportunity might
exist for an attacker to substitute a service after it has been
authenticated. Once the connection is lost that's where your solution
is useful.
Cheers,
Peter.
Re: Service Wrapper Example
Posted by Sim IJskes - QCG <si...@qcg.nl>.
Tom Hobbs wrote:
> The only thing I think needs clarification is this bit;
>
>> Assuming we agree, if you have a wrapped service (with error recovery) but
> the wrapper is allowed to return a
>> Remote reference which is passed through the program (without beeing
> wrapped) a transport (or remote)
>> error will cause effects outside the control of the wrapper. (So you
> basically need a recursive wrapper to
>> counter this.)
>
> You're right, in that if you manage to get hold of a proxy that hasn't been
> wrapped, then any kind of exception in any layer could occur anywhere that
> the client code uses that proxy. But isn't this obvious? If you have a
> strategy for self-healing then you need to make sure your client code
> accessing the proxies only through that strategy. In this case, wrapping
> the remote reference in a Proxy. That wrapped proxy is then passed around
> inside the client code.
I especially like the 'recursive' nature of JERI. The idea of beeing
able to serve a reference to a service that is not even on your VM is
very inspiring.
The wrapper solution presented, did only take into account a 'flat'
service (no value judgement there!). As a thinking excercise i wanted to
take this a step further and think about a 'automatic' recursive
wrapper. To counter transport errors, this would be easy, for service
redundancy this would mean glueing a terracotta cloud behind jini in
order to keep object references.
> Certainly in "normal" circumstances, this Proxy wrapper wouldn't allow
> access to the underlying wrapped remote proxy so all calls would go through
> the reflection code without the client code being any wiser.
>
> The "service finding" mechanism inside your strategy can be River or normal
> RMI or whatever.
>
> Does that make sense? Are we argumentatively agreeing with each other? :-)
Yes! :-) Let's "stir the pot" some more!
Gr. Sim
--
QCG, Software voor het MKB, 071-5890970, http://www.qcg.nl
Quality Consultancy Group b.v., Leiderdorp, Kvk Leiden: 28088397
Re: Service Wrapper Example
Posted by Tom Hobbs <tv...@googlemail.com>.
Hi Sim,
You did indeed answer my questions. Thanks for the "registered vs exported"
description. I've spent so much time looking at Jini/River services that I
tend to forget that other things exist!
The only thing I think needs clarification is this bit;
> Assuming we agree, if you have a wrapped service (with error recovery) but
the wrapper is allowed to return a
> Remote reference which is passed through the program (without beeing
wrapped) a transport (or remote)
> error will cause effects outside the control of the wrapper. (So you
basically need a recursive wrapper to
> counter this.)
You're right, in that if you manage to get hold of a proxy that hasn't been
wrapped, then any kind of exception in any layer could occur anywhere that
the client code uses that proxy. But isn't this obvious? If you have a
strategy for self-healing then you need to make sure your client code
accessing the proxies only through that strategy. In this case, wrapping
the remote reference in a Proxy. That wrapped proxy is then passed around
inside the client code.
Certainly in "normal" circumstances, this Proxy wrapper wouldn't allow
access to the underlying wrapped remote proxy so all calls would go through
the reflection code without the client code being any wiser.
The "service finding" mechanism inside your strategy can be River or normal
RMI or whatever.
Does that make sense? Are we argumentatively agreeing with each other? :-)
Cheers,
Tom
Re: Service Wrapper Example
Posted by Sim IJskes - QCG <si...@qcg.nl>.
Tom Hobbs wrote:
> I think that it's possible my terminology is letting me down. If I appear
> to be getting defensive over my example, then I apologise and it's not
> intended, I just haven't experienced the problems that you describe.
No. Please. It's me. If i cannot find a word quickly enough, i tend to
reinvent one. Most of the time it works out fine! :-)
>> For instance in the transport layer. A server can detect that an ack/nack
> is overdue and start a
>> retransmission.
>
> But will this not only be appropriate if the remote service remains
> discoverable and in a (business context) working state? The following is a
> question that I genuinely do not know the answer to; if there is a
> transmission problem, what is quicker? To "start a retransmission" and
> possibly have to wait for another timeout to occur, or drop the remote
> reference and discover a new service with the assumption that there wouldn't
> be any transmission problems between the client and the new service?
> Obviously this might be problematic if you are relying on your business
> services to be having a certain amount of state.
I was a bit aspecific here. I used transport layer in an abstract way,
but in this paragraph one could replace it with TCP/IP.
>> I haven't seen any self healing behaviour in the jeri transports, or the
> layers between the actual
>> java.lang.Proxy of the service and its transport
>
> Neither have I, which is why when outside of low level transport layers,
> recovery and self-healing are all down to the client.
Indeed, thats an option. No problem with that.
>> retrieves a new RemoteReference for every transport error
>
> Actually, the service could throw a business-context exception (as long as
> it extends RemoteException) and that would prompt the retrieval of a new
> Remote Reference also.
Indeed, the beauty of jini is, that one has several intercept points to
choose from.
>> (not registered, bu exported) remote reference gets serialized as for
> instance a return value from a call to
>> the service, and this reference is passed through the system, it will
> still experience transport errors.
>
> Can you explain this for me, please? If I have wrapped a remote proxy and
> call the "sayHello()" method on, it returns a String which gets deserialised
> on the client side, i.e. in the Wrapper, into a "normal" Java String which
> can be read and have all the wonderful String methods called on it without
> risking a RemoteException. Right? Or have I got the wrong end of the
> stick?
>
> Also, can you point me to somewhere that I can read up on the differences
> between "not registered, but exported", please? That bit went over my head.
If a exported service returns a exported service, this returned service
reference does not need to be registered (in reggie for instance) but it
still is exported. When a service is exported it is turned into a
serializable reference for use outside the JVM it exists in, and it can
be passed to another JVM. (see the javadoc of java.rmi.Remote).
You can view this process by setting a breakpoint on the first line of
the ObjectOutputStream.writeSerialData method, and inspect the passed
'obj' parameter.
Assuming we agree, if you have a wrapped service (with error recovery)
but the wrapper is allowed to return a Remote reference which is passed
through the program (without beeing wrapped) a transport (or remote)
error will cause effects outside the control of the wrapper. (So you
basically need a recursive wrapper to counter this.)
Am i on the right track here?
> I'm trying to understand exactly what you and Peter are trying to solve (and
> if this example helps any). Then maybe I can start contributing more to
> helping you guys.
I wasn't specifically solving a problem. Saw your wiki entry, recognized
the underlying problem as one i also have on my mind, became inspired
and started freewheeling beyond the horizon.
Gr. Sim
--
QCG, Software voor het MKB, 071-5890970, http://www.qcg.nl
Quality Consultancy Group b.v., Leiderdorp, Kvk Leiden: 28088397
Re: Service Wrapper Example
Posted by Tom Hobbs <tv...@googlemail.com>.
I think that it's possible my terminology is letting me down. If I appear
to be getting defensive over my example, then I apologise and it's not
intended, I just haven't experienced the problems that you describe.
> For instance in the transport layer. A server can detect that an ack/nack
is overdue and start a
> retransmission.
But will this not only be appropriate if the remote service remains
discoverable and in a (business context) working state? The following is a
question that I genuinely do not know the answer to; if there is a
transmission problem, what is quicker? To "start a retransmission" and
possibly have to wait for another timeout to occur, or drop the remote
reference and discover a new service with the assumption that there wouldn't
be any transmission problems between the client and the new service?
Obviously this might be problematic if you are relying on your business
services to be having a certain amount of state.
> I haven't seen any self healing behaviour in the jeri transports, or the
layers between the actual
> java.lang.Proxy of the service and its transport
Neither have I, which is why when outside of low level transport layers,
recovery and self-healing are all down to the client.
> retrieves a new RemoteReference for every transport error
Actually, the service could throw a business-context exception (as long as
it extends RemoteException) and that would prompt the retrieval of a new
Remote Reference also.
> (not registered, bu exported) remote reference gets serialized as for
instance a return value from a call to
> the service, and this reference is passed through the system, it will
still experience transport errors.
Can you explain this for me, please? If I have wrapped a remote proxy and
call the "sayHello()" method on, it returns a String which gets deserialised
on the client side, i.e. in the Wrapper, into a "normal" Java String which
can be read and have all the wonderful String methods called on it without
risking a RemoteException. Right? Or have I got the wrong end of the
stick?
Also, can you point me to somewhere that I can read up on the differences
between "not registered, but exported", please? That bit went over my head.
> i'm thinking about: during the change reducing the number of cluster
members, upgrading the freed cluster
> members, and a hot switchover for the 2 groups
This is indeed what this service wrapping method can be used for. If
Service_A needs Service_B to do it's job and there are multiple Service_Bs
on the subnet. If the machine with the B on that A is using explodes, or if
it's JVM crashes, or if the service abruptly shuts down or stop responding
or if it is purposely shutdown or etc; then when A next calls a method on
the wrapped B, it will automatically find a different B (assuming one is
available) all without A having to understand the it's original B reference
is defunct and needs to be replaced.
In my experience, all the return values that A got from it's original B
reference are all still valid and able to be used. Which is why I'm
struggling to understand your comment about "a return value from a call to
the service...it will still experience transport errors". But I can fully
believe that because I'm misunderstanding you, which is probably my fault.
:-)
I'm trying to understand exactly what you and Peter are trying to solve (and
if this example helps any). Then maybe I can start contributing more to
helping you guys.
Sorry if the answers to these questions are in the specs somewhere, I just
missed them.
Cheers,
Tom
On Mon, Feb 15, 2010 at 5:43 PM, Sim IJskes - QCG <si...@qcg.nl> wrote:
> Tom Hobbs wrote:
>
> Certainly in my experience, detecting errors and recovering is always the
>> job of the client. To use a daft example; why would a web page detect
>> that
>> a browser has unexpectedly disappeared and try to find a new browser to
>> display itself on? But in the event of a web server going down, it's
>> always
>> the browser/etc that needs to go any find another copy of the page
>> somewhere.
>>
>
> This is not always the case. For instance in the transport layer. A server
> can detect that an ack/nack is overdue and start a retransmission.
>
> But thats not what i tried to express. In that specific email i meant a
> client of the service. I haven't seen any self healing behaviour in the jeri
> transports, or the layers between the actual java.lang.Proxy of the service
> and its transport, so any hickup there will lead to a RemoteException. So i
> guess, with the current state of affairs, the only place for selfhealing
> (with keeping the RemoteReference the same) is for a SmartProxy.
>
> What you have done, is created a ServiceWrapper which does the
> wrapping/proxying on the clients initiative, and retrieves a new
> RemoteReference for every transport error. This is also a perfectly valid
> approach.
>
> The only problem i see, is (in both scenarios) that when an anonymous (not
> registered, bu exported) remote reference gets serialized as for instance a
> return value from a call to the service, and this reference is passed
> through the system, it will still experience transport errors. So this
> remote reference needs to be wrapped also, either at the server or the
> client side.
>
> While writing this, i'm thinking this might be also fixed in the invocation
> layer. Altough it still only guards against transport errors, and not
> against dropping a member of a server cluster.
>
>
> This style of service wrapping worked very well in a complex trading
>> platform that I was previously involved in. It enabled us, with the
>> provision of some additional business rules - especially regarding state,
>> to
>> take down services at random and have the system automatically recover
>> without interrupting the client. It truly was a self-healing system.
>>
>
> Indeed, i can see this. And very practical for dynamic cluster scaling
> issues, for instance during a deployment of a new version. (i'm thinking
> about: during the change reducing the number of cluster members, upgrading
> the freed cluster members, and a hot switchover for the 2 groups).
>
> Gr. Sim
>
> --
> QCG, Software voor het MKB, 071-5890970, http://www.qcg.nl
> Quality Consultancy Group b.v., Leiderdorp, Kvk Leiden: 28088397
>
Re: Service Wrapper Example
Posted by Sim IJskes - QCG <si...@qcg.nl>.
Tom Hobbs wrote:
> Certainly in my experience, detecting errors and recovering is always the
> job of the client. To use a daft example; why would a web page detect that
> a browser has unexpectedly disappeared and try to find a new browser to
> display itself on? But in the event of a web server going down, it's always
> the browser/etc that needs to go any find another copy of the page
> somewhere.
This is not always the case. For instance in the transport layer. A
server can detect that an ack/nack is overdue and start a retransmission.
But thats not what i tried to express. In that specific email i meant a
client of the service. I haven't seen any self healing behaviour in the
jeri transports, or the layers between the actual java.lang.Proxy of the
service and its transport, so any hickup there will lead to a
RemoteException. So i guess, with the current state of affairs, the only
place for selfhealing (with keeping the RemoteReference the same) is for
a SmartProxy.
What you have done, is created a ServiceWrapper which does the
wrapping/proxying on the clients initiative, and retrieves a new
RemoteReference for every transport error. This is also a perfectly
valid approach.
The only problem i see, is (in both scenarios) that when an anonymous
(not registered, bu exported) remote reference gets serialized as for
instance a return value from a call to the service, and this reference
is passed through the system, it will still experience transport errors.
So this remote reference needs to be wrapped also, either at the server
or the client side.
While writing this, i'm thinking this might be also fixed in the
invocation layer. Altough it still only guards against transport errors,
and not against dropping a member of a server cluster.
> This style of service wrapping worked very well in a complex trading
> platform that I was previously involved in. It enabled us, with the
> provision of some additional business rules - especially regarding state, to
> take down services at random and have the system automatically recover
> without interrupting the client. It truly was a self-healing system.
Indeed, i can see this. And very practical for dynamic cluster scaling
issues, for instance during a deployment of a new version. (i'm thinking
about: during the change reducing the number of cluster members,
upgrading the freed cluster members, and a hot switchover for the 2 groups).
Gr. Sim
--
QCG, Software voor het MKB, 071-5890970, http://www.qcg.nl
Quality Consultancy Group b.v., Leiderdorp, Kvk Leiden: 28088397
Re: Service Wrapper Example
Posted by Tom Hobbs <tv...@googlemail.com>.
> It is certainly interesting. It would only work when one has only one
remote reference in memory, would it?
> I mean, if you lookup the main service object and via a remote call it
returns some other remote reference,
> and you lose contact, how would it recover? Or is the intention only to
iterate over multiple instances of one
> service during setup?
Well, assuming I understand your question correctly. In any one JVM you
could have as many "ServiceWrapper" instances as you want, (all created with
the same ServiceTemplate), which may or may not all be pointing to the same
remote service. The way I have used it before is with static method calls,
e.g.
MyService myService1 = ServiceWrapper.findMeAService(someTemplate);
MyService myService2 = ServiceWrapper.findMeAService(someTemplate);
Here, there is no guarantee as to whether myService1 and 2 are wrappers to
the same remote proxy or not. But you can guarantee that two remote lookups
have been made and two proxies have been returned.
Certainly in my experience, detecting errors and recovering is always the
job of the client. To use a daft example; why would a web page detect that
a browser has unexpectedly disappeared and try to find a new browser to
display itself on? But in the event of a web server going down, it's always
the browser/etc that needs to go any find another copy of the page
somewhere.
The ServiceWrapper example I provided essentially wraps all calls to the
remote proxy, so if any kind of exception is thrown on invocation, that
exception can be caught and handled in the wrapper. In the case where the
service has gone away or just broken, the wrapper can attempt to find a
replacement service and then reissue the method call. This would manifest
itself in the client as just a remote call that takes slightly longer to
execute (because another lookup is being done).
If you have a situation where your remote proxy needs some kind of state,
then you have a more complex job of getting any new service to the same
state as the old one before the ServiceWrapper executes the original method
call.
What the wrapper does is essentially provide a Reflection-based AOP around
all calls to the service, so you can make it as complex or simple, generic
or business-specific, as your needs require.
This style of service wrapping worked very well in a complex trading
platform that I was previously involved in. It enabled us, with the
provision of some additional business rules - especially regarding state, to
take down services at random and have the system automatically recover
without interrupting the client. It truly was a self-healing system.
On Fri, Feb 12, 2010 at 9:22 PM, Sim IJskes - QCG <si...@qcg.nl> wrote:
> Tom Hobbs wrote:
>
>> Hi,
>>
>> I mentioned in another thread that I had come across code which provides
>> service fail over and auto-rediscovery. I've posted details of the kind
>> of
>> code that was used to (note this has been reinvented in my head just now
>> and
>> only loosely tested);
>>
>> http://wiki.apache.org/river/AutomaticServiceReplacement
>>
>> I hope that someone finds it useful and/or interesting.
>>
>
> It is certainly interesting. It would only work when one has only one
> remote reference in memory, would it? I mean, if you lookup the main service
> object and via a remote call it returns some other remote reference, and you
> lose contact, how would it recover? Or is the intention only to iterate over
> multiple instances of one service during setup?
>
> It is certainly something what starts to be a recurring theme (for me), a
> soon as you start to offer a more complex service, how do you handle errors?
> It looks to me, that the client is totally responsible for recovery, and
> that all of jini is built with this premise. All the recovery you get at the
> moment is the TCP retry mechanism. But if a laptop resumes from a suspend
> and gets a new IP address via DHCP an error is thrown back to the client
> (user).
>
> Currently i have some recovery built into our 'rpc/ip through http tunnel',
> but this covers only transport errors. No recovery to fix disappearing
> individual service instances in a redundant cluster.
>
> I wonder if it was the intention by 'the elders' that we go back to the
> lookup service, and try to find a service again. I'm not sure if we can hide
> this recovery in a serviceproxy. Any ideas?
>
> Gr. Sim
>
>
Re: Service Wrapper Example
Posted by Sim IJskes - QCG <si...@qcg.nl>.
Tom Hobbs wrote:
> Hi,
>
> I mentioned in another thread that I had come across code which provides
> service fail over and auto-rediscovery. I've posted details of the kind of
> code that was used to (note this has been reinvented in my head just now and
> only loosely tested);
>
> http://wiki.apache.org/river/AutomaticServiceReplacement
>
> I hope that someone finds it useful and/or interesting.
It is certainly interesting. It would only work when one has only one
remote reference in memory, would it? I mean, if you lookup the main
service object and via a remote call it returns some other remote
reference, and you lose contact, how would it recover? Or is the
intention only to iterate over multiple instances of one service during
setup?
It is certainly something what starts to be a recurring theme (for me),
a soon as you start to offer a more complex service, how do you handle
errors? It looks to me, that the client is totally responsible for
recovery, and that all of jini is built with this premise. All the
recovery you get at the moment is the TCP retry mechanism. But if a
laptop resumes from a suspend and gets a new IP address via DHCP an
error is thrown back to the client (user).
Currently i have some recovery built into our 'rpc/ip through http
tunnel', but this covers only transport errors. No recovery to fix
disappearing individual service instances in a redundant cluster.
I wonder if it was the intention by 'the elders' that we go back to the
lookup service, and try to find a service again. I'm not sure if we can
hide this recovery in a serviceproxy. Any ideas?
Gr. Sim
RE: Service Wrapper Example
Posted by Christopher Dolan <ch...@avid.com>.
Tom,
Sharp eyes. :-) I should have left IRedundantService out of it --
that's just a detail of our implementation.
Our client-side code looks for primary=true if IFooService is an
IRedundantService and doesn't look for that Entry if not. This has
allowed us to change our mind about whether a service should be a pool
of equals or a primary-backup hierarchy without changing client-side
code.
I'm glossing over a ton of the details (like that the primary=true
filtering happens client-side instead of in the registrar), but I think
you can see what I mean.
Chris
-----Original Message-----
From: Tom Hobbs [mailto:tvhobbs@googlemail.com]
Sent: Tuesday, February 16, 2010 10:42 AM
To: river-dev@incubator.apache.org
Subject: Re: Service Wrapper Example
Hi Chris,
<snip>
What is the purpose of "IRedundantService". Surely "IFooService" could
just
join the groups of all the other "IFooService" services?
Interesting none the less though.
Thanks for sharing.
Tom
Re: Service Wrapper Example
Posted by Tom Hobbs <tv...@googlemail.com>.
Hi Chris,
That's an interesting approach and one that avoids the single point of
failure that the similar approach I've encountered introduces....
1. When a Service starts is checks to see if there is a Primary service of
the same type running
2. If so, it makes itself as Secondary and keep pinging the primary waiting
for it to die
3. On a Primary's death (or when there are no primaries), all the
secondaries of that type hold a vote and one of them gets promoted. The
rest revert back to Secondary.
Clients still search for and use the Primary service only.
In the specific example I was involved with, there was a known good Java
Space where voting happened.
The Good:
1. In addition to Chris'
2. No need for intercommunication other than pinging the primary
The Bad:
1. In addition to Chris'
2. The one good known Java Space for voting in is a single point of failure*
* The assumption that was made in our scenario was that Jini (or River)
"core" implementations were far less likely to unexpectedly disappear than
'normal' business services. We could therefore assume that once one was
started on the subnet it would "always" be available.
In reality, this single point of failure never actually manifested itself as
an issue. Yes, on a complete system restart we might forget to start this
Java Space before the Primary/Secondary services, but with appropriate
logging the situation is easy to diagnose. Day-to-day this Java Space
(Outrigger) behaved itself very well and didn't go away when we didn't want
it to. Also, this specific Java Space was never used by the rest of the
system for anything other than voting.
What is the purpose of "IRedundantService". Surely "IFooService" could just
join the groups of all the other "IFooService" services?
Interesting none the less though.
Thanks for sharing.
Tom
On Tue, Feb 16, 2010 at 4:26 PM, Christopher Dolan <
christopher.dolan@avid.com> wrote:
> Tom's technique below is nice for automatic failover. Just for a point
> of reference, I'll describe the hot-standby approach we've taken at
> Avid.
>
> * Declare that your IFooService is an IRedundantService
> * The IFooService instances in the group discover each other and hold
> an election
> * The winner calls joinMgr.modifyAttributes() to mark itself as primary
> * The IFooService instances ping each other and if any one stops
> responding, another election is triggered
> * Clients search for the IFooService that is marked primary=true and
> talk only to it
>
> Limitations of this approach:
> * need for IFooService instances to inter-communicate
> * moments during election when there is no primary
> * rare partial-disconnect conditions where you get two primaries
>
> Wins of this approach:
> * election protocol is entirely up to the IFooServices, no client logic
> except to trust the Entry that says primary=true
> * only one primary at a time
> * the IFooService instances are already in contact, so adding state
> synchronization requires little extra infrastructure
>
> Good-and-bad aspects:
> * clients can decide how to handle the no-primary case themselves.
> More code, but more control too.
>
>
> I'd love to hear comments/criticisms of our approach, and if anyone has
> implemented something similar.
>
> Chris
>
> -----Original Message-----
> From: Tom Hobbs [mailto:tvhobbs@googlemail.com]
> Sent: Tuesday, February 09, 2010 4:21 AM
> To: river-dev@incubator.apache.org
> Subject: Service Wrapper Example
>
> Hi,
>
> I mentioned in another thread that I had come across code which provides
> service fail over and auto-rediscovery. I've posted details of the kind
> of
> code that was used to (note this has been reinvented in my head just now
> and
> only loosely tested);
>
> http://wiki.apache.org/river/AutomaticServiceReplacement
>
> I hope that someone finds it useful and/or interesting.
>
> It's important to note that in the interests of simplicity the
> ServiceWrapper class in the article is explicitly linked to the dummy
> service described. Obviously using more reflection magic it's possible
> to
> remove this linkage so that ServiceWrapper can wrap any service you like
> and
> it's invoke method would invoke the method supplied to it, rather than
> the
> only method available in the dummy article.
>
> Thanks to Jukka for his Wiki account suggestion.
>
> Enjoy,
>
> Tom
>
Re: Service Wrapper Example
Posted by Dennis Reedy <de...@gmail.com>.
On Feb 16, 2010, at 210PM, Christopher Dolan wrote:
> Dennis,
>
> That's interesting, and is a reminder to me that I need to study Rio
> more. In your Utilization case, does that allow for arbitrary
> server-side decision making?
The Utilization strategy is tapped into compute resource utilization, generally surrounding depletion oriented resources like memory, disk, etc ...
> For example, if a service loses its
> connection to a backend database, can it demote itself?
The utilization strategy is not focused on that, but it would be simple enough to create a Strategy that allows something like that that to happen. I suppose coming up with what the right attribute (or property) to indicate the service is not capable of servicing requests is the first thing to do.
Maybe setting the net.jini.lookup.entry.Status entry to have a severity of net.jini.lookup.entry.StatusType.ERROR, and the client-side generated proxy would not include a service that has a StatusType of StatusType.ERROR. How do you currently indicate that a service loses it's backend database?
> That's an
> important use case that led us to service-side elections and relatively
> dumb clients.
Yep.
Regards
Dennis
RE: Service Wrapper Example
Posted by Christopher Dolan <ch...@avid.com>.
Dennis,
That's interesting, and is a reminder to me that I need to study Rio
more. In your Utilization case, does that allow for arbitrary
server-side decision making? For example, if a service loses its
connection to a backend database, can it demote itself? That's an
important use case that led us to service-side elections and relatively
dumb clients.
Chris
-----Original Message-----
From: Dennis Reedy [mailto:dennis.reedy@gmail.com]
Sent: Tuesday, February 16, 2010 10:56 AM
To: river-dev@incubator.apache.org
Subject: Re: Service Wrapper Example
Similar to whats being discussed here is wrapped up in something called
associations in Rio.
Associations are a way to declare that a service uses (has an
association to) another service. Associations define a usage model for
services in and across architectures/deployments. Associations can be
used either at deployment time to assist in how a service get created in
the context of the service(s) it is associated to, or to wire-up
distributed services, injected service references into the declaring
service.
Associations are declared in service deployment configuration, and if
they have been declared with the 'property' attribute, discovered
services will be injected into your service using an IoC approach. Based
on the setter method's signature, the injection type differs.
Given the examples below:
public void setFoos(Iterable<Foo> foos) {
this.foos = foos;
}
public void setFooAssociation(Association<Foo> foo) {
this.foo = foo;
}
public void setFoo(Foo foo) {
this.foo = foo;
}
If the injection point is the the last approach, Rio generates a
client-side proxy and *wraps* discovered associated service instances
with a service selection policy. The service selection strategy provides
a way to determine how services in the collection of discovered services
are invoked. The current service selection strategies are fail-over,
round-robin and utilization (note all service selection strategies will
also prefer local services over remote services).
* The Fail-Over strategy will invoke the first service in it's
collection for each method invocation until that service is no longer
reachable. If the associated service is unavailable, the fail-over
strategy invokes the next service in it's list.
* The Round Robin strategy alternates selection of services from the
association, alternating the selection of associated services using a
round-robin approach.
* The Utilization strategy is a round-robin selector that selects
services running on compute resources whose system resources are not
depleted. System resource depletion is determined by
org.rioproject.system.MeasuredResource provided as part of the
org.rioproject.system.ComputeResourceUtilization object returned as part
of the deployment map. If any of a Cybernode's resources are depleted,
the service hosted in that Cybernode will not be invoked. This is of
particular use in cases where out of memory conditions may occur. Using
the utilization strategy a service running in a memory constrained
Cybernode will not be invoked until the JVM performs garbage collection
and memory is reclaimed.
Associations can be injected in a lazy or eager mode. Lazy injection is
the default, and injection occurs when a service is discovered. Eager
injection occurs immediately, even if there are no discovered services.
Associations must also deal with proxy failure modes. In a typical
distributed environment, if there are no discovered (available) services
and a remote method invocation is attempted on a service, a
RemoteException is thrown. This seems to make sense, although a better
approach may be in order. Perhaps the dynamic proxy can either wait
until a service is discovered to make the invocation, or provide at
least a retry or timeout. It would seem that the invoking client would
need to do this in any case, and from a coding point of view this
behavior would be in the generated proxy, not in the application code.
The difference here is in a fail-fast approach, service invocation will
fail in an immediate and visible way, allowing the caller to be notified
immediately that something is wrong. In our context here, we would not
immediately raise the RemoteException (at least not right away), we
would go onto the next service. We would fail-fast if there are no
services available. In association terminology, the association would be
broken.
With a fail-safe approach, we would want to constrain the notification
that no services are available (as opposed to failing fast by throwing a
RemoteException) in a managed way having the association proxy retry for
a certain period of time. In this case the fail-safe mode would have the
caller blocking on the service's remote method invocation until a
service becomes available, or until the retry logic exhausts itself. If
the retries are exhausted and there are still no available services, a
RemoteException will be thrown.
Re: Service Wrapper Example
Posted by Dennis Reedy <de...@gmail.com>.
Similar to whats being discussed here is wrapped up in something called associations in Rio.
Associations are a way to declare that a service uses (has an association to) another service. Associations define a usage model for services in and across architectures/deployments. Associations can be used either at deployment time to assist in how a service get created in the context of the service(s) it is associated to, or to wire-up distributed services, injected service references into the declaring service.
Associations are declared in service deployment configuration, and if they have been declared with the 'property' attribute, discovered services will be injected into your service using an IoC approach. Based on the setter method's signature, the injection type differs.
Given the examples below:
public void setFoos(Iterable<Foo> foos) {
this.foos = foos;
}
public void setFooAssociation(Association<Foo> foo) {
this.foo = foo;
}
public void setFoo(Foo foo) {
this.foo = foo;
}
If the injection point is the the last approach, Rio generates a client-side proxy and *wraps* discovered associated service instances with a service selection policy. The service selection strategy provides a way to determine how services in the collection of discovered services are invoked. The current service selection strategies are fail-over, round-robin and utilization (note all service selection strategies will also prefer local services over remote services).
• The Fail-Over strategy will invoke the first service in it's collection for each method invocation until that service is no longer reachable. If the associated service is unavailable, the fail-over strategy invokes the next service in it's list.
• The Round Robin strategy alternates selection of services from the association, alternating the selection of associated services using a round-robin approach.
• The Utilization strategy is a round-robin selector that selects services running on compute resources whose system resources are not depleted. System resource depletion is determined by org.rioproject.system.MeasuredResource provided as part of the org.rioproject.system.ComputeResourceUtilization object returned as part of the deployment map. If any of a Cybernode's resources are depleted, the service hosted in that Cybernode will not be invoked. This is of particular use in cases where out of memory conditions may occur. Using the utilization strategy a service running in a memory constrained Cybernode will not be invoked until the JVM performs garbage collection and memory is reclaimed.
Associations can be injected in a lazy or eager mode. Lazy injection is the default, and injection occurs when a service is discovered. Eager injection occurs immediately, even if there are no discovered services.
Associations must also deal with proxy failure modes. In a typical distributed environment, if there are no discovered (available) services and a remote method invocation is attempted on a service, a RemoteException is thrown. This seems to make sense, although a better approach may be in order. Perhaps the dynamic proxy can either wait until a service is discovered to make the invocation, or provide at least a retry or timeout. It would seem that the invoking client would need to do this in any case, and from a coding point of view this behavior would be in the generated proxy, not in the application code.
The difference here is in a fail-fast approach, service invocation will fail in an immediate and visible way, allowing the caller to be notified immediately that something is wrong. In our context here, we would not immediately raise the RemoteException (at least not right away), we would go onto the next service. We would fail-fast if there are no services available. In association terminology, the association would be broken.
With a fail-safe approach, we would want to constrain the notification that no services are available (as opposed to failing fast by throwing a RemoteException) in a managed way having the association proxy retry for a certain period of time. In this case the fail-safe mode would have the caller blocking on the service's remote method invocation until a service becomes available, or until the retry logic exhausts itself. If the retries are exhausted and there are still no available services, a RemoteException will be thrown.
On Feb 16, 2010, at 1126AM, Christopher Dolan wrote:
> Tom's technique below is nice for automatic failover. Just for a point
> of reference, I'll describe the hot-standby approach we've taken at
> Avid.
>
> * Declare that your IFooService is an IRedundantService
> * The IFooService instances in the group discover each other and hold
> an election
> * The winner calls joinMgr.modifyAttributes() to mark itself as primary
> * The IFooService instances ping each other and if any one stops
> responding, another election is triggered
> * Clients search for the IFooService that is marked primary=true and
> talk only to it
>
> Limitations of this approach:
> * need for IFooService instances to inter-communicate
> * moments during election when there is no primary
> * rare partial-disconnect conditions where you get two primaries
>
> Wins of this approach:
> * election protocol is entirely up to the IFooServices, no client logic
> except to trust the Entry that says primary=true
> * only one primary at a time
> * the IFooService instances are already in contact, so adding state
> synchronization requires little extra infrastructure
>
> Good-and-bad aspects:
> * clients can decide how to handle the no-primary case themselves.
> More code, but more control too.
>
>
> I'd love to hear comments/criticisms of our approach, and if anyone has
> implemented something similar.
>
> Chris
>
> -----Original Message-----
> From: Tom Hobbs [mailto:tvhobbs@googlemail.com]
> Sent: Tuesday, February 09, 2010 4:21 AM
> To: river-dev@incubator.apache.org
> Subject: Service Wrapper Example
>
> Hi,
>
> I mentioned in another thread that I had come across code which provides
> service fail over and auto-rediscovery. I've posted details of the kind
> of
> code that was used to (note this has been reinvented in my head just now
> and
> only loosely tested);
>
> http://wiki.apache.org/river/AutomaticServiceReplacement
>
> I hope that someone finds it useful and/or interesting.
>
> It's important to note that in the interests of simplicity the
> ServiceWrapper class in the article is explicitly linked to the dummy
> service described. Obviously using more reflection magic it's possible
> to
> remove this linkage so that ServiceWrapper can wrap any service you like
> and
> it's invoke method would invoke the method supplied to it, rather than
> the
> only method available in the dummy article.
>
> Thanks to Jukka for his Wiki account suggestion.
>
> Enjoy,
>
> Tom
RE: Service Wrapper Example
Posted by Christopher Dolan <ch...@avid.com>.
Tom's technique below is nice for automatic failover. Just for a point
of reference, I'll describe the hot-standby approach we've taken at
Avid.
* Declare that your IFooService is an IRedundantService
* The IFooService instances in the group discover each other and hold
an election
* The winner calls joinMgr.modifyAttributes() to mark itself as primary
* The IFooService instances ping each other and if any one stops
responding, another election is triggered
* Clients search for the IFooService that is marked primary=true and
talk only to it
Limitations of this approach:
* need for IFooService instances to inter-communicate
* moments during election when there is no primary
* rare partial-disconnect conditions where you get two primaries
Wins of this approach:
* election protocol is entirely up to the IFooServices, no client logic
except to trust the Entry that says primary=true
* only one primary at a time
* the IFooService instances are already in contact, so adding state
synchronization requires little extra infrastructure
Good-and-bad aspects:
* clients can decide how to handle the no-primary case themselves.
More code, but more control too.
I'd love to hear comments/criticisms of our approach, and if anyone has
implemented something similar.
Chris
-----Original Message-----
From: Tom Hobbs [mailto:tvhobbs@googlemail.com]
Sent: Tuesday, February 09, 2010 4:21 AM
To: river-dev@incubator.apache.org
Subject: Service Wrapper Example
Hi,
I mentioned in another thread that I had come across code which provides
service fail over and auto-rediscovery. I've posted details of the kind
of
code that was used to (note this has been reinvented in my head just now
and
only loosely tested);
http://wiki.apache.org/river/AutomaticServiceReplacement
I hope that someone finds it useful and/or interesting.
It's important to note that in the interests of simplicity the
ServiceWrapper class in the article is explicitly linked to the dummy
service described. Obviously using more reflection magic it's possible
to
remove this linkage so that ServiceWrapper can wrap any service you like
and
it's invoke method would invoke the method supplied to it, rather than
the
only method available in the dummy article.
Thanks to Jukka for his Wiki account suggestion.
Enjoy,
Tom