You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@geronimo.apache.org by Gianny Damour <gi...@optusnet.com.au> on 2008/01/26 04:32:10 UTC
Clustering of SFSB - Heads-up
Hello,
David B. kindly reminded me I should mention the SFSB clustering work
I am currently working on on the OpenEJB dev. list - so far, I only
mentioned this work on the Geronimo dev. list.
Firstly, apology for only mentioning this work now.
Now, let me talk a little bit of this work: I will soon add support
for the clustering of SFSB. By now, this clustering is only available
with OpenEJB embedded within Geronimo. It should be easy to have this
feature available with the standalone OpenEJB container by moving
across an upcoming Geronimo sub-project to a new OpenEJB sub-project.
For the history, my initial attempt had clustering classes in openejb-
core and I discovered these clustering classes should be packaged in
a separate JAR for class loading reasons - it was not such a big
surprise.
The clustering of SFSB is based on the same approach than for the
clustering of Jetty or Tomcat Web-applications over WADI: we add a
clustered SFSB container. When an EJB invocation enters this
container, it traverses a stack of WADI Contextualisers whose
responsibility is to locate the SFSB being invoked and ensure that
the invoked SFSB instance is available locally, i.e. in this JVM by
the SFSB instance manager. When the SFSB being invoked is on a remote
clustered SFSB container, it is first passivated on this remote
container and then activated on the container processing the EJB
invocation.
When an EJB client initially performs a remote JNDI lookup for a SFSB
deployed to the clustered SFSB container, it connects to the
configured Context.PROVIDER_URL. In the case of a successful lookup,
the JNDI response contains the locations, e.g. [ejbd://localhost:
4201, ejbd://localhost:4202], of the clustered SFSB containers
running the looked up SFSB. Based on these locations, the client
knows where the SFSB is currently running and, hence, it can fail-
over to another clustered SFSB container if the "primary" container
becomes unavailable. On each EJB invocations, an updated list of
locations may be returned with EJB responses if client locations
differ from server-side ones. The server knows that it needs to
provide an updated list of locations based on a non collision free
integer derived from client locations which is attached to each EJB
invocations.
Clustered SFSB instances are replicated as per the same approach than
for HTTPSessions in the case of Jetty or Tomcat Web-applications:
configure the number of replicas to be maintained. As for
HTTPSessions, it is possible to only replicate field level changes by
leveraging the wadi-aop module.
I will ping the list once again when my integration tests are
completed. Meanwhile, if there are any concerns, then please feel
free to raise them.
Thanks,
Gianny
Re: Clustering of SFSB - Heads-up
Posted by Mohammad Nour El-Din <no...@gmail.com>.
Sounds gr8 :), can you tell me when this feature is going to be available in
the stand-alone version, cause I want to play with it and I want to include
these info in a presentation I make.
On Jan 26, 2008 5:32 AM, Gianny Damour <gi...@optusnet.com.au>
wrote:
> Hello,
>
> David B. kindly reminded me I should mention the SFSB clustering work
> I am currently working on on the OpenEJB dev. list - so far, I only
> mentioned this work on the Geronimo dev. list.
>
> Firstly, apology for only mentioning this work now.
>
> Now, let me talk a little bit of this work: I will soon add support
> for the clustering of SFSB. By now, this clustering is only available
> with OpenEJB embedded within Geronimo. It should be easy to have this
> feature available with the standalone OpenEJB container by moving
> across an upcoming Geronimo sub-project to a new OpenEJB sub-project.
> For the history, my initial attempt had clustering classes in openejb-
> core and I discovered these clustering classes should be packaged in
> a separate JAR for class loading reasons - it was not such a big
> surprise.
>
> The clustering of SFSB is based on the same approach than for the
> clustering of Jetty or Tomcat Web-applications over WADI: we add a
> clustered SFSB container. When an EJB invocation enters this
> container, it traverses a stack of WADI Contextualisers whose
> responsibility is to locate the SFSB being invoked and ensure that
> the invoked SFSB instance is available locally, i.e. in this JVM by
> the SFSB instance manager. When the SFSB being invoked is on a remote
> clustered SFSB container, it is first passivated on this remote
> container and then activated on the container processing the EJB
> invocation.
>
> When an EJB client initially performs a remote JNDI lookup for a SFSB
> deployed to the clustered SFSB container, it connects to the
> configured Context.PROVIDER_URL. In the case of a successful lookup,
> the JNDI response contains the locations, e.g. [ejbd://localhost:
> 4201, ejbd://localhost:4202], of the clustered SFSB containers
> running the looked up SFSB. Based on these locations, the client
> knows where the SFSB is currently running and, hence, it can fail-
> over to another clustered SFSB container if the "primary" container
> becomes unavailable. On each EJB invocations, an updated list of
> locations may be returned with EJB responses if client locations
> differ from server-side ones. The server knows that it needs to
> provide an updated list of locations based on a non collision free
> integer derived from client locations which is attached to each EJB
> invocations.
>
> Clustered SFSB instances are replicated as per the same approach than
> for HTTPSessions in the case of Jetty or Tomcat Web-applications:
> configure the number of replicas to be maintained. As for
> HTTPSessions, it is possible to only replicate field level changes by
> leveraging the wadi-aop module.
>
> I will ping the list once again when my integration tests are
> completed. Meanwhile, if there are any concerns, then please feel
> free to raise them.
>
> Thanks,
> Gianny
>
>
--
Thanks
- Mohammad Nour
Re: Clustering of SFSB - Heads-up
Posted by Gianny Damour <gi...@optusnet.com.au>.
Hi,
Sorry for this belated response. Now that we have some code I think
we are in a better position to continue this discussion.
On 05/02/2008, at 11:19 AM, David Blevins wrote:
> On Jan 27, 2008, at 6:37 PM, Gianny Damour wrote:
>> I will add a sub-project plugins/openejb/geronimo-openejb-
>> clustering-wadi to Geronimo. In this sub--project is defined a
>> "Default Clustered Stateful Container" container, which depends on
>> the Geronimo Clustering API and WADI. The idea is to move this
>> project from G. to OpenEJB. When I say move, I mean remove it from
>> the Geronimo repository, repackage it in the OpenEJB SVN tree and
>> update accordingly Geronimo following the relocation.
>
> Cool, we'll have to figure something out here. In 3.x we fixed the
> circular dependency that we used to have between Geronimo and
> OpenEJB and definitely don't want to readd it. Should be simple to
> hack up interfaces in OpenEJB and implement them in the Geronimo/
> OpenEJB integration code that lives in Geronimo. We can deal with
> that later.
geronimo-openejb-clustering-wadi defines some classes in
org.apache.geronimo.openejb.cluster.stateful.container which can be
moved to OpenEJB. These classes must not be defined within openejb-
core as they need to be loaded by a distinct CL than the CL importing
the openejb-core classes.
I believe that the Geronimo changes to accommodate this relocation are:
1. update GBean DefaultClusteredStatefulContainer of configuration
openejb-clustering-wadi to reflect the new class name; and
2. update GBean OpenEJBClusteringBuilder of configuration openejb-
clustering-builder-wadi to use the new container id for the clustered
SFSB container.
>
> When creating the StatefulInstanceManager we were sort of aiming at
> clustering, but it's grown in complexity such that I doubt anyone
> would want to claim responsibility for that whole piece anymore
> just to claim activation and passivation as their own. Once we get
> rolling we might want to take a look at rebalancing the
> StatefulContainer and StatefulInstanceManager so you don't have to
> do so much subclassing.
i agree; as a matter of fact, the sub-classing approach was hard to
test as proven by ClusteredStatefulInstanceManagerTest which relies
on a "RMock intercept" mocking style to say
testInboundSessionMigrationActivateAndPoolBeanEntry.
>
>>> Did you tackle the affinity between a stateful bean holding onto
>>> an entitymanager with EXTENDED persistence context?
>>
>> Not yet; and you will have to explain that to me.
>
> Essentially, when a stateful bean has a reference to a JPA
> persistence unit using the scope PersistenceContext.EXTENDED (as
> opposed to PersistenceContext.TRANSACTION), the EntityManager is
> more or less cached "in" the stateful bean (so to speak).
> Persistent objects created through the "EXTENDED" EntityManager in
> the stateful bean do not detach around transactions as they would
> with a "TRANSACTION" EntityManager -- instead the persistence
> objects stay attached as long as the stateful bean lives. If the
> stateful bean creates another stateful bean, they share the same
> EntityManager, thus propogating the extended persistence context.
Thanks for this explanation.
>
> What I'm not sure of is if there are any activation/passivation
> rules associated with entity managers (specifically regarding
> "EXTENDED"). The transaction scenario should be fairly easy as
> currently the container won't call passivate on an instance in a
> transaction -- at least that's the way it works in the vanilla
> stateful container we have now.
I think that an approach would be to put the persistent objects
created by SFSBs using EXTENDED persistence in the same
o~.a~.g~.c~.Session holding the SFSB instance.
When a clustered SFSB is created, it is put into a ClusteredBeanEntry
which is in turn put into a Session. The state of this Session is
migrated and replicated by the clustering framework. So, if there is
a way to receive notifications when persistent objects "attached" to
a SFSB are created, then they could be put into the SFSB Session.
This may be easier to say than to do, though.
>>>
>>> Also the current stateful container can't actually passivate/
>>> activate ejb3 stateful beans as we haven't gotten around to
>>> writing the code that deals with them not implementing
>>> serializable. Wonder if you found a solution for that.
>>
>> Not yet. I am doing my testing with the simple SFSB example,
>> CounterImp, and I had to update its implementation so that it
>> implements Serializable. I believe wadi-aop will help here.
>> Currently, wadi-aop relies on serialization for the "initial"
>> replication of POJOs. However, this dependency can be removed if
>> need be.
>
> What does wadi-aop do that removes the need to implement
> serializable? Curious on the technique, we might steal it :)
> Though if it's byte code modification, we'd probably have to do
> something else. (byte code modding is cool for optional things
> like clustering or jpa)
wadi-aop visits object graphs via reflection and records the
reflective operations to be executed to re-instantiate such an object
graph from scratch by solely using its primitives fields. The AOP
stuff is there to track which fields or methods have been updated or
executed between two object graph walks.
Also, from a Developer perspective, I believe it is acceptable to be
required to implement Serializable types if I want them to be moved
in and out of the JVM.
If you disagree, then this is fine :) After all, it is quite easy to
implement from scratch an object graph serializer which does not
mandate Serializable - as long as performance is not a concerns.
>>
>> I extracted the logic you are talking about behind this contract:
>> public interface ConnectionFactoryStrategy extends Serializable {
>> Connection connect(URI[] locations, Request request) throws
>> RemoteException;
>> }
>>
>> This contract allows an implementation to decide how to fail-over
>> and route requests.
>
> Yea, the ConnectionFactory interface was originally "public
> Connection getConnection(ServerMetaData serverMetaData) throws
> java.io.IOException;" It might be better to revert back to that
> and let the implementor choose how they'd like to handle the URIs
> inside the serverMetaData.
Currently, the ConnectionFactoryStrategy is hard-coded in
ServerMetaData to StickToLastServerConnectionFactoryStrategy. I
believe the strategy should be configured at the EJB level within the
OpenEJB DD.
>> All the SFSB instances for a given EJB module are part of the same
>> Geronimo SessionManager, or WADI ServiceSpace.
> [...]
>> The list of locations is for a given EJB module and hence for
>> multiple SFSBs. This can be refined if need be. As long as all the
>> SFSB defined by a given EJB module are running, this approach
>> works fine. Obviously, when an EJB module is partially running
>> this is a problem.
>
> When you say running, you mean that there are DeploymentInfo
> objects in existence for the stateful beans in question? Or that
> there are actual stateful bean instances running?
I improved at the very last moment the way location tracking is
working: tracking is now done at the deployment level and no more at
the EJB module level which solves the "when an EJB module is
partially running this is a problem" concern.
When I say running, I mean that the deployment is running. There may
or may not have SFSB instances for the deployment. The tracking of
deployment locations is in NetworkConnectorMonitor. I am aware of a
race condition bug happening when two deployments are starting at
nearly the same time which causes locations to be incorrect for the
node running the deployment which has started last. I will fix that;
meanwhile, if you observe locations to be incorrect, then simply
restart the deployment on the node defining the wrong locations.
>
> So if the clustering is at a module level and there is only one
> module in the server, is there really any affect of the clustering
> support as it stands now? It's fine if the answer is no as it's
> still a really good start. If you can group and move things by
> module, you're really just one step away from further subgrouping
> by client and module. Obviously there's no distinct client ID now,
> but this would be one of the big advantages to wrapping the
> protocol. You could create client session ids, negotiate the
> server meta data, do redirects all in the clustering protocol layer.
I do not understand the question; so, I do not know if the response
is yes or no. From an user perspective, what are the features you are
thinking about?
>
> All sounds like a really good start. It's such a hard problem that
> having any code working at all is a big win in itself. Making it
> "fit" better, etc. is way easier once there's something there to
> refactor. Looking forward to working with you on it :)
I am also looking forward to working with you on this integration to
make it better!
Thanks,
Gianny
Re: Clustering of SFSB - Heads-up
Posted by David Blevins <da...@visi.com>.
On Feb 4, 2008, at 4:19 PM, David Blevins wrote:
>
> On Jan 27, 2008, at 6:37 PM, Gianny Damour wrote:
>
>> I extracted the logic you are talking about behind this contract:
>> public interface ConnectionFactoryStrategy extends Serializable {
>> Connection connect(URI[] locations, Request request) throws
>> RemoteException;
>> }
>>
>> This contract allows an implementation to decide how to fail-over
>> and route requests.
>
> Yea, the ConnectionFactory interface was originally "public
> Connection getConnection(ServerMetaData serverMetaData) throws
> java.io.IOException;" It might be better to revert back to that and
> let the implementor choose how they'd like to handle the URIs inside
> the serverMetaData.
Changed this up a bit. ConnectionFactoryStrategy is now essentially
what ConnectionFactory used to be when it took ServerMetaData as the
only arg. Also, the org.apache.openejb.client.Client class once again
uses the ConnectionManager to get a connection. ConnectionManager in
turn delegates to the ConnectionStrategy named in the ServerMetaData
or the default if none is mentioned. The ConnectionStrategy in turn
will likely invoke several possible ConnectionFactories in its attempt
to get a connection.
public interface ConnectionStrategy {
public Connection connect(ServerMetaData server) throws
IOException;
}
public interface ConnectionFactory {
public Connection getConnection(URI uri) throws
java.io.IOException;
}
The ConnectionManager serves as a place to register use the
ConnectionStrategy in addition to the ConnectionFactory as before.
ConnectionStrategy(s) and ConnectionFactory(s) are now equally
pluggable. New implementations can be included in the classpath and
we'll pick them up automagically or you can register/unregister one
explicitly in code.
I looked for implementations of ConnectionFactoryStrategy other than
the stick to last server one. Let me know if there are any I should
update.
Check out the code and let me know what you think.
-David
Re: Clustering of SFSB - Heads-up
Posted by David Blevins <da...@visi.com>.
On Jan 27, 2008, at 6:37 PM, Gianny Damour wrote:
> Hello David,
>
> On 28/01/2008, at 11:23 AM, David Blevins wrote:
>
>> Thanks, Gianny! Hopefully you can help me catch up :) Sounds
>> exciting.
>>
>> On Jan 25, 2008, at 7:32 PM, Gianny Damour wrote:
>>
>>> It should be easy to have this feature available with the
>>> standalone OpenEJB container by moving across an upcoming Geronimo
>>> sub-project to a new OpenEJB sub-project.
>>
>> Elaborate on this if you can.
>
> I will add a sub-project plugins/openejb/geronimo-openejb-clustering-
> wadi to Geronimo. In this sub--project is defined a "Default
> Clustered Stateful Container" container, which depends on the
> Geronimo Clustering API and WADI. The idea is to move this project
> from G. to OpenEJB. When I say move, I mean remove it from the
> Geronimo repository, repackage it in the OpenEJB SVN tree and update
> accordingly Geronimo following the relocation.
Cool, we'll have to figure something out here. In 3.x we fixed the
circular dependency that we used to have between Geronimo and OpenEJB
and definitely don't want to readd it. Should be simple to hack up
interfaces in OpenEJB and implement them in the Geronimo/OpenEJB
integration code that lives in Geronimo. We can deal with that later.
>> What direction did you go with implementing the container, modify/
>> extending the existing one, copy/paste reuse, or new one from
>> scratch? I assume when you say stateless container you mean an
>> impl of o.a.o.RpcContainer.
>
> I simply extracted a couple of methods, e.g. activation and
> passivation methods, of StatefulContainer and
> StatefulInstanceManager so that I can sub-class these two classes
> and hook in the Geronimo clustering API and WADI. At a very high
> level, here are what these sub-classes provide:
> * activation and passivation of SFSB instances when they are
> migrated from one container to another one;
> * intercept businessMethod and removeEJBObject methods so that a
> stack of WADI Contextualiser is traversed before the actual
> execution of these two methods;
> * listen to SFSB migration events in order to maintain the
> StatefulInstanceManager up-to-date; and
> * listen to distributed EJB Module start/stop events so that a
> container knows where a given EJB module is running on.
When creating the StatefulInstanceManager we were sort of aiming at
clustering, but it's grown in complexity such that I doubt anyone
would want to claim responsibility for that whole piece anymore just
to claim activation and passivation as their own. Once we get rolling
we might want to take a look at rebalancing the StatefulContainer and
StatefulInstanceManager so you don't have to do so much subclassing.
>> Did you tackle the affinity between a stateful bean holding onto an
>> entitymanager with EXTENDED persistence context?
>
> Not yet; and you will have to explain that to me.
Essentially, when a stateful bean has a reference to a JPA persistence
unit using the scope PersistenceContext.EXTENDED (as opposed to
PersistenceContext.TRANSACTION), the EntityManager is more or less
cached "in" the stateful bean (so to speak). Persistent objects
created through the "EXTENDED" EntityManager in the stateful bean do
not detach around transactions as they would with a "TRANSACTION"
EntityManager -- instead the persistence objects stay attached as long
as the stateful bean lives. If the stateful bean creates another
stateful bean, they share the same EntityManager, thus propogating the
extended persistence context.
What I'm not sure of is if there are any activation/passivation rules
associated with entity managers (specifically regarding "EXTENDED").
The transaction scenario should be fairly easy as currently the
container won't call passivate on an instance in a transaction -- at
least that's the way it works in the vanilla stateful container we
have now.
>>
>> Also the current stateful container can't actually passivate/
>> activate ejb3 stateful beans as we haven't gotten around to writing
>> the code that deals with them not implementing serializable.
>> Wonder if you found a solution for that.
>
> Not yet. I am doing my testing with the simple SFSB example,
> CounterImp, and I had to update its implementation so that it
> implements Serializable. I believe wadi-aop will help here.
> Currently, wadi-aop relies on serialization for the "initial"
> replication of POJOs. However, this dependency can be removed if
> need be.
What does wadi-aop do that removes the need to implement
serializable? Curious on the technique, we might steal it :) Though
if it's byte code modification, we'd probably have to do something
else. (byte code modding is cool for optional things like clustering
or jpa)
>> Sounds like you'd need to extend the EJB protocol to support that
>> (obviously :) and then some pluggable way to get the list of
>> locations. What kind of API did you come up with and how is it
>> plugged in? I suppose it might be possible to wrap the ejbd
>> protocol like we do when we wrap it with http, you still might need
>> some plug points.
>
> The list of locations is obtained from the clustered RPC container,
> whose interface is:
> public interface ClusteredRPCContainer extends RpcContainer {
> URI[] getLocations(DeploymentInfo info);
> }
>
> JNDIResponse and EJBResponse now implement this interface:
> public interface ClusterableResponse extends Response {
> void setServer(ServerMetaData server);
> }
>
> EjbRequestHandler uses a ClusterableRequestHandler implementation to
> update or not the list of locations to be returned with JNDIResponse
> and EJBResponse when need be. The API is:
> public interface ClusterableRequestHandler {
> void updateServer(DeploymentInfo deploymentInfo,
> ClusterableRequest req, ClusterableResponse res);
> }
This looks very workable as a first start. Later, we can probably
refactor things a bit. It should be possible to wrap the existing
protocol so you could have a quite robust protocol layer of your own
underneath. For example, we implement EJBd over HTTP by simply
writing first the HTTP info in one layer, then writing the EJB data.
On the receiving side, we have a HTTP server that reads in the HTTP
data, then passes the stream to the EJBd server who reads of the EJBd
request. Might be easier said than done though :)
>> The client package already sort of has the ability to connect to a
>> list of URLs (tries them in order), so would this magically also
>> result in failover for all ejb proxies? Is there one list for the
>> whole server, one list per stateful bean instance or something else
>> all together? I guess I'm assuming the list is the one that
>> eventually ends up in the ServerMetaData object, let me know if
>> that's not a safe assumption :)
>
> I extracted the logic you are talking about behind this contract:
> public interface ConnectionFactoryStrategy extends Serializable {
> Connection connect(URI[] locations, Request request) throws
> RemoteException;
> }
>
> This contract allows an implementation to decide how to fail-over
> and route requests.
Yea, the ConnectionFactory interface was originally "public Connection
getConnection(ServerMetaData serverMetaData) throws
java.io.IOException;" It might be better to revert back to that and
let the implementor choose how they'd like to handle the URIs inside
the serverMetaData.
>>> Clustered SFSB instances are replicated as per the same approach
>>> than for HTTPSessions in the case of Jetty or Tomcat Web-
>>> applications: configure the number of replicas to be maintained.
>>> As for HTTPSessions, it is possible to only replicate field level
>>> changes by leveraging the wadi-aop module.
>>
>> So with HTTP sessions, there's one per app and it's also tied to a
>> user. In EJB there's potentially several per app and their not
>> directly identifiable by a user (each has it's own id). Lot's of
>> ways to go with that, which way did you go? :) One could manage
>> ejb session independently perhaps resulting in "sprawl". Maybe
>> group them toghether by app. Maybe find some way to give the
>> client an id and group them by client (and potentially app as well,
>> though a client could access multiple "apps"). And possibly many
>> more ways I haven't yet thought of....
>
> All the SFSB instances for a given EJB module are part of the same
> Geronimo SessionManager, or WADI ServiceSpace.
[...]
> The list of locations is for a given EJB module and hence for
> multiple SFSBs. This can be refined if need be. As long as all the
> SFSB defined by a given EJB module are running, this approach works
> fine. Obviously, when an EJB module is partially running this is a
> problem.
When you say running, you mean that there are DeploymentInfo objects
in existence for the stateful beans in question? Or that there are
actual stateful bean instances running?
So if the clustering is at a module level and there is only one module
in the server, is there really any affect of the clustering support as
it stands now? It's fine if the answer is no as it's still a really
good start. If you can group and move things by module, you're really
just one step away from further subgrouping by client and module.
Obviously there's no distinct client ID now, but this would be one of
the big advantages to wrapping the protocol. You could create client
session ids, negotiate the server meta data, do redirects all in the
clustering protocol layer.
All sounds like a really good start. It's such a hard problem that
having any code working at all is a big win in itself. Making it
"fit" better, etc. is way easier once there's something there to
refactor. Looking forward to working with you on it :)
Great work!
-David
Re: Clustering of SFSB - Heads-up
Posted by Gianny Damour <gi...@optusnet.com.au>.
Hello David,
On 28/01/2008, at 11:23 AM, David Blevins wrote:
> Thanks, Gianny! Hopefully you can help me catch up :) Sounds
> exciting.
>
> On Jan 25, 2008, at 7:32 PM, Gianny Damour wrote:
>
>> It should be easy to have this feature available with the
>> standalone OpenEJB container by moving across an upcoming Geronimo
>> sub-project to a new OpenEJB sub-project.
>
> Elaborate on this if you can.
I will add a sub-project plugins/openejb/geronimo-openejb-clustering-
wadi to Geronimo. In this sub--project is defined a "Default
Clustered Stateful Container" container, which depends on the
Geronimo Clustering API and WADI. The idea is to move this project
from G. to OpenEJB. When I say move, I mean remove it from the
Geronimo repository, repackage it in the OpenEJB SVN tree and update
accordingly Geronimo following the relocation.
>
>> The clustering of SFSB is based on the same approach than for the
>> clustering of Jetty or Tomcat Web-applications over WADI: we add a
>> clustered SFSB container. When an EJB invocation enters this
>> container, it traverses a stack of WADI Contextualisers whose
>> responsibility is to locate the SFSB being invoked and ensure that
>> the invoked SFSB instance is available locally, i.e. in this JVM
>> by the SFSB instance manager. When the SFSB being invoked is on a
>> remote clustered SFSB container, it is first passivated on this
>> remote container and then activated on the container processing
>> the EJB invocation.
>
> What direction did you go with implementing the container, modify/
> extending the existing one, copy/paste reuse, or new one from
> scratch? I assume when you say stateless container you mean an
> impl of o.a.o.RpcContainer.
I simply extracted a couple of methods, e.g. activation and
passivation methods, of StatefulContainer and StatefulInstanceManager
so that I can sub-class these two classes and hook in the Geronimo
clustering API and WADI. At a very high level, here are what these
sub-classes provide:
* activation and passivation of SFSB instances when they are migrated
from one container to another one;
* intercept businessMethod and removeEJBObject methods so that a
stack of WADI Contextualiser is traversed before the actual execution
of these two methods;
* listen to SFSB migration events in order to maintain the
StatefulInstanceManager up-to-date; and
* listen to distributed EJB Module start/stop events so that a
container knows where a given EJB module is running on.
>
> Did you tackle the affinity between a stateful bean holding onto an
> entitymanager with EXTENDED persistence context?
Not yet; and you will have to explain that to me.
At this stage, I am only aware of transactions spanning multiple
invocations of the same SFSB instance. Somehow, I need to know that a
given SFSB instance is currently enlisted in a Tx and hence should
not be migrated to another container. I will look at this problem
after this first spike of a clustering solution over WADI. I do not
know if it will be useful, however it is possible to configure how
much time to wait for an exclusive resource - in this context SFSB
instance - lock prior to throw a resource busy exception to a client
trying to access the resource from a remote node. I believe that by
setting this timeout to a value greater than the timeout of the Tx a
to be migrated SFSB instance is a solution.
>
> Also the current stateful container can't actually passivate/
> activate ejb3 stateful beans as we haven't gotten around to writing
> the code that deals with them not implementing serializable.
> Wonder if you found a solution for that.
Not yet. I am doing my testing with the simple SFSB example,
CounterImp, and I had to update its implementation so that it
implements Serializable. I believe wadi-aop will help here.
Currently, wadi-aop relies on serialization for the "initial"
replication of POJOs. However, this dependency can be removed if need
be.
>
>> When an EJB client initially performs a remote JNDI lookup for a
>> SFSB deployed to the clustered SFSB container, it connects to the
>> configured Context.PROVIDER_URL. In the case of a successful
>> lookup, the JNDI response contains the locations, e.g. [ejbd://
>> localhost:4201, ejbd://localhost:4202], of the clustered SFSB
>> containers running the looked up SFSB. Based on these locations,
>> the client knows where the SFSB is currently running and, hence,
>> it can fail-over to another clustered SFSB container if the
>> "primary" container becomes unavailable. On each EJB invocations,
>> an updated list of locations may be returned with EJB responses if
>> client locations differ from server-side ones. The server knows
>> that it needs to provide an updated list of locations based on a
>> non collision free integer derived from client locations which is
>> attached to each EJB invocations.
>
> Sounds like you'd need to extend the EJB protocol to support that
> (obviously :) and then some pluggable way to get the list of
> locations. What kind of API did you come up with and how is it
> plugged in? I suppose it might be possible to wrap the ejbd
> protocol like we do when we wrap it with http, you still might need
> some plug points.
The list of locations is obtained from the clustered RPC container,
whose interface is:
public interface ClusteredRPCContainer extends RpcContainer {
URI[] getLocations(DeploymentInfo info);
}
JNDIResponse and EJBResponse now implement this interface:
public interface ClusterableResponse extends Response {
void setServer(ServerMetaData server);
}
EjbRequestHandler uses a ClusterableRequestHandler implementation to
update or not the list of locations to be returned with JNDIResponse
and EJBResponse when need be. The API is:
public interface ClusterableRequestHandler {
void updateServer(DeploymentInfo deploymentInfo,
ClusterableRequest req, ClusterableResponse res);
}
On the client-side, the ServerMetadata attached to JNDIResponse and
EJBResponse, if any, is used to update the current server definition.
>
> The client package already sort of has the ability to connect to a
> list of URLs (tries them in order), so would this magically also
> result in failover for all ejb proxies? Is there one list for the
> whole server, one list per stateful bean instance or something else
> all together? I guess I'm assuming the list is the one that
> eventually ends up in the ServerMetaData object, let me know if
> that's not a safe assumption :)
I extracted the logic you are talking about behind this contract:
public interface ConnectionFactoryStrategy extends Serializable {
Connection connect(URI[] locations, Request request) throws
RemoteException;
}
This contract allows an implementation to decide how to fail-over and
route requests.
The list of locations is for a given EJB module and hence for
multiple SFSBs. This can be refined if need be. As long as all the
SFSB defined by a given EJB module are running, this approach works
fine. Obviously, when an EJB module is partially running this is a
problem.
>
>> Clustered SFSB instances are replicated as per the same approach
>> than for HTTPSessions in the case of Jetty or Tomcat Web-
>> applications: configure the number of replicas to be maintained.
>> As for HTTPSessions, it is possible to only replicate field level
>> changes by leveraging the wadi-aop module.
>
> So with HTTP sessions, there's one per app and it's also tied to a
> user. In EJB there's potentially several per app and their not
> directly identifiable by a user (each has it's own id). Lot's of
> ways to go with that, which way did you go? :) One could manage
> ejb session independently perhaps resulting in "sprawl". Maybe
> group them toghether by app. Maybe find some way to give the
> client an id and group them by client (and potentially app as well,
> though a client could access multiple "apps"). And possibly many
> more ways I haven't yet thought of....
All the SFSB instances for a given EJB module are part of the same
Geronimo SessionManager, or WADI ServiceSpace.
Thanks for this first round!
Gianny
>
> That's it off the top of my head. Likely more questions when I get
> the first round back :)
>
>
> -David
>
Re: Clustering of SFSB - Heads-up
Posted by David Blevins <da...@visi.com>.
Thanks, Gianny! Hopefully you can help me catch up :) Sounds exciting.
On Jan 25, 2008, at 7:32 PM, Gianny Damour wrote:
> It should be easy to have this feature available with the standalone
> OpenEJB container by moving across an upcoming Geronimo sub-project
> to a new OpenEJB sub-project.
Elaborate on this if you can.
> The clustering of SFSB is based on the same approach than for the
> clustering of Jetty or Tomcat Web-applications over WADI: we add a
> clustered SFSB container. When an EJB invocation enters this
> container, it traverses a stack of WADI Contextualisers whose
> responsibility is to locate the SFSB being invoked and ensure that
> the invoked SFSB instance is available locally, i.e. in this JVM by
> the SFSB instance manager. When the SFSB being invoked is on a
> remote clustered SFSB container, it is first passivated on this
> remote container and then activated on the container processing the
> EJB invocation.
What direction did you go with implementing the container, modify/
extending the existing one, copy/paste reuse, or new one from
scratch? I assume when you say stateless container you mean an impl
of o.a.o.RpcContainer.
Did you tackle the affinity between a stateful bean holding onto an
entitymanager with EXTENDED persistence context?
Also the current stateful container can't actually passivate/activate
ejb3 stateful beans as we haven't gotten around to writing the code
that deals with them not implementing serializable. Wonder if you
found a solution for that.
> When an EJB client initially performs a remote JNDI lookup for a
> SFSB deployed to the clustered SFSB container, it connects to the
> configured Context.PROVIDER_URL. In the case of a successful lookup,
> the JNDI response contains the locations, e.g. [ejbd://localhost:
> 4201, ejbd://localhost:4202], of the clustered SFSB containers
> running the looked up SFSB. Based on these locations, the client
> knows where the SFSB is currently running and, hence, it can fail-
> over to another clustered SFSB container if the "primary" container
> becomes unavailable. On each EJB invocations, an updated list of
> locations may be returned with EJB responses if client locations
> differ from server-side ones. The server knows that it needs to
> provide an updated list of locations based on a non collision free
> integer derived from client locations which is attached to each EJB
> invocations.
Sounds like you'd need to extend the EJB protocol to support that
(obviously :) and then some pluggable way to get the list of
locations. What kind of API did you come up with and how is it
plugged in? I suppose it might be possible to wrap the ejbd protocol
like we do when we wrap it with http, you still might need some plug
points.
The client package already sort of has the ability to connect to a
list of URLs (tries them in order), so would this magically also
result in failover for all ejb proxies? Is there one list for the
whole server, one list per stateful bean instance or something else
all together? I guess I'm assuming the list is the one that
eventually ends up in the ServerMetaData object, let me know if that's
not a safe assumption :)
> Clustered SFSB instances are replicated as per the same approach
> than for HTTPSessions in the case of Jetty or Tomcat Web-
> applications: configure the number of replicas to be maintained. As
> for HTTPSessions, it is possible to only replicate field level
> changes by leveraging the wadi-aop module.
So with HTTP sessions, there's one per app and it's also tied to a
user. In EJB there's potentially several per app and their not
directly identifiable by a user (each has it's own id). Lot's of ways
to go with that, which way did you go? :) One could manage ejb
session independently perhaps resulting in "sprawl". Maybe group them
toghether by app. Maybe find some way to give the client an id and
group them by client (and potentially app as well, though a client
could access multiple "apps"). And possibly many more ways I haven't
yet thought of....
That's it off the top of my head. Likely more questions when I get
the first round back :)
-David