You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@servicemix.apache.org by Pavan Sheel <pa...@yahoo.com> on 2007/04/04 21:10:33 UTC
Non-removal of jbiendpoints from routing table
Folks , I would like to describe a scenario + problem and ask for
suggestions -
We are developing a 24x7 operation for running an enterprise application.
This system is architected as a cluster of many smaller service apps (normal
j2ee apps).
We intend to use servicemix (SMX) as a communication fabric between these
service apps.
Each service app embeds a SMX container , with all containers part of an SMX
cluster (via JMS flow).
Each service app exposes a specific set of jbiendpoints (configured such
that the jbiendpoint names remain constant across restarts).
There will be multiple instances of a given service app , therefore multiple
consumers for it's set of jbiendpoints. This allows activeMQ to loadbalance
smx requests across the instances.
FACT - During 24x7 operation , there will be situations where all instances
of a specific service app will be permanently undeployed.
REQUIREMENT - Clients which now attempt to send messages to any of these
jbiendpoints , should receive an exception.
ISSUE - SMX containers do not remove entries from their routing table when
they get an advisory saying a specific container is shutting down.
Now clients can send messages , but will never get a response.
ISSUE - SMX containers do not update their routing tables via heartbeats
(which would detect container failure , and therefore remove endpoints from
routing tables).
If an smx container fails , no shutting down' broadcast is sent , & clients
clients which send messages will never get a response.
Any suggestions ?
PS - I have included below a snippet on this topic, from a thread in another
forum.
regards
Pavan Bhatnagar
Guillaume Nodet schrieb:
> That's the base asumption for ServiceMix clustering.
> A JBI endpoint is uniquely identified by its service QName and
> endpoint name.
> When two JBI endpoints are activated in two different nodes in the
> cluster
> (this is not possible in the same node), it is considered as the same
> JBI endpoint,
> so that requests will be load-balanced between both nodes in the cluster.
> That's the reason why a single JBI endpoint may have a local
> implementation
> and 0 or more remote implementations.
>
> When a node goes down, the jms and jca flows receive the message
> RemoveInfo.
> All remote endpoints for this node should be removed from the
> InternalEndpoint
> remote map.
>
> On 11/20/06, Christian Schneider <ch...@die-schneider.net> wrote:
>> I will try to create a patch but I need some more information to do it
>> right.
>>
>> Reading the sources I am not so sure what exactly service and endpoint
>> mean. Could you define the two terms for me or point me to some external
>> docs.
>> from the JBI spec I presumed the following:
>>
>> - a service would is an implementation of a service interface
>> - an endpoint is the concrete instance of a service that resides in a
>> given container
>>
>> In the sources an internal endpoint has a map of remotes. So it seems
>> that an endpoint is not really exactly one service instance?!
>>
>> Christian
>>
>>
>> Guillaume Nodet schrieb:
>> > The benefit of not removing the endpoints is that if the remote
>> > servicemix comes up later, the exchange will not fail. The problem
>> > is that we can not make the difference between a servicemix which
>> > is put down (because of a crash or temporary stop) and a service unit
>> > which is undeployed (the endpoint will not be available at a later
>> > time, until the SU is redeployed).
>> >
>> > I think this could be argued and the behavior may need
>> > to be chosen using a flag, so that endpoint can be removed
>> > from the registry. If you can provide a patch, i'd be glad to
>> > check it in.
>> >
>> > On 11/19/06, Christian Schneider <ch...@die-schneider.net> wrote:
>> >> I have the following setup that I am evaluating for a high
>> availability
>> >> scenario.
>> >>
>> >> - 1 Sender
>> >> - 2 Receivers
>> >>
>> >> each in separate servicemix containers connected by a jmsflow.
>> >>
>> >> When the receivers are running and the sender comes up it gets the
>> >> endpoints of the receivers. But when then one receiver goes down the
>> >> corresponding endpoint is not deleted. So when the sender later
>> chooses
>> >> the endpoint of the receiver that is down the message will be
>> still sent
>> >> to this container and the sender will hang in a sync send.
>> >>
>> >> I would expect that the endpoints of a container that goes down are
>> >> removed in the other containers in the cluster. I debuged the
>> handling
>> >> of the Advisory message in jmsflow.
>> >> The removeinfo message is received in the sender but there is no
>> code to
>> >> remove endpoints.
>> >>
>> >> Am I missing something here or is this only incomplete?
>> >>
>> >> Greetins
>> >>
>> >> Christian
>> >>
>> >>
>> >> Here is the code in jmsflow:
>> >>
>> >> Form JmsFlow / protected void onAdvisoryMessage(Object obj) {
>> >> } else if (obj instanceof RemoveInfo) {
>> >> ConsumerId id = (ConsumerId) ((RemoveInfo)
>> >> obj).getObjectId();
>> >> subscriberSet.remove(id.getConnectionId());
>> >> removeAllPackets(id.getConnectionId());
>> >> }
>> >>
>> >> --
>> >>
>> >> Christian Schneider
>> >> ---
>> >> http://www.liquid-reality.de
>> >>
>> >>
>> >
>> >
>>
>>
>> --
>>
>> Christian Schneider
>> ---
>> http://www.liquid-reality.de
>>
>>
>
>
--
View this message in context: http://www.nabble.com/Non-removal-of-jbiendpoints-from-routing-table-tf3527705s12049.html#a9843643
Sent from the ServiceMix - User mailing list archive at Nabble.com.
Re: Non-removal of jbiendpoints from routing table
Posted by Guillaume Nodet <gn...@gmail.com>.
I think the main problem is that we need to find a way
to tell wether an endoint has been undeployed or just stopped
(because of a crash or shutdown for example). There is no
way to really know that because endpoints are deactivated when
the SU is stopped, which always happen before undeploying the
SU. In addition, endpoints are not easily linked to a SU
(the container can not really know that an endpoint has been
activated as part of a SU deployment). I think we might find
some acceptable way to do that, if we assume that components
activate / deactivate endpoints in the same thread that start / stop
the SU. If we can link an endpoint to its SU, it should be easier
to tell that the endpoint is down temporary (shutdown) or
definitely (SU undeployment).
You should raise a JIRA, and if you need any help to work
on this, i''d be glad to give you any needed informations so that
you can provide a patch.
On 4/4/07, Pavan Sheel <pa...@yahoo.com> wrote:
>
>
>
> Folks , I would like to describe a scenario + problem and ask for
> suggestions -
>
> We are developing a 24x7 operation for running an enterprise application.
> This system is architected as a cluster of many smaller service apps
> (normal
> j2ee apps).
> We intend to use servicemix (SMX) as a communication fabric between these
> service apps.
> Each service app embeds a SMX container , with all containers part of an
> SMX
> cluster (via JMS flow).
> Each service app exposes a specific set of jbiendpoints (configured such
> that the jbiendpoint names remain constant across restarts).
> There will be multiple instances of a given service app , therefore
> multiple
> consumers for it's set of jbiendpoints. This allows activeMQ to
> loadbalance
> smx requests across the instances.
>
>
> FACT - During 24x7 operation , there will be situations where all
> instances
> of a specific service app will be permanently undeployed.
>
> REQUIREMENT - Clients which now attempt to send messages to any of these
> jbiendpoints , should receive an exception.
>
> ISSUE - SMX containers do not remove entries from their routing table when
> they get an advisory saying a specific container is shutting down.
> Now clients can send messages , but will never get a response.
>
> ISSUE - SMX containers do not update their routing tables via heartbeats
> (which would detect container failure , and therefore remove endpoints
> from
> routing tables).
> If an smx container fails , no shutting down' broadcast is sent , &
> clients
> clients which send messages will never get a response.
>
> Any suggestions ?
>
> PS - I have included below a snippet on this topic, from a thread in
> another
> forum.
>
>
> regards
> Pavan Bhatnagar
>
>
> Guillaume Nodet schrieb:
> > That's the base asumption for ServiceMix clustering.
> > A JBI endpoint is uniquely identified by its service QName and
> > endpoint name.
> > When two JBI endpoints are activated in two different nodes in the
> > cluster
> > (this is not possible in the same node), it is considered as the same
> > JBI endpoint,
> > so that requests will be load-balanced between both nodes in the
> cluster.
> > That's the reason why a single JBI endpoint may have a local
> > implementation
> > and 0 or more remote implementations.
> >
> > When a node goes down, the jms and jca flows receive the message
> > RemoveInfo.
> > All remote endpoints for this node should be removed from the
> > InternalEndpoint
> > remote map.
> >
> > On 11/20/06, Christian Schneider <ch...@die-schneider.net> wrote:
> >> I will try to create a patch but I need some more information to do it
> >> right.
> >>
> >> Reading the sources I am not so sure what exactly service and endpoint
> >> mean. Could you define the two terms for me or point me to some
> external
> >> docs.
> >> from the JBI spec I presumed the following:
> >>
> >> - a service would is an implementation of a service interface
> >> - an endpoint is the concrete instance of a service that resides in a
> >> given container
> >>
> >> In the sources an internal endpoint has a map of remotes. So it seems
> >> that an endpoint is not really exactly one service instance?!
> >>
> >> Christian
> >>
> >>
> >> Guillaume Nodet schrieb:
> >> > The benefit of not removing the endpoints is that if the remote
> >> > servicemix comes up later, the exchange will not fail. The problem
> >> > is that we can not make the difference between a servicemix which
> >> > is put down (because of a crash or temporary stop) and a service unit
> >> > which is undeployed (the endpoint will not be available at a later
> >> > time, until the SU is redeployed).
> >> >
> >> > I think this could be argued and the behavior may need
> >> > to be chosen using a flag, so that endpoint can be removed
> >> > from the registry. If you can provide a patch, i'd be glad to
> >> > check it in.
> >> >
> >> > On 11/19/06, Christian Schneider <ch...@die-schneider.net> wrote:
> >> >> I have the following setup that I am evaluating for a high
> >> availability
> >> >> scenario.
> >> >>
> >> >> - 1 Sender
> >> >> - 2 Receivers
> >> >>
> >> >> each in separate servicemix containers connected by a jmsflow.
> >> >>
> >> >> When the receivers are running and the sender comes up it gets the
> >> >> endpoints of the receivers. But when then one receiver goes down the
> >> >> corresponding endpoint is not deleted. So when the sender later
> >> chooses
> >> >> the endpoint of the receiver that is down the message will be
> >> still sent
> >> >> to this container and the sender will hang in a sync send.
> >> >>
> >> >> I would expect that the endpoints of a container that goes down are
> >> >> removed in the other containers in the cluster. I debuged the
> >> handling
> >> >> of the Advisory message in jmsflow.
> >> >> The removeinfo message is received in the sender but there is no
> >> code to
> >> >> remove endpoints.
> >> >>
> >> >> Am I missing something here or is this only incomplete?
> >> >>
> >> >> Greetins
> >> >>
> >> >> Christian
> >> >>
> >> >>
> >> >> Here is the code in jmsflow:
> >> >>
> >> >> Form JmsFlow / protected void onAdvisoryMessage(Object obj) {
> >> >> } else if (obj instanceof RemoveInfo) {
> >> >> ConsumerId id = (ConsumerId) ((RemoveInfo)
> >> >> obj).getObjectId();
> >> >> subscriberSet.remove(id.getConnectionId());
> >> >> removeAllPackets(id.getConnectionId());
> >> >> }
> >> >>
> >> >> --
> >> >>
> >> >> Christian Schneider
> >> >> ---
> >> >> http://www.liquid-reality.de
> >> >>
> >> >>
> >> >
> >> >
> >>
> >>
> >> --
> >>
> >> Christian Schneider
> >> ---
> >> http://www.liquid-reality.de
> >>
> >>
> >
> >
> --
> View this message in context:
> http://www.nabble.com/Non-removal-of-jbiendpoints-from-routing-table-tf3527705s12049.html#a9843643
> Sent from the ServiceMix - User mailing list archive at Nabble.com.
>
>
--
Cheers,
Guillaume Nodet
------------------------
Architect, LogicBlaze (http://www.logicblaze.com/)
Blog: http://gnodet.blogspot.com/