You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@servicemix.apache.org by Anne Noseda <no...@gmail.com> on 2008/11/18 10:16:10 UTC
WS-Notification Component Improvment - Clustering
Hello,
To continue our previous discussion about the clustering feature which is
missing in the wsn-component :
- I've tried to use the same subscription ID for 2 different subscribers but
it doesn't work : only one subscriber can be alive at one time. Then the
other can resume the subscription but they cannot be alive in the same time.
So, if we choose this way to resolve the duplication of notification, we
must code a system to know when one instance of Servicemix is down and to
resume the subscription on another instance ... it seems to be very complex
- So I've tried the second solution : virtual topic. I've only tested with
one instance but it seems to work very well so I will continue in this way.
- For the administrative topic, in which we will post all the subscriptions
and registrations requests to warn all the Servicemix instances, where do
you think it must be implement ?
I hesitate between the "doInit()" method of the
"org.apache.servicemix.wsn.component.WSNLifeCycle" class or the "init()"
method of the "org.apache.servicemix.wsn.jms.JmsNotificationBroker" class.
Thanks in advance,
Anne.
++++++++++ PREVIOUS DISCUSSION +++++++++++++++++++++++++
>>>>>>> - what about clustering ? I see that if we have two endpoints
>>>>>>> clustered,
>>>>>>> they will receive two notifications. But I see also you open a JIRA
>>>>>>> for
>>>>>>> it
>>>>>>> (https://issues.apache.org/activemq/browse/SM-418) where you say
>>>>>>> that
>>>>>>> durable subscription can correct it ?
>
>>>>>> I think there are two disctincts use case here. If a subscriber is
>>>>>> a
>>>>>> JBI endpoints (deployed in a service unit for example), this endpoint
>>>>>> may exist in several servicemix instances in the cluster. So they
>>>>>> should act as a *single* subscriber and not receive multiple copies
>>>>>> of
>>>>>> the same notification.
>
>>>>> So, it is an improvment done recently ? Because on your website, you
>>>>> write :
>>>>> "
>>>>> The current implementation has several limitations:
>>>>> * subscriptions can not be clustered: if you register the same
>>>>> subscriber in a cluster, each node will receive all notifications
>>>>> "
>
>>>> No, and that's the problem. They *should* act as a single subscriber,
>>>> but it does not work this way at the moment.
>
>>> Ok, but have you an idea how to correct this ? I have one but I don't
>>> know
>>> if it is the best ... maybe you have a best way to correct that. My idea
>>> is
>>> the following :
>>> 1. when the servicemix instance receives the JMS notifications from the
>>> topic, it tries to write the JMSMessageID property in a DB table (INSERT
>>> operation, ID is the unique column and the primary key)
>>> 2. if it succeed, the servicemix instance sends the notification
>>> 3. if it fails, the servicemix instance don't send the notification
>>> (writes
>>> a log for example)
>
>> Using durable subscriptions would work. If two JMS consumers have the
>> same durable subscription id, only one of them will receive a given
>> message afaik.
>> We may also want to leverage ActiveMQ virtual destinations instead.
>> See http://activemq.apache.org/virtual-destinations.html for more
>> informations.
>
> About virtual destinations :
> - isn't it too specific to ActiveMQ broker ?
> - if I have well understood the concept, the topic is virtual and for each
> subscriber, one physical queue is created : will the notifications be
> duplicated in all these queues for each subscriber ? if yes, isn't it too
> expansive in term of time and place ?
« [hide part of quote]
We may be able to find a way to do both by adding a configuration flag
on the component.
>>> Another point, when an external subscriber sends a subscription message,
>>> this message will be oriented (by an alteon) to one servicemix instance.
>>> This instance will create the JBI endpoint only on this instance or also
>>> on
>>> all the servicemix instances in the cluster ???
>>> Same question for registration of publishers.
>
> You didn't reply ...
I missed that one.
Currently, the WS-Notification components have no knowledge of
clustering, so the subscription / publisher is only valid for the
component that received the request.
At some point in the discussion, we talked about persisting this data
in a database, but it would only work across restart of a single
component. If you need full failover, it means that when a component
receives a subscription or registration, it has to inform the other
components to do so too. I guess we could have a topic for that where
all components would subscribe to and which would be use to send
commands to other components.
At some point for such a system, I guess we might want to use
transactions too...
--
View this message in context: http://www.nabble.com/WS-Notification-Component-Improvment---Clustering-tp20555931p20555931.html
Sent from the ServiceMix - Dev mailing list archive at Nabble.com.
Re: WS-Notification Component Improvment - Clustering & Persistence
Posted by ilyas_asg <il...@asg.com>.
I am impressed by Anne's work.
I also have a question regarding restoring subscriptions after restart:
If I create a PullPoint and send some messages to it, then restart the SM,
we lose the subscription information in SM, while the queue in AMQ is still
holding the messages.
Question: what information has to be persisted externally from the
subscription, so that after restart we can restore the SM subscription and
get all messages from that PullPoint?
Thanks in advance!
ilya
--
View this message in context: http://www.nabble.com/WS-Notification-Component-Improvment---Clustering---Persistence-tp20555931p22213799.html
Sent from the ServiceMix - Dev mailing list archive at Nabble.com.
Re: WS-Notification Component Improvment - Clustering & Persistence
Posted by Guillaume Nodet <gn...@gmail.com>.
On Fri, Dec 5, 2008 at 10:57 AM, Anne Noseda <no...@gmail.com> wrote:
>
> Hello,
>
> I've finished to implement clustering, so the questions in the previous post
> below are obsolete.
> I'll attach to this post my source code and the documentation (in French for
> the moment but if there are needs, I can translate it into English). I can
> also open a JIRA if you prefer, just tell me. This version includes also the
> security features.
Thanks a lot !
A JIRA issue would be better as attachments are removed from the posts ;-)
>
> I've also finished to add the persistence functionality to the component but
> I have a big problem :o) !
> When the component receives a subscription or a registration, it persists it
> into a DB. No problem. When the component starts, it reads the DB and
> creates all the subscriptions and the registrations. There's a problem
> because this operation is made at the initialization of the component BUT
> all the other SA are not necessarily deployed. So the component don't
> retrieve the EPR to associate the subscription to an endpoint on the NMR ...
> For the moment, my implementation works only if we delete the 'data'
> repository in SMX_ROOT before restarting the ESB AND the wsn component must
> be the LAST ONE copied into hotdeploy.
> The only solution, I see is to delete the EPR validation at the subscription
> (same problem for registration) and to make this check at the moment the
> first notification comes ...
> What do you think ?
We could do both I guess. When the subscription is created, check if the EPR
is valid. If not, check again later: it could be a timer or as you
said when the first
notification come (which is fine imho).
>
> Thanks in advance,
>
> Anne.
>
> http://www.nabble.com/file/p20850812/GuideTechWSNCluster.pdf
> GuideTechWSNCluster.pdf
> http://www.nabble.com/file/p20850812/servicemix-wsn2005-etnic.zip
> servicemix-wsn2005-etnic.zip
>
>
> Anne Noseda wrote:
>>
>> Hello,
>>
>> To continue our previous discussion about the clustering feature which is
>> missing in the wsn-component :
>>
>> - I've tried to use the same subscription ID for 2 different subscribers
>> but it doesn't work : only one subscriber can be alive at one time. Then
>> the other can resume the subscription but they cannot be alive in the same
>> time. So, if we choose this way to resolve the duplication of
>> notification, we must code a system to know when one instance of
>> Servicemix is down and to resume the subscription on another instance ...
>> it seems to be very complex
>>
>> - So I've tried the second solution : virtual topic. I've only tested with
>> one instance but it seems to work very well so I will continue in this
>> way.
>>
>> - For the administrative topic, in which we will post all the
>> subscriptions and registrations requests to warn all the Servicemix
>> instances, where do you think it must be implement ?
>> I hesitate between the "doInit()" method of the
>> "org.apache.servicemix.wsn.component.WSNLifeCycle" class or the "init()"
>> method of the "org.apache.servicemix.wsn.jms.JmsNotificationBroker" class.
>>
>> Thanks in advance,
>>
>> Anne.
>>
>> ++++++++++ PREVIOUS DISCUSSION +++++++++++++++++++++++++
>>
>>>>>>>>> - what about clustering ? I see that if we have two endpoints
>>>>>>>>> clustered,
>>>>>>>>> they will receive two notifications. But I see also you open a JIRA
>>>>>>>>> for
>>>>>>>>> it
>>>>>>>>> (https://issues.apache.org/activemq/browse/SM-418) where you say
>>>>>>>>> that
>>>>>>>>> durable subscription can correct it ?
>>>
>>>>>>>> I think there are two disctincts use case here. If a subscriber is
>>>>>>>> a
>>>>>>>> JBI endpoints (deployed in a service unit for example), this
>>>>>>>> endpoint
>>>>>>>> may exist in several servicemix instances in the cluster. So they
>>>>>>>> should act as a *single* subscriber and not receive multiple copies
>>>>>>>> of
>>>>>>>> the same notification.
>>>
>>>>>>> So, it is an improvment done recently ? Because on your website, you
>>>>>>> write :
>>>>>>> "
>>>>>>> The current implementation has several limitations:
>>>>>>> * subscriptions can not be clustered: if you register the same
>>>>>>> subscriber in a cluster, each node will receive all notifications
>>>>>>> "
>>>
>>>>>> No, and that's the problem. They *should* act as a single subscriber,
>>>>>> but it does not work this way at the moment.
>>>
>>>>> Ok, but have you an idea how to correct this ? I have one but I don't
>>>>> know
>>>>> if it is the best ... maybe you have a best way to correct that. My
>>>>> idea
>>>>> is
>>>>> the following :
>>>>> 1. when the servicemix instance receives the JMS notifications from the
>>>>> topic, it tries to write the JMSMessageID property in a DB table
>>>>> (INSERT
>>>>> operation, ID is the unique column and the primary key)
>>>>> 2. if it succeed, the servicemix instance sends the notification
>>>>> 3. if it fails, the servicemix instance don't send the notification
>>>>> (writes
>>>>> a log for example)
>>>
>>>> Using durable subscriptions would work. If two JMS consumers have the
>>>> same durable subscription id, only one of them will receive a given
>>>> message afaik.
>>>> We may also want to leverage ActiveMQ virtual destinations instead.
>>>> See http://activemq.apache.org/virtual-destinations.html for more
>>>> informations.
>>>
>>> About virtual destinations :
>>> - isn't it too specific to ActiveMQ broker ?
>>> - if I have well understood the concept, the topic is virtual and for
>>> each
>>> subscriber, one physical queue is created : will the notifications be
>>> duplicated in all these queues for each subscriber ? if yes, isn't it too
>>> expansive in term of time and place ?
>> « [hide part of quote]
>>
>> We may be able to find a way to do both by adding a configuration flag
>> on the component.
>>
>>>>> Another point, when an external subscriber sends a subscription
>>>>> message,
>>>>> this message will be oriented (by an alteon) to one servicemix
>>>>> instance.
>>>>> This instance will create the JBI endpoint only on this instance or
>>>>> also
>>>>> on
>>>>> all the servicemix instances in the cluster ???
>>>>> Same question for registration of publishers.
>>>
>>> You didn't reply ...
>>
>> I missed that one.
>> Currently, the WS-Notification components have no knowledge of
>> clustering, so the subscription / publisher is only valid for the
>> component that received the request.
>> At some point in the discussion, we talked about persisting this data
>> in a database, but it would only work across restart of a single
>> component. If you need full failover, it means that when a component
>> receives a subscription or registration, it has to inform the other
>> components to do so too. I guess we could have a topic for that where
>> all components would subscribe to and which would be use to send
>> commands to other components.
>>
>> At some point for such a system, I guess we might want to use
>> transactions too...
>>
>
> --
> View this message in context: http://www.nabble.com/WS-Notification-Component-Improvment---Clustering---Persistence-tp20555931p20850812.html
> Sent from the ServiceMix - Dev mailing list archive at Nabble.com.
>
>
--
Cheers,
Guillaume Nodet
------------------------
Blog: http://gnodet.blogspot.com/
------------------------
Open Source SOA
http://fusesource.com
Re: WS-Notification Component Improvment - Clustering & Persistence
Posted by Anne Noseda <no...@gmail.com>.
Hello,
I've finished to implement clustering, so the questions in the previous post
below are obsolete.
I'll attach to this post my source code and the documentation (in French for
the moment but if there are needs, I can translate it into English). I can
also open a JIRA if you prefer, just tell me. This version includes also the
security features.
I've also finished to add the persistence functionality to the component but
I have a big problem :o) !
When the component receives a subscription or a registration, it persists it
into a DB. No problem. When the component starts, it reads the DB and
creates all the subscriptions and the registrations. There's a problem
because this operation is made at the initialization of the component BUT
all the other SA are not necessarily deployed. So the component don't
retrieve the EPR to associate the subscription to an endpoint on the NMR ...
For the moment, my implementation works only if we delete the 'data'
repository in SMX_ROOT before restarting the ESB AND the wsn component must
be the LAST ONE copied into hotdeploy.
The only solution, I see is to delete the EPR validation at the subscription
(same problem for registration) and to make this check at the moment the
first notification comes ...
What do you think ?
Thanks in advance,
Anne.
http://www.nabble.com/file/p20850812/GuideTechWSNCluster.pdf
GuideTechWSNCluster.pdf
http://www.nabble.com/file/p20850812/servicemix-wsn2005-etnic.zip
servicemix-wsn2005-etnic.zip
Anne Noseda wrote:
>
> Hello,
>
> To continue our previous discussion about the clustering feature which is
> missing in the wsn-component :
>
> - I've tried to use the same subscription ID for 2 different subscribers
> but it doesn't work : only one subscriber can be alive at one time. Then
> the other can resume the subscription but they cannot be alive in the same
> time. So, if we choose this way to resolve the duplication of
> notification, we must code a system to know when one instance of
> Servicemix is down and to resume the subscription on another instance ...
> it seems to be very complex
>
> - So I've tried the second solution : virtual topic. I've only tested with
> one instance but it seems to work very well so I will continue in this
> way.
>
> - For the administrative topic, in which we will post all the
> subscriptions and registrations requests to warn all the Servicemix
> instances, where do you think it must be implement ?
> I hesitate between the "doInit()" method of the
> "org.apache.servicemix.wsn.component.WSNLifeCycle" class or the "init()"
> method of the "org.apache.servicemix.wsn.jms.JmsNotificationBroker" class.
>
> Thanks in advance,
>
> Anne.
>
> ++++++++++ PREVIOUS DISCUSSION +++++++++++++++++++++++++
>
>>>>>>>> - what about clustering ? I see that if we have two endpoints
>>>>>>>> clustered,
>>>>>>>> they will receive two notifications. But I see also you open a JIRA
>>>>>>>> for
>>>>>>>> it
>>>>>>>> (https://issues.apache.org/activemq/browse/SM-418) where you say
>>>>>>>> that
>>>>>>>> durable subscription can correct it ?
>>
>>>>>>> I think there are two disctincts use case here. If a subscriber is
>>>>>>> a
>>>>>>> JBI endpoints (deployed in a service unit for example), this
>>>>>>> endpoint
>>>>>>> may exist in several servicemix instances in the cluster. So they
>>>>>>> should act as a *single* subscriber and not receive multiple copies
>>>>>>> of
>>>>>>> the same notification.
>>
>>>>>> So, it is an improvment done recently ? Because on your website, you
>>>>>> write :
>>>>>> "
>>>>>> The current implementation has several limitations:
>>>>>> * subscriptions can not be clustered: if you register the same
>>>>>> subscriber in a cluster, each node will receive all notifications
>>>>>> "
>>
>>>>> No, and that's the problem. They *should* act as a single subscriber,
>>>>> but it does not work this way at the moment.
>>
>>>> Ok, but have you an idea how to correct this ? I have one but I don't
>>>> know
>>>> if it is the best ... maybe you have a best way to correct that. My
>>>> idea
>>>> is
>>>> the following :
>>>> 1. when the servicemix instance receives the JMS notifications from the
>>>> topic, it tries to write the JMSMessageID property in a DB table
>>>> (INSERT
>>>> operation, ID is the unique column and the primary key)
>>>> 2. if it succeed, the servicemix instance sends the notification
>>>> 3. if it fails, the servicemix instance don't send the notification
>>>> (writes
>>>> a log for example)
>>
>>> Using durable subscriptions would work. If two JMS consumers have the
>>> same durable subscription id, only one of them will receive a given
>>> message afaik.
>>> We may also want to leverage ActiveMQ virtual destinations instead.
>>> See http://activemq.apache.org/virtual-destinations.html for more
>>> informations.
>>
>> About virtual destinations :
>> - isn't it too specific to ActiveMQ broker ?
>> - if I have well understood the concept, the topic is virtual and for
>> each
>> subscriber, one physical queue is created : will the notifications be
>> duplicated in all these queues for each subscriber ? if yes, isn't it too
>> expansive in term of time and place ?
> « [hide part of quote]
>
> We may be able to find a way to do both by adding a configuration flag
> on the component.
>
>>>> Another point, when an external subscriber sends a subscription
>>>> message,
>>>> this message will be oriented (by an alteon) to one servicemix
>>>> instance.
>>>> This instance will create the JBI endpoint only on this instance or
>>>> also
>>>> on
>>>> all the servicemix instances in the cluster ???
>>>> Same question for registration of publishers.
>>
>> You didn't reply ...
>
> I missed that one.
> Currently, the WS-Notification components have no knowledge of
> clustering, so the subscription / publisher is only valid for the
> component that received the request.
> At some point in the discussion, we talked about persisting this data
> in a database, but it would only work across restart of a single
> component. If you need full failover, it means that when a component
> receives a subscription or registration, it has to inform the other
> components to do so too. I guess we could have a topic for that where
> all components would subscribe to and which would be use to send
> commands to other components.
>
> At some point for such a system, I guess we might want to use
> transactions too...
>
--
View this message in context: http://www.nabble.com/WS-Notification-Component-Improvment---Clustering---Persistence-tp20555931p20850812.html
Sent from the ServiceMix - Dev mailing list archive at Nabble.com.