Clean deregistration of publishers

[Kuppe <ku...@360t.com>]

I currently have many clients registering for client specific market data
updates from a market data server. The market data server keeps a list of
currently registered clients vs. symbols and sends the relevant updates to a
topic registered by the client (there may be more than one client interested
in the same market data - eg. two users in the same company).

I also use the message limit and eviction policy to throttle the messages at
the topic level to handle slow consumers.

Under normal conditions a client will be shutdown and deregister all market
data registrations. However if a client is abruptly disconnected at a
transport level, this clean deregistration does not take place.

The question is, i would like to know when the server is sending updates to
a topic that has no subscribers. I could then cleanly deregister the market
data requests and save valuable resources.

Can you please help me with the most effective way to handle this scenario?
-- 
View this message in context: http://www.nabble.com/Clean-deregistration-of-publishers-tf1964529.html#a5390888
Sent from the ActiveMQ - User forum at Nabble.com.

Re: Clean deregistration of publishers

[James Strachan <ja...@gmail.com>]

On 7/20/06, Kuppe <ku...@360t.com> wrote:
>
> One final question on this topic - is there additional technical overhead in
> the broker managing large numbers of topics, for example one topic per user
> per symbol? Of course this could grow very quickly with the number of
> users...

Topics are intended to be consumed by many consumers; so 1 topic per
symbol is fine. You can then use the last image caching to get a
market data cache as well :)

http://incubator.apache.org/activemq/subscription-recovery-policy.html


> There is certainly an administrative/support overhead in that monitoring the
> number of topics through a JMS console will be more cumbersome, but this is
> another story... Any thoughts on how best to handle this?

So I really just mean on startup...

* subscribe to the advisory topics you're interested in
* connect to the broker via JMX and query the current consumer state
* now just watch advisories

Otherwise your program could terminate, restart and miss messages.
i.e. think of it as using JMX as your 'last image cache'..

-- 

James
-------
http://radio.weblogs.com/0112098/

Re: Clean deregistration of publishers

[Kuppe <ku...@360t.com>]

One final question on this topic - is there additional technical overhead in
the broker managing large numbers of topics, for example one topic per user
per symbol? Of course this could grow very quickly with the number of
users... 

There is certainly an administrative/support overhead in that monitoring the
number of topics through a JMS console will be more cumbersome, but this is
another story... Any thoughts on how best to handle this?
-- 
View this message in context: http://www.nabble.com/Clean-deregistration-of-publishers-tf1964529.html#a5409052
Sent from the ActiveMQ - User forum at Nabble.com.

Re: Clean deregistration of publishers

[James Strachan <ja...@gmail.com>]

On 7/19/06, Kuppe <ku...@360t.com> wrote:
>
> Wow, very powerful feature!!
>
> First question, how do you suggest that advisory messages can help with the
> following?
>
> > I currently have many clients registering for client specific market data
> > updates from a market data server. The market data server keeps a list of
> > currently registered clients vs. symbols and sends the relevant updates to
> > a
> > topic registered by the client (there may be more than one client
> > interested
> > in the same market data - eg. two users in the same company).

So advisory messages broadcast changes to consumers (e.g. create/close
consumers) on any topics or queues. Also you can query the current
state of the system (e.g. current consumers) via JMX - so you kinda
have a way in default ActiveMQ of knowing what clients are looking at
what symbols (assuming 1 topic per symbol etc)


> Do you suggest including the encoded market data registration in the name of
> the topic or is it possible to include some business related information in
> the advisory messages? It would be really cool if i could restart my
> business service and recover business level registration information from
> some state maintained in the bus!

Am not sure how the business related information hooks into the
consumer information; it might be worth creating your own facade on
top of ActiveMQ's JMX or advisory messages (or both) to join the
advisory message/MBean state with any business related information.
e.g. joining the clientID/username to the details of their business
stuff etc


> I suppose in the simplest form, i could create a consumer of the
> ActiveMQ.Advisory.NoConsumer.Topic.xxxxx for the market data update topic,
> and simply deregister the subscription at a business level when receiving
> messages in this consumer.

That sounds perfect.

A common requirement in market data type stuff is to do 'interactive
feeds' (terminology changes from place to place) - where basically you
monitor the JMS consumers - when folks subscribe to a new topic that
the feed is not publishing, you subscribe to the feed so that it will
be published on the JMS bus. Similarly when no consumers are using a
particular topic, you can trim your subscriptions on the feed.

(As a bit of background, the reason for this is often a combination of
network bandwidth (don't send prices around for stuff that noone is
interested in) and cost (sometimes feed suppliers charge per symbol or
for certain kinds of prices on certain exchanges etc - so keeping the
feed subscription list down saves $$$).

The one thing to be a little careful of is that advisory messages are
kinda events, typically tranisent notifications of things changing. On
startup you often need to look at the current snapshot (so JMX). So on
startup of your InteractiveFeedThingy you'd look at the current
consumers available in JMX, then process any advisory messages going
forward - so both JMX and advisory messages are probably required.

It might make sense to add a helper library for modelling the
connections, producers, consumers and destinations available on a
running system - kinda like a real time POJO model of these things
hiding the details of the JMX code and the advisory messages, so your
tool can just use that . I'm sure some UI tooling could use these
kinds of real time POJOs for visualising the network etc
-- 

James
-------
http://radio.weblogs.com/0112098/

Re: Clean deregistration of publishers

[Kuppe <ku...@360t.com>]

Wow, very powerful feature!!

First question, how do you suggest that advisory messages can help with the
following? 

> I currently have many clients registering for client specific market data
> updates from a market data server. The market data server keeps a list of
> currently registered clients vs. symbols and sends the relevant updates to
> a
> topic registered by the client (there may be more than one client
> interested
> in the same market data - eg. two users in the same company). 

Do you suggest including the encoded market data registration in the name of
the topic or is it possible to include some business related information in
the advisory messages? It would be really cool if i could restart my
business service and recover business level registration information from
some state maintained in the bus!

I suppose in the simplest form, i could create a consumer of the
ActiveMQ.Advisory.NoConsumer.Topic.xxxxx for the market data update topic,
and simply deregister the subscription at a business level when receiving
messages in this consumer. Would you perhaps recommend an alternative
approach?

Thanks again!
-- 
View this message in context: http://www.nabble.com/Clean-deregistration-of-publishers-tf1964529.html#a5392548
Sent from the ActiveMQ - User forum at Nabble.com.

Re: Clean deregistration of publishers

[James Strachan <ja...@gmail.com>]

On 7/19/06, Kuppe <ku...@360t.com> wrote:
> I currently have many clients registering for client specific market data
> updates from a market data server. The market data server keeps a list of
> currently registered clients vs. symbols and sends the relevant updates to a
> topic registered by the client (there may be more than one client interested
> in the same market data - eg. two users in the same company).

Incidentally the advisory messages in ActiveMQ do a similar thing...
http://activemq.org/site/advisory-message.html

not exactly the same but kinda similar


> I also use the message limit and eviction policy to throttle the messages at
> the topic level to handle slow consumers.
>
> Under normal conditions a client will be shutdown and deregister all market
> data registrations. However if a client is abruptly disconnected at a
> transport level, this clean deregistration does not take place.

Though the broker is aware of the subscriptions that each connection
has so subscriptions are closed down properly and the advisory
messages should be sent out correctly.


> The question is, i would like to know when the server is sending updates to
> a topic that has no subscribers. I could then cleanly deregister the market
> data requests and save valuable resources.
>
> Can you please help me with the most effective way to handle this scenario?

Advisory messages are your friend. So you can subscribe to
notifications of messages sent to topics or queues which have no
consumers. Just subscribe to the advisory topics and you'll be
notified.

http://activemq.org/site/advisory-message.html
-- 

James
-------
http://radio.weblogs.com/0112098/