You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by falconair <sh...@gmail.com> on 2009/03/24 20:48:46 UTC
how to programatically create thousands of topics (for marketdata)?
I would like to publish market data through qpid. Since I have massive amount of data flowing through the pipes, I can't publish all that data to a single topic, since the client may only be interested in one or two symbols. The usual solution to use a hierarchical topic naming scheme. In other words, rather than publishing every thing to "level1," I want to split data according to symbols "level1.msft," "level2.orcl," etc. Clearly I can't have a pre-defined list of topics in some properties file. What is the best way of solving this?
I am using qpid m4 or MRG (C++ broker) with jms client api.
Thanks
--
View this message in context: http://n2.nabble.com/how-to-programatically-create-thousands-of-topics-%28for-marketdata%29--tp2528658p2528658.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: how to programatically create thousands of topics (for marketdata)?
Posted by Carl Trieloff <cc...@redhat.com>.
falconair wrote:
> I would like to publish market data through qpid. Since I have massive amount of data flowing through the pipes, I can't publish all that data to a single topic, since the client may only be interested in one or two symbols. The usual solution to use a hierarchical topic naming scheme. In other words, rather than publishing every thing to "level1," I want to split data according to symbols "level1.msft," "level2.orcl," etc. Clearly I can't have a pre-defined list of topics in some properties file. What is the best way of solving this?
>
> I am using qpid m4 or MRG (C++ broker) with jms client api.
>
> Thanks
>
The fastest way to load so many queues is to use async session when
declaring them. the default for declare is sync. This will mean that the
creation
will get batched. once they are all created call sync() on session.
regards
Carl.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: how to programatically create thousands of topics (for
marketdata)?
Posted by falconair <sh...@gmail.com>.
I just tried your solution, it works :)
I didn't realize I could create topics on the fly.
falconair,
Since I don't do this kind of thing myself, I may not understand all of
the issues in your application domain. Let me see if I understand what
the issues are.
1. There are many topics. You want to use hierarchies to manage them.
Wildcards do that well in the topic exchange.
2. You want to create topics dynamically, at run time, without using a
predefined properties file. I think this is possible using
'session.createTopic(topicName)', where topicName can use wildcards like
'usa.*', '*.news', etc.
What am I missing? What other issues are there?
Jonathan
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
--
View this message in context: http://n2.nabble.com/how-to-programatically-create-thousands-of-topics-%28for-marketdata%29--tp2528658p2534085.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: how to programatically create thousands of topics (for marketdata)?
Posted by Jonathan Robie <jo...@redhat.com>.
falconair,
Since I don't do this kind of thing myself, I may not understand all of
the issues in your application domain. Let me see if I understand what
the issues are.
1. There are many topics. You want to use hierarchies to manage them.
Wildcards do that well in the topic exchange.
2. You want to create topics dynamically, at run time, without using a
predefined properties file. I think this is possible using
'session.createTopic(topicName)', where topicName can use wildcards like
'usa.*', '*.news', etc.
What am I missing? What other issues are there?
Jonathan
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: how to programatically create thousands of topics (for marketdata)?
Posted by Gordon Sim <gs...@redhat.com>.
falconair wrote:
> You are right, that is indeed how I currently use qpid. However, for
> marketdata, there might be thousands of symbols.
>
> Currently, every topic I use has to be defined in my properties file
> as such: topic.a=x.a topic.b=x.b ...
>
> I can't list all the symbols (once for level1, again for level2 then
> again for timeandsales): partly because the properties file would
> grow too large, partly because managing these symbols (and their
> changes) would be a nightmare.
>
> That's why I wanted to define some top level topics (level1.*,
> level2.*, etc.) and somehow avoid having to pre-define the topics at
> the symbol level (level1.msft, level1.orcl, level1.ebay...).
>
> I think the solution is to use Headers exchange (if it works through
> JMS)
Ok, if I understand correctly you want to avoid the configuration
overhead of defining each each symbol as a separate JMS Destination? If
so I'm not sure the headers exchange offers any way round that (at least
not on the consumer side).
The issue is not that the routing algorithm in use doesn't fit, but that
the details of the routing are hidden behind the Destination object and
you don't want to configure each symbol as a separate Destination
You could create the Destinations in code to avoid this since the
application must have knowledge of the symbols in question anyway if you
are not relying on configuration.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: how to programatically create thousands of topics (for
marketdata)?
Posted by falconair <sh...@gmail.com>.
You are right, that is indeed how I currently use qpid. However, for marketdata, there might be thousands of symbols.
Currently, every topic I use has to be defined in my properties file as such:
topic.a=x.a
topic.b=x.b
...
I can't list all the symbols (once for level1, again for level2 then again for timeandsales): partly because the properties file would grow too large, partly because managing these symbols (and their changes) would be a nightmare.
That's why I wanted to define some top level topics (level1.*, level2.*, etc.) and somehow avoid having to pre-define the topics at the symbol level (level1.msft, level1.orcl, level1.ebay...).
I think the solution is to use Headers exchange (if it works through JMS)
Forgive me if I'm being over simplistic, but isn't that exactly what the
topic exchange gives you?
I.e. you publish your data using "level1.msft", "level2.orcl" etc as the
routing key and then your subscribers can bind queues as appropriate.
E.g. "level1.msft" would only match that exact symbol
"#.msft" would match level1.msft or level2.msft
"level1.#" would match all level1 symbols
You can bind the same subscription queue more than once, so you can set
it up to e.g. receive anything matching "#.msft" or "#.orcl". How
exactly you do that using the binding URL syntax for the JMS client I
will leave to those more qualified to comment.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
--
View this message in context: http://n2.nabble.com/how-to-programatically-create-thousands-of-topics-%28for-marketdata%29--tp2528658p2532404.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: how to programatically create thousands of topics (for marketdata)?
Posted by Gordon Sim <gs...@redhat.com>.
falconair wrote:
> I would like to publish market data through qpid. Since I have
> massive amount of data flowing through the pipes, I can't publish all
> that data to a single topic, since the client may only be interested
> in one or two symbols. The usual solution to use a hierarchical
> topic naming scheme. In other words, rather than publishing every
> thing to "level1," I want to split data according to symbols
> "level1.msft," "level2.orcl," etc. Clearly I can't have a
> pre-defined list of topics in some properties file. What is the best
> way of solving this?
Forgive me if I'm being over simplistic, but isn't that exactly what the
topic exchange gives you?
I.e. you publish your data using "level1.msft", "level2.orcl" etc as the
routing key and then your subscribers can bind queues as appropriate.
E.g. "level1.msft" would only match that exact symbol
"#.msft" would match level1.msft or level2.msft
"level1.#" would match all level1 symbols
You can bind the same subscription queue more than once, so you can set
it up to e.g. receive anything matching "#.msft" or "#.orcl". How
exactly you do that using the binding URL syntax for the JMS client I
will leave to those more qualified to comment.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: how to programatically create thousands of topics (for marketdata)?
Posted by Carl Trieloff <cc...@redhat.com>.
falconair wrote:
> The main reason I can't have pre-defined topics (in properties file or database) is because symbols are changed every day (some may be added, some may be deleted, some may just change slightly).
>
> Maintaining an accurate list of these symbols is not trivial (some large banks have groups dedicated to this).
>
>
> After sending the original message, I realized that I might be able to take advantage of selectors or 'where clause' like functionality. In other words, I could subscribe to level1 but limit the values to "symbol=MSFT" (which means only MSFT should travel down the wire to the client, nothing else). Can't find if qpid has this capability or not though.
>
you can also use the exclusive binding key feature. it has been used for
this use case before
http://qpid.apache.org/cheat-sheet-for-configuring-exchange-options.html
Carl.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: how to programatically create thousands of topics (for
marketdata)?
Posted by falconair <sh...@gmail.com>.
Hi think Headers exchange might be the one I need. Unfortunately I can't find any developer-friendly docs on it. MRG manual doesn't mention it and qpid.apache.org doesn't seem to describe how to use it from a Java client (google doesn't help much).
Any pointers on how I can access Headers exchange from java? Or perhaps I can just continue to use the JMS api, but pass in sql-where clause like selector expressions to createConsumer(destination, <selector>)?
Thanks
2009/3/24 falconair <sh...@gmail.com>:
> After sending the original message, I realized that I might be able to take advantage of selectors or 'where clause' like functionality. In other words, I could subscribe to level1 but limit the values to "symbol=MSFT" (which means only MSFT should travel down the wire to the client, nothing else). Can't find if qpid has this capability or not though.
You could use the Headers exchange. This allows you to bind a queue to
an exchange and routing to the queue is only done when one or more
message headers match.
So in your message header you could have a common header called (e.g.)
RIC and then just bind the queue to the exchange with RIC=MSFT (or
whatever). You can OR multiple values together to check for values in
a set.
Selectors would also work but are not implemented in the C++ broker
yet (on the server, they do have client selectors but not a good idea
for your use case).
RG
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
--
View this message in context: http://n2.nabble.com/how-to-programatically-create-thousands-of-topics-%28for-marketdata%29--tp2528658p2530724.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: how to programatically create thousands of topics (for
marketdata)?
Posted by Robert Greig <ro...@gmail.com>.
2009/3/24 falconair <sh...@gmail.com>:
> After sending the original message, I realized that I might be able to take advantage of selectors or 'where clause' like functionality. In other words, I could subscribe to level1 but limit the values to "symbol=MSFT" (which means only MSFT should travel down the wire to the client, nothing else). Can't find if qpid has this capability or not though.
You could use the Headers exchange. This allows you to bind a queue to
an exchange and routing to the queue is only done when one or more
message headers match.
So in your message header you could have a common header called (e.g.)
RIC and then just bind the queue to the exchange with RIC=MSFT (or
whatever). You can OR multiple values together to check for values in
a set.
Selectors would also work but are not implemented in the C++ broker
yet (on the server, they do have client selectors but not a good idea
for your use case).
RG
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: how to programatically create thousands of topics (for
marketdata)?
Posted by falconair <sh...@gmail.com>.
The main reason I can't have pre-defined topics (in properties file or database) is because symbols are changed every day (some may be added, some may be deleted, some may just change slightly).
Maintaining an accurate list of these symbols is not trivial (some large banks have groups dedicated to this).
After sending the original message, I realized that I might be able to take advantage of selectors or 'where clause' like functionality. In other words, I could subscribe to level1 but limit the values to "symbol=MSFT" (which means only MSFT should travel down the wire to the client, nothing else). Can't find if qpid has this capability or not though.
falconair wrote:
> I would like to publish market data through qpid. Since I have massive amount of data flowing through the pipes, I can't publish all that data to a single topic, since the client may only be interested in one or two symbols. The usual solution to use a hierarchical topic naming scheme. In other words, rather than publishing every thing to "level1," I want to split data according to symbols "level1.msft," "level2.orcl," etc. Clearly I can't have a pre-defined list of topics in some properties file. What is the best way of solving this?
>
> I am using qpid m4 or MRG (C++ broker) with jms client api.
>
I assume you are familiar with the topic exchange (used in the
public-subscribe example), and your only question is how to communicate
what topics are available for subscriptions?
Why not keep the available topics on a database on the server, with a
"topic server", a client that reports on the available topics?
Jonathan
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
--
View this message in context: http://n2.nabble.com/how-to-programatically-create-thousands-of-topics-%28for-marketdata%29--tp2528658p2529000.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: how to programatically create thousands of topics (for marketdata)?
Posted by Jonathan Robie <jo...@redhat.com>.
falconair wrote:
> I would like to publish market data through qpid. Since I have massive amount of data flowing through the pipes, I can't publish all that data to a single topic, since the client may only be interested in one or two symbols. The usual solution to use a hierarchical topic naming scheme. In other words, rather than publishing every thing to "level1," I want to split data according to symbols "level1.msft," "level2.orcl," etc. Clearly I can't have a pre-defined list of topics in some properties file. What is the best way of solving this?
>
> I am using qpid m4 or MRG (C++ broker) with jms client api.
>
I assume you are familiar with the topic exchange (used in the
public-subscribe example), and your only question is how to communicate
what topics are available for subscriptions?
Why not keep the available topics on a database on the server, with a
"topic server", a client that reports on the available topics?
Jonathan
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org