You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by markgize <ma...@pimco.com> on 2013/08/08 01:22:11 UTC
Creating then starting a JMS consumer around another component
We are interested in Camel but have a specific startup sequence that we need
to follow:
1) Open a JMS topic consumer for our updates, but, do not get any of the
messages.
2) Query our database and build a state of the world
3) Start to get the update messages from the consumer we opened
We cannot adopt 2, 1, 3, as during the time we are done with our query but
are still constructing the state of the world, we might miss an update. 1,
3, 2, is also a problem as an update could present an object which our state
of the world subsequently tries to build, resulting in a duplicate. We
could endeavor to handle this duplicate, but it is powerful to load the bulk
of our data in the state of the world phase and demand absolutely no
exceptions.
What we have currently implemented can also result in duplicates. The state
of the world might contain an object which an update then subsequently
provides, but in this case we merely reject just one update, as opposed to
having an issue with the much more significant state of the world operation.
How can we achieve 1, 2, 3 using Camel? The difficulty that I see is that
our consumer must be receiving messages, however, it must not consume them
until the state of the world completes.
Thanks for your time.
--
View this message in context: http://camel.465427.n5.nabble.com/Creating-then-starting-a-JMS-consumer-around-another-component-tp5736955.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Creating then starting a JMS consumer around another component
Posted by Christian Posta <ch...@gmail.com>.
Actually, you should be able to use a RoutePolicy to accomplish this:
from("jms:queue").routePolicy(customPolicy).to(...) continue route ...
http://camel.apache.org/routepolicy.html
In the onExchangeBegin() method, do not allow the processing to continue
(block) until the "world has been constructed"
On Thu, Aug 15, 2013 at 9:30 AM, nigel <ni...@pimco.com> wrote:
> This is a common pattern I've seen before. It's not practical to have
> persistent queues/topics with each update to the database table being
> stored
> due to the transient nature of the consumer population (number of consumers
> is variable and their liveliness has no pattern). As Mark points out you
> need to subscribe to updates, buffer them, get the current 'view of the
> world' from the authoritative source (ie the database) and then let the
> updates flow in - potentially discarding duplicates at the start which were
> already in the image due to the timing.
> So in camel how can we start a 'route' so that it registers the
> subscription
> but 'pause' it whilst we get the table? It seems like we might need three
> routes and some funky 'on/off' tap logic. I don't see an EIP that supports
> this.
> Nigel
>
>
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/Creating-then-starting-a-JMS-consumer-around-another-component-tp5736955p5737347.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
--
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta
Re: Creating then starting a JMS consumer around another component
Posted by nigel <ni...@pimco.com>.
This is a common pattern I've seen before. It's not practical to have
persistent queues/topics with each update to the database table being stored
due to the transient nature of the consumer population (number of consumers
is variable and their liveliness has no pattern). As Mark points out you
need to subscribe to updates, buffer them, get the current 'view of the
world' from the authoritative source (ie the database) and then let the
updates flow in - potentially discarding duplicates at the start which were
already in the image due to the timing.
So in camel how can we start a 'route' so that it registers the subscription
but 'pause' it whilst we get the table? It seems like we might need three
routes and some funky 'on/off' tap logic. I don't see an EIP that supports
this.
Nigel
--
View this message in context: http://camel.465427.n5.nabble.com/Creating-then-starting-a-JMS-consumer-around-another-component-tp5736955p5737347.html
Sent from the Camel - Users mailing list archive at Nabble.com.
RE: Creating then starting a JMS consumer around another component
Posted by markgize <ma...@pimco.com>.
Thanks for your responses.
Q: Why not setup your consumer before you start up your camel context?
At the moment we prepare our context:
CamelContext context = new DefaultCamelContext();
context.addComponent(JMS_QUEUE_COMPONENT_NAME, new JmsComponent(new JmsConfiguration(JmsFactory.getQueueConnectionFactory())));
context.addComponent(JMS_TOPIC_COMPONENT_NAME, new JmsComponent(new JmsConfiguration(JmsFactory.getTopicConnectionFactory())));
Apply a route builder:
String requestRouteUri = CamelHelper.buildQueueRouteUri(ServiceFactory.getRequestQueue());
String responseRouteUri = CamelHelper.buildTopicRouteUri(ServiceFactory.getResponseTopic());
String logRouteUri = CamelHelper.buildLogRouteUri(LOGGER, LoggingLevel.TRACE);
from(requestRouteUri).to(logRouteUri).process(engineProcessor).to(logRouteUri).to(responseRouteUri);
from(responseRouteUri).to(logRouteUri).process(messageTranslator).to(logRouteUri).process(engineProcessor);
Call start and Camel does its magic. We could start our consumer outside of Camel and then insert into the context, but then we are doing work outside of Camel and lose some of the elegance. It would be nice if we could handle this use case entirely within Camel, I also would have thought our use case is fairly typical (state of the world from the DB with updates via messaging), might we possibly create a new EIP to handle this situation or is that ridiculous?
Q: Could a better solution be to use durable subscriptions or ActiveMQ virtual topics so you're not depending on the timing of your consumer starting up as the basis for not missing messages?
We have an arbitrary number of clients who come and go as they please so I don't see durable subscriptions working that well, however, I think this could work. Nevertheless, I would note that our current approach is relatively simple and works perfectly, it is purely implementing this approach in Camel with which we have had trouble. Ideally, I think we would not have to tweak our architecture in order to adopt Camel.
From: ceposta [via Camel] [mailto:ml-node+s465427n5737003h96@n5.nabble.com]
Sent: Thursday, August 08, 2013 8:37 AM
To: Gizejewski, Mark
Subject: Re: Creating then starting a JMS consumer around another component
Could a better solution be to use durable subscriptions or ActiveMQ virtual
topics so you're not depending on the timing of your consumer starting up
as the basis for not missing messages?
On Wed, Aug 7, 2013 at 6:43 PM, John D. Ament <[hidden email]</user/SendEmail.jtp?type=node&node=5737003&i=0>>wrote:
> Hi,
>
> Why isn't 2,1,3 possible?
> Why not setup your consumer before you start up your camel context?
>
> On Wed, Aug 7, 2013 at 7:22 PM, markgize <[hidden email]</user/SendEmail.jtp?type=node&node=5737003&i=1>>
> wrote:
> > We are interested in Camel but have a specific startup sequence that we
> need
> > to follow:
> >
> > 1) Open a JMS topic consumer for our updates, but, do not get any of the
> > messages.
> > 2) Query our database and build a state of the world
> > 3) Start to get the update messages from the consumer we opened
> >
> > We cannot adopt 2, 1, 3, as during the time we are done with our query
> but
> > are still constructing the state of the world, we might miss an update.
> 1,
> > 3, 2, is also a problem as an update could present an object which our
> state
> > of the world subsequently tries to build, resulting in a duplicate. We
> > could endeavor to handle this duplicate, but it is powerful to load the
> bulk
> > of our data in the state of the world phase and demand absolutely no
> > exceptions.
> >
> > What we have currently implemented can also result in duplicates. The
> state
> > of the world might contain an object which an update then subsequently
> > provides, but in this case we merely reject just one update, as opposed
> to
> > having an issue with the much more significant state of the world
> operation.
> >
> > How can we achieve 1, 2, 3 using Camel? The difficulty that I see is
> that
> > our consumer must be receiving messages, however, it must not consume
> them
> > until the state of the world completes.
> >
> > Thanks for your time.
> >
> >
> >
> > --
> > View this message in context:
> http://camel.465427.n5.nabble.com/Creating-then-starting-a-JMS-consumer-around-another-component-tp5736955.html
> > Sent from the Camel - Users mailing list archive at Nabble.com.
>
--
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta
http://www.christianposta.com/blog
________________________________
If you reply to this email, your message will be added to the discussion below:
http://camel.465427.n5.nabble.com/Creating-then-starting-a-JMS-consumer-around-another-component-tp5736955p5737003.html
To unsubscribe from Creating then starting a JMS consumer around another component, click here<http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=5736955&code=bWFyay5naXplamV3c2tpQHBpbWNvLmNvbXw1NzM2OTU1fC0xODE4MjkzMjc0>.
NAML<http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
This message contains confidential information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute, alter or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmissions cannot be guaranteed to be secure or without error as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender, therefore, does not accept liability for any errors or omissions in the contents of this message which arise during or as a result of e-mail transmission. If verification is required, please request a hard-copy version. This message is provided for information purposes and should not be construed as a solicitation or offer to buy or sell any securities or related financial instruments in any jurisdiction.
--
View this message in context: http://camel.465427.n5.nabble.com/Creating-then-starting-a-JMS-consumer-around-another-component-tp5736955p5737004.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Creating then starting a JMS consumer around another component
Posted by Christian Posta <ch...@gmail.com>.
Could a better solution be to use durable subscriptions or ActiveMQ virtual
topics so you're not depending on the timing of your consumer starting up
as the basis for not missing messages?
On Wed, Aug 7, 2013 at 6:43 PM, John D. Ament <jo...@gmail.com>wrote:
> Hi,
>
> Why isn't 2,1,3 possible?
> Why not setup your consumer before you start up your camel context?
>
> On Wed, Aug 7, 2013 at 7:22 PM, markgize <ma...@pimco.com>
> wrote:
> > We are interested in Camel but have a specific startup sequence that we
> need
> > to follow:
> >
> > 1) Open a JMS topic consumer for our updates, but, do not get any of the
> > messages.
> > 2) Query our database and build a state of the world
> > 3) Start to get the update messages from the consumer we opened
> >
> > We cannot adopt 2, 1, 3, as during the time we are done with our query
> but
> > are still constructing the state of the world, we might miss an update.
> 1,
> > 3, 2, is also a problem as an update could present an object which our
> state
> > of the world subsequently tries to build, resulting in a duplicate. We
> > could endeavor to handle this duplicate, but it is powerful to load the
> bulk
> > of our data in the state of the world phase and demand absolutely no
> > exceptions.
> >
> > What we have currently implemented can also result in duplicates. The
> state
> > of the world might contain an object which an update then subsequently
> > provides, but in this case we merely reject just one update, as opposed
> to
> > having an issue with the much more significant state of the world
> operation.
> >
> > How can we achieve 1, 2, 3 using Camel? The difficulty that I see is
> that
> > our consumer must be receiving messages, however, it must not consume
> them
> > until the state of the world completes.
> >
> > Thanks for your time.
> >
> >
> >
> > --
> > View this message in context:
> http://camel.465427.n5.nabble.com/Creating-then-starting-a-JMS-consumer-around-another-component-tp5736955.html
> > Sent from the Camel - Users mailing list archive at Nabble.com.
>
--
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta
Re: Creating then starting a JMS consumer around another component
Posted by "John D. Ament" <jo...@gmail.com>.
Hi,
Why isn't 2,1,3 possible?
Why not setup your consumer before you start up your camel context?
On Wed, Aug 7, 2013 at 7:22 PM, markgize <ma...@pimco.com> wrote:
> We are interested in Camel but have a specific startup sequence that we need
> to follow:
>
> 1) Open a JMS topic consumer for our updates, but, do not get any of the
> messages.
> 2) Query our database and build a state of the world
> 3) Start to get the update messages from the consumer we opened
>
> We cannot adopt 2, 1, 3, as during the time we are done with our query but
> are still constructing the state of the world, we might miss an update. 1,
> 3, 2, is also a problem as an update could present an object which our state
> of the world subsequently tries to build, resulting in a duplicate. We
> could endeavor to handle this duplicate, but it is powerful to load the bulk
> of our data in the state of the world phase and demand absolutely no
> exceptions.
>
> What we have currently implemented can also result in duplicates. The state
> of the world might contain an object which an update then subsequently
> provides, but in this case we merely reject just one update, as opposed to
> having an issue with the much more significant state of the world operation.
>
> How can we achieve 1, 2, 3 using Camel? The difficulty that I see is that
> our consumer must be receiving messages, however, it must not consume them
> until the state of the world completes.
>
> Thanks for your time.
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Creating-then-starting-a-JMS-consumer-around-another-component-tp5736955.html
> Sent from the Camel - Users mailing list archive at Nabble.com.