You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by Pete Mueller <pe...@routecloud.com> on 2009/11/26 01:42:44 UTC
Lifecycle of a Camel Component
Does any one have any documentation on the lifecycle of a camel component and
it's associated entities (Endpoint,
Consumer, Producer). I am attempting to write a component for a
event-driving wireline protocol. I understand the basics of:
0. Component is created.
1. URI in DSL gets passed to to component.
2. Component creates Endpoint for each unique URI (if using a singleton)
3. Endpoint creates Consumers and Producers.
But I'm looking for a little more detail as to what triggers the creation
and removal of say Consumers. Specifically,
- Is a single Consumer around for the life of the CamelContext?
- Does an Endpoint get notified if a Consumer it created is stop()'d?
- What is Consumer.stop() used for? Only during shutdown of the
CamelContext? Or is it more like a "pause" than a stop?
- Is there a way (say in the case of a loss of network connection) to stop()
an Endpoint and all associated Consumers and Producers? Then later restart
them after connection has been regained?
- Is there a general example/recommendation on how to handle connections to
unreliable external entities?
I've been using the IRC component as somewhat of a model, but my glance
through the code doesn't really indicate how the component deals with
connection loss or net-splits and other networking pit-falls.
Here's a short outline of the wire protocol I have to implement if it helps:
1. Open TWO TCP connections with Server
2. Register Myself with Server, twice, once as a SENDER, once as a RECEIVER.
Both connections are bi-directional, but the functions I can do on each
connection is limited by how I register myself.
3. At this point I will receive data packets (messages) for some length of
time over the Receiver connection.
4. At some point in the future the Server will send me an "ALL DONE"
message. At which point I must un-register my sender and receiver, but the
TCP connections remail open.
5. After the unregister I need to notify Consumers that there is no more
messages coming and stop Producers from attempting to send more messages.
6. After waiting X seconds (X in most cases is 180) I will attempt to
re-register myself and if successful need to re-activate Consumers and
Producers.
7. I also need to account for the fact that at any time, I may lose a
network connection to the Server, and will need to stop all Consumers and
Producers until connection is restored.
Thanks for any help
-p
--
View this message in context: http://old.nabble.com/Lifecycle-of-a-Camel-Component-tp26522808p26522808.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Lifecycle of a Camel Component
Posted by Claus Ibsen <cl...@gmail.com>.
On Fri, Nov 27, 2009 at 5:33 AM, Pete Mueller <pe...@routecloud.com> wrote:
>
> Thanks very much for your answers, I also found an excellent programming
> guide at the FUSEsource site. That helped a lot as well.
>
> I understand what you wrote below, but it raises some concerns in my mind
> with the reliability of Camel when dealing with unreliable external
> entities. I'm guessing that just stems from a lack of understanding on my
> part. If it helps, I come from an EDI background where systems are designed
> to gaurantee delivery of millions of messages a day to thousands of
> different endpoints. I'll try and illustrate my concern.
>
> I have two routes, the input, from a custom component, is a graphic image
> that is analyized (which can take 30-90 seconds) and them placed on a queue.
> The second route takes items off the queue and emails them to a destination
> AND puts the modified image back on the server.
> Something like:
>
> from("mycomp:host")
> .to(new MyImageProcessor())
> .to("amq:input_queue");
>
> from("amq:"input_queue")
> multicast().to("smtp:test@test.com", "mycomp:host");
>
> Here what I need to account for:
> 1. Most importantly, I cannot lose any messages. So let's say that a
> message is taken off the queue and passed to the smtp component, but the
> mail server is not up, so it throws an exception.
> - What happens to the message? Does it get placed back on the queue? Is
> there a concept of transactions around routes so that only after BOTH
> endpoints confirm a sucessful delivery does the message get removed from the
> queue?
> - What if the smtp server sucessfully delivers and "mycomp" throws an
> exception?
>
> I am aware of the exceptionHandler() idea, but it seems exceptionHandlers
> are placed on a RouteCollection, not a single route. Each instance of
> RouteBuilder could have one route in it. But that creates a lot of extra
> classes.
If you bought the book and read chapter 5 about error handling you got
35 good pages on this matter.
Yes just referring to the fact that the book has a lot of information
about error handling.
Even though the online docu as well, but its not structure as well and
walking you through it so you would understand it well, as the book
does.
Or if you started reading from here
http://camel.apache.org/error-handling-in-camel.html
You will see that it supports 2 scopes
Camel supports 2 scopes for the non transactional type:
- global
- per route
And if you looked at this page
http://camel.apache.org/error-handler.html
There is a few examples on scopes.
>
> 2. After ensuring that i did not lose the message pending at the smtp
> component, I need to shut down the routes since the smtp service is dead
> (for the moment). This would seem to be a job for exceptionHandler(),
> however that handler would need to know which object threw the exception (in
> this case the smtp Producer) and what route it belongs to. then, it would
> need to scan all other route that use the same endpoint, and stop them too.
> I see no way of doing this from within the handler, since it is not notified
> which Producer threw the exception. Also, I see no way of consistently
> getting from a Producer to it's owner endpoint.
:) Have you actually looked at the Producer interface? There is a
getEndpoint() which is the owner endpoint.
public interface Producer extends Processor, Service, IsSingleton {
/**
* Gets the endpoint this producer sends to.
*
* @return the endpoint
*/
Endpoint getEndpoint();
> - How do you gracefully and without data loss bring down a route?
> - Does bringing down a route destroy the consumer/producer objects? or just
> stop() them?
>
> 3. Assuming I could figure out which routes to shut down. I can execute
> route.stop(). But what happens to messages that are "in process" take for
> example a message from mycomp that is currently being analyzed, when the
> analyzer finishes (say 20-30 seconds after the route.stop() was called:
> - Is the queue Producer still available?
> - Will route.stop() block until all parts of the route are shut down?
> - Will route.stop() stop ONLY the consumer that is part of it's route, or
> the entire route?
> - Really, I would not need to bring down the route, just stop() the consumer
> at the beginning. Do the Producers/Consumers/Endpoint have any knowledge
> about the route they are on? Or can they get it some how?
> - Can a Consumer/Producer be shared amongst several routes?
>
> It could be that I have the wrong idea of what Camel is meant to do. All of
> the examples I've seen deal with short routes from some system to some other
> system where it seems the assumption is made that everything is up all the
> time. That is the nature of examples. When dealing with Enterprise
> Integrations (well at least the integration I deal with), the biggest issue
> is always data integrity during connection failures. It could be that all
> this transactional nature and data integrity is in Camel, I just don't see
> it. It is also a valid answer to say that I need to "roll my own" system,
> but that lessens the usefulness of Camel as an "Enterprise" tool.
>
> BTW, if we can figure all this out with Camel, then I'll be happy to add an
> example to the collection that shows how to setup message "integrity".
> -p
>
>
>
>
> Claus Ibsen-2 wrote:
>>
>> Hi
>>
>> A lot of questions. Let me give a start and try answering a couple of
>> those.
>>
>>
>>
>> On Thu, Nov 26, 2009 at 1:42 AM, Pete Mueller <pe...@routecloud.com> wrote:
>>>
>>> Does any one have any documentation on the lifecycle of a camel component
>>> and
>>> it's associated entities (Endpoint,
>>> Consumer, Producer). I am attempting to write a component for a
>>> event-driving wireline protocol. I understand the basics of:
>>> 0. Component is created.
>>> 1. URI in DSL gets passed to to component.
>>> 2. Component creates Endpoint for each unique URI (if using a singleton)
>>> 3. Endpoint creates Consumers and Producers.
>>>
>>> But I'm looking for a little more detail as to what triggers the creation
>>> and removal of say Consumers. Specifically,
>>> - Is a single Consumer around for the life of the CamelContext?
>>
>> Consumers and Producers are in reality more dynamic as you can have
>> create a consumer, use it once and then discard it.
>> CamelContext does not hold a repository for all created
>> consumers/producers etc. As you can create then without involvement
>> from CamelContext.
>>
>> On the other hand when a consumer is used as a input to a route, then
>> that consumer is around for the life of that route.
>>
>>
>>> - Does an Endpoint get notified if a Consumer it created is stop()'d?
>>
>> No there is no such callback. Camel have a very liberal and light API
>> for Endpoint, Consumer, Producer so its easy to work with an implement
>> custom components etc.
>>
>> There is a event API added in Camel 2.1 which have notifications when
>> routes are started/stopped etc. This can be used in case you need to
>> do some custom stuff.
>>
>>> - What is Consumer.stop() used for? Only during shutdown of the
>>> CamelContext? Or is it more like a "pause" than a stop?
>>
>> Both. You can stop a route at runtime. But when Camel shutdown it will
>> go through all routes, components, endpoints, services etc. Basically
>> all what it has and stop them one by one.
>>
>>
>>> - Is there a way (say in the case of a loss of network connection) to
>>> stop()
>>> an Endpoint and all associated Consumers and Producers? Then later
>>> restart
>>> them after connection has been regained?
>>
>> Yeah you can for example use Camel error handling and stop a route in
>> case of an ConnectionException or what you like.
>>
>>
>> In Camel 2.1 you can use RoutePolicy to that as well. However you must
>> implement some logic yourself that monitors that particular connection
>> and can start/stop the route depending on its state. Camel does not do
>> that for you.
>>
>>
>>> - Is there a general example/recommendation on how to handle connections
>>> to
>>> unreliable external entities?
>>>
>>
>> No not 2 business are the same. Some wants to self heal and just try
>> at next interval. Others want to try 5 times with X delays and then
>> alert someone if still a problem.
>>
>>
>>> I've been using the IRC component as somewhat of a model, but my glance
>>> through the code doesn't really indicate how the component deals with
>>> connection loss or net-splits and other networking pit-falls.
>>>
>>
>> Again connectivity is not that easy to encompass into a single model
>> for dealing with lost connections.
>> Often you only got the wacky java Exceptions that may or may not
>> indicate a io/connection problem.
>>
>> However when you have a Camel consumer = input to a route then you got
>> a chicken egg problem as Camel does not take over
>> before the consumer has successfully got a message and created an
>> Exchange to be routed in Camel. At this point Camel take over and you
>> can use Camel error handling. But before that all error handling etc.
>> is component specify.
>>
>>
>> We love contributions so dig into it and help out.
>> http://camel.apache.org/contributing.html
>>
>>
>>> Here's a short outline of the wire protocol I have to implement if it
>>> helps:
>>>
>>> 1. Open TWO TCP connections with Server
>>> 2. Register Myself with Server, twice, once as a SENDER, once as a
>>> RECEIVER.
>>> Both connections are bi-directional, but the functions I can do on each
>>> connection is limited by how I register myself.
>>> 3. At this point I will receive data packets (messages) for some length
>>> of
>>> time over the Receiver connection.
>>> 4. At some point in the future the Server will send me an "ALL DONE"
>>> message. At which point I must un-register my sender and receiver, but
>>> the
>>> TCP connections remail open.
>>> 5. After the unregister I need to notify Consumers that there is no more
>>> messages coming and stop Producers from attempting to send more messages.
>>> 6. After waiting X seconds (X in most cases is 180) I will attempt to
>>> re-register myself and if successful need to re-activate Consumers and
>>> Producers.
>>> 7. I also need to account for the fact that at any time, I may lose a
>>> network connection to the Server, and will need to stop all Consumers and
>>> Producers until connection is restored.
>>>
>>> Thanks for any help
>>> -p
>>> --
>>> View this message in context:
>>> http://old.nabble.com/Lifecycle-of-a-Camel-Component-tp26522808p26522808.html
>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>>
>> --
>> Claus Ibsen
>> Apache Camel Committer
>>
>> Author of Camel in Action: http://www.manning.com/ibsen/
>> Open Source Integration: http://fusesource.com
>> Blog: http://davsclaus.blogspot.com/
>> Twitter: http://twitter.com/davsclaus
>>
>>
>
> --
> View this message in context: http://old.nabble.com/Lifecycle-of-a-Camel-Component-tp26522808p26535877.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>
--
Claus Ibsen
Apache Camel Committer
Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
Re: Lifecycle of a Camel Component
Posted by Claus Ibsen <cl...@gmail.com>.
> The same applies for HTTP based as well.
Should be that they do NOT support transactions. (well maybe that
WebService transaction does but I have newer seen that in action)
Re: Lifecycle of a Camel Component
Posted by Claus Ibsen <cl...@gmail.com>.
Hi Peter
On Fri, Nov 27, 2009 at 5:33 AM, Pete Mueller <pe...@routecloud.com> wrote:
>
> Thanks very much for your answers, I also found an excellent programming
> guide at the FUSEsource site. That helped a lot as well.
>
> I understand what you wrote below, but it raises some concerns in my mind
> with the reliability of Camel when dealing with unreliable external
> entities. I'm guessing that just stems from a lack of understanding on my
> part. If it helps, I come from an EDI background where systems are designed
> to gaurantee delivery of millions of messages a day to thousands of
> different endpoints. I'll try and illustrate my concern.
>
> I have two routes, the input, from a custom component, is a graphic image
> that is analyized (which can take 30-90 seconds) and them placed on a queue.
> The second route takes items off the queue and emails them to a destination
> AND puts the modified image back on the server.
> Something like:
>
> from("mycomp:host")
> .to(new MyImageProcessor())
> .to("amq:input_queue");
>
> from("amq:"input_queue")
> multicast().to("smtp:test@test.com", "mycomp:host");
>
> Here what I need to account for:
> 1. Most importantly, I cannot lose any messages. So let's say that a
> message is taken off the queue and passed to the smtp component, but the
> mail server is not up, so it throws an exception.
> - What happens to the message? Does it get placed back on the queue? Is
> there a concept of transactions around routes so that only after BOTH
> endpoints confirm a sucessful delivery does the message get removed from the
> queue?
Yes you can use transactions so the message will be rolled back to the
JMS queue.
> - What if the smtp server sucessfully delivers and "mycomp" throws an
> exception?
>
SMTP protocol does not support transactions, its in fact only a few
protocols that does this such as JMS, JDBC.
The same applies for HTTP based as well.
Its not a limitation in Camel but just in general that these protocols
cannot participate in distributed transactions.
> I am aware of the exceptionHandler() idea, but it seems exceptionHandlers
> are placed on a RouteCollection, not a single route. Each instance of
> RouteBuilder could have one route in it. But that creates a lot of extra
> classes.
>
No you can have error handlers on a global and route scope.
> 2. After ensuring that i did not lose the message pending at the smtp
> component, I need to shut down the routes since the smtp service is dead
> (for the moment). This would seem to be a job for exceptionHandler(),
> however that handler would need to know which object threw the exception (in
> this case the smtp Producer) and what route it belongs to. then, it would
> need to scan all other route that use the same endpoint, and stop them too.
> I see no way of doing this from within the handler, since it is not notified
> which Producer threw the exception. Also, I see no way of consistently
> getting from a Producer to it's owner endpoint.
In Camel 2.1 a header will be enriched with the endpoint that failed
so you got that information at hand now.
Exchange.FAILURE_ENDPOINT
> - How do you gracefully and without data loss bring down a route?
> - Does bringing down a route destroy the consumer/producer objects? or just
> stop() them?
>
See for example RoutePolicy. Also if you use JMS as "safe grounds" for
messages then you will not loose messages.
So if a remote SMTP server is temporary down the need to shutdown a
route is not as important.
> 3. Assuming I could figure out which routes to shut down. I can execute
> route.stop(). But what happens to messages that are "in process" take for
> example a message from mycomp that is currently being analyzed, when the
> analyzer finishes (say 20-30 seconds after the route.stop() was called:
> - Is the queue Producer still available?
> - Will route.stop() block until all parts of the route are shut down?
> - Will route.stop() stop ONLY the consumer that is part of it's route, or
> the entire route?
We have graceful shutdown on the roadmap for 2.2. However the key
pieces are in 2.1 already.
There is an in flight registry you can poke to see if there are any
exchanges in progress on a particular route.
And thus know when to stop it.
> - Really, I would not need to bring down the route, just stop() the consumer
> at the beginning. Do the Producers/Consumers/Endpoint have any knowledge
> about the route they are on? Or can they get it some how?
> - Can a Consumer/Producer be shared amongst several routes?
>
Yes but then you need to do that manually as the fluent builder will
create a new consumer/producer per route.
And you can design your custom endpoint to return a shared
consumer/producer if you really need this?
> It could be that I have the wrong idea of what Camel is meant to do. All of
> the examples I've seen deal with short routes from some system to some other
> system where it seems the assumption is made that everything is up all the
> time. That is the nature of examples. When dealing with Enterprise
> Integrations (well at least the integration I deal with), the biggest issue
> is always data integrity during connection failures. It could be that all
> this transactional nature and data integrity is in Camel, I just don't see
> it. It is also a valid answer to say that I need to "roll my own" system,
> but that lessens the usefulness of Camel as an "Enterprise" tool.
>
> BTW, if we can figure all this out with Camel, then I'll be happy to add an
> example to the collection that shows how to setup message "integrity".
> -p
>
Consider looking at the ActiveMQ in Action book which covers such
scenarios as the broker is the backbone for such an infrastructure
with messaging "integrity".
There are some case studies here
http://fusesource.com/resources/collateral/
For example Sabre have been running for more than 1,5 years and not
had a downtime and lost a message.
Camel itself is just a framework/API that makes working with the
zillion of other protocols/frameworks much simpler.
And if you want to know much more about Camel and how it works, the
online documentation is of course avail.
But we are writing a book at the moment which covers many details how
it works inside Camel and whatnot.
Not trying to sell but just saying that the book should be considered
a well structured reference and user guide for Camel.
The wiki pages just dont cut it, its a bit to messy. But they are
great for searching, so just type some keywords in the searchbox on
the frontpage.
>
>
>
> Claus Ibsen-2 wrote:
>>
>> Hi
>>
>> A lot of questions. Let me give a start and try answering a couple of
>> those.
>>
>>
>>
>> On Thu, Nov 26, 2009 at 1:42 AM, Pete Mueller <pe...@routecloud.com> wrote:
>>>
>>> Does any one have any documentation on the lifecycle of a camel component
>>> and
>>> it's associated entities (Endpoint,
>>> Consumer, Producer). I am attempting to write a component for a
>>> event-driving wireline protocol. I understand the basics of:
>>> 0. Component is created.
>>> 1. URI in DSL gets passed to to component.
>>> 2. Component creates Endpoint for each unique URI (if using a singleton)
>>> 3. Endpoint creates Consumers and Producers.
>>>
>>> But I'm looking for a little more detail as to what triggers the creation
>>> and removal of say Consumers. Specifically,
>>> - Is a single Consumer around for the life of the CamelContext?
>>
>> Consumers and Producers are in reality more dynamic as you can have
>> create a consumer, use it once and then discard it.
>> CamelContext does not hold a repository for all created
>> consumers/producers etc. As you can create then without involvement
>> from CamelContext.
>>
>> On the other hand when a consumer is used as a input to a route, then
>> that consumer is around for the life of that route.
>>
>>
>>> - Does an Endpoint get notified if a Consumer it created is stop()'d?
>>
>> No there is no such callback. Camel have a very liberal and light API
>> for Endpoint, Consumer, Producer so its easy to work with an implement
>> custom components etc.
>>
>> There is a event API added in Camel 2.1 which have notifications when
>> routes are started/stopped etc. This can be used in case you need to
>> do some custom stuff.
>>
>>> - What is Consumer.stop() used for? Only during shutdown of the
>>> CamelContext? Or is it more like a "pause" than a stop?
>>
>> Both. You can stop a route at runtime. But when Camel shutdown it will
>> go through all routes, components, endpoints, services etc. Basically
>> all what it has and stop them one by one.
>>
>>
>>> - Is there a way (say in the case of a loss of network connection) to
>>> stop()
>>> an Endpoint and all associated Consumers and Producers? Then later
>>> restart
>>> them after connection has been regained?
>>
>> Yeah you can for example use Camel error handling and stop a route in
>> case of an ConnectionException or what you like.
>>
>>
>> In Camel 2.1 you can use RoutePolicy to that as well. However you must
>> implement some logic yourself that monitors that particular connection
>> and can start/stop the route depending on its state. Camel does not do
>> that for you.
>>
>>
>>> - Is there a general example/recommendation on how to handle connections
>>> to
>>> unreliable external entities?
>>>
>>
>> No not 2 business are the same. Some wants to self heal and just try
>> at next interval. Others want to try 5 times with X delays and then
>> alert someone if still a problem.
>>
>>
>>> I've been using the IRC component as somewhat of a model, but my glance
>>> through the code doesn't really indicate how the component deals with
>>> connection loss or net-splits and other networking pit-falls.
>>>
>>
>> Again connectivity is not that easy to encompass into a single model
>> for dealing with lost connections.
>> Often you only got the wacky java Exceptions that may or may not
>> indicate a io/connection problem.
>>
>> However when you have a Camel consumer = input to a route then you got
>> a chicken egg problem as Camel does not take over
>> before the consumer has successfully got a message and created an
>> Exchange to be routed in Camel. At this point Camel take over and you
>> can use Camel error handling. But before that all error handling etc.
>> is component specify.
>>
>>
>> We love contributions so dig into it and help out.
>> http://camel.apache.org/contributing.html
>>
>>
>>> Here's a short outline of the wire protocol I have to implement if it
>>> helps:
>>>
>>> 1. Open TWO TCP connections with Server
>>> 2. Register Myself with Server, twice, once as a SENDER, once as a
>>> RECEIVER.
>>> Both connections are bi-directional, but the functions I can do on each
>>> connection is limited by how I register myself.
>>> 3. At this point I will receive data packets (messages) for some length
>>> of
>>> time over the Receiver connection.
>>> 4. At some point in the future the Server will send me an "ALL DONE"
>>> message. At which point I must un-register my sender and receiver, but
>>> the
>>> TCP connections remail open.
>>> 5. After the unregister I need to notify Consumers that there is no more
>>> messages coming and stop Producers from attempting to send more messages.
>>> 6. After waiting X seconds (X in most cases is 180) I will attempt to
>>> re-register myself and if successful need to re-activate Consumers and
>>> Producers.
>>> 7. I also need to account for the fact that at any time, I may lose a
>>> network connection to the Server, and will need to stop all Consumers and
>>> Producers until connection is restored.
>>>
>>> Thanks for any help
>>> -p
>>> --
>>> View this message in context:
>>> http://old.nabble.com/Lifecycle-of-a-Camel-Component-tp26522808p26522808.html
>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>>
>> --
>> Claus Ibsen
>> Apache Camel Committer
>>
>> Author of Camel in Action: http://www.manning.com/ibsen/
>> Open Source Integration: http://fusesource.com
>> Blog: http://davsclaus.blogspot.com/
>> Twitter: http://twitter.com/davsclaus
>>
>>
>
> --
> View this message in context: http://old.nabble.com/Lifecycle-of-a-Camel-Component-tp26522808p26535877.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>
--
Claus Ibsen
Apache Camel Committer
Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
Re: Lifecycle of a Camel Component
Posted by Pete Mueller <pe...@routecloud.com>.
Thanks very much for your answers, I also found an excellent programming
guide at the FUSEsource site. That helped a lot as well.
I understand what you wrote below, but it raises some concerns in my mind
with the reliability of Camel when dealing with unreliable external
entities. I'm guessing that just stems from a lack of understanding on my
part. If it helps, I come from an EDI background where systems are designed
to gaurantee delivery of millions of messages a day to thousands of
different endpoints. I'll try and illustrate my concern.
I have two routes, the input, from a custom component, is a graphic image
that is analyized (which can take 30-90 seconds) and them placed on a queue.
The second route takes items off the queue and emails them to a destination
AND puts the modified image back on the server.
Something like:
from("mycomp:host")
.to(new MyImageProcessor())
.to("amq:input_queue");
from("amq:"input_queue")
multicast().to("smtp:test@test.com", "mycomp:host");
Here what I need to account for:
1. Most importantly, I cannot lose any messages. So let's say that a
message is taken off the queue and passed to the smtp component, but the
mail server is not up, so it throws an exception.
- What happens to the message? Does it get placed back on the queue? Is
there a concept of transactions around routes so that only after BOTH
endpoints confirm a sucessful delivery does the message get removed from the
queue?
- What if the smtp server sucessfully delivers and "mycomp" throws an
exception?
I am aware of the exceptionHandler() idea, but it seems exceptionHandlers
are placed on a RouteCollection, not a single route. Each instance of
RouteBuilder could have one route in it. But that creates a lot of extra
classes.
2. After ensuring that i did not lose the message pending at the smtp
component, I need to shut down the routes since the smtp service is dead
(for the moment). This would seem to be a job for exceptionHandler(),
however that handler would need to know which object threw the exception (in
this case the smtp Producer) and what route it belongs to. then, it would
need to scan all other route that use the same endpoint, and stop them too.
I see no way of doing this from within the handler, since it is not notified
which Producer threw the exception. Also, I see no way of consistently
getting from a Producer to it's owner endpoint.
- How do you gracefully and without data loss bring down a route?
- Does bringing down a route destroy the consumer/producer objects? or just
stop() them?
3. Assuming I could figure out which routes to shut down. I can execute
route.stop(). But what happens to messages that are "in process" take for
example a message from mycomp that is currently being analyzed, when the
analyzer finishes (say 20-30 seconds after the route.stop() was called:
- Is the queue Producer still available?
- Will route.stop() block until all parts of the route are shut down?
- Will route.stop() stop ONLY the consumer that is part of it's route, or
the entire route?
- Really, I would not need to bring down the route, just stop() the consumer
at the beginning. Do the Producers/Consumers/Endpoint have any knowledge
about the route they are on? Or can they get it some how?
- Can a Consumer/Producer be shared amongst several routes?
It could be that I have the wrong idea of what Camel is meant to do. All of
the examples I've seen deal with short routes from some system to some other
system where it seems the assumption is made that everything is up all the
time. That is the nature of examples. When dealing with Enterprise
Integrations (well at least the integration I deal with), the biggest issue
is always data integrity during connection failures. It could be that all
this transactional nature and data integrity is in Camel, I just don't see
it. It is also a valid answer to say that I need to "roll my own" system,
but that lessens the usefulness of Camel as an "Enterprise" tool.
BTW, if we can figure all this out with Camel, then I'll be happy to add an
example to the collection that shows how to setup message "integrity".
-p
Claus Ibsen-2 wrote:
>
> Hi
>
> A lot of questions. Let me give a start and try answering a couple of
> those.
>
>
>
> On Thu, Nov 26, 2009 at 1:42 AM, Pete Mueller <pe...@routecloud.com> wrote:
>>
>> Does any one have any documentation on the lifecycle of a camel component
>> and
>> it's associated entities (Endpoint,
>> Consumer, Producer). I am attempting to write a component for a
>> event-driving wireline protocol. I understand the basics of:
>> 0. Component is created.
>> 1. URI in DSL gets passed to to component.
>> 2. Component creates Endpoint for each unique URI (if using a singleton)
>> 3. Endpoint creates Consumers and Producers.
>>
>> But I'm looking for a little more detail as to what triggers the creation
>> and removal of say Consumers. Specifically,
>> - Is a single Consumer around for the life of the CamelContext?
>
> Consumers and Producers are in reality more dynamic as you can have
> create a consumer, use it once and then discard it.
> CamelContext does not hold a repository for all created
> consumers/producers etc. As you can create then without involvement
> from CamelContext.
>
> On the other hand when a consumer is used as a input to a route, then
> that consumer is around for the life of that route.
>
>
>> - Does an Endpoint get notified if a Consumer it created is stop()'d?
>
> No there is no such callback. Camel have a very liberal and light API
> for Endpoint, Consumer, Producer so its easy to work with an implement
> custom components etc.
>
> There is a event API added in Camel 2.1 which have notifications when
> routes are started/stopped etc. This can be used in case you need to
> do some custom stuff.
>
>> - What is Consumer.stop() used for? Only during shutdown of the
>> CamelContext? Or is it more like a "pause" than a stop?
>
> Both. You can stop a route at runtime. But when Camel shutdown it will
> go through all routes, components, endpoints, services etc. Basically
> all what it has and stop them one by one.
>
>
>> - Is there a way (say in the case of a loss of network connection) to
>> stop()
>> an Endpoint and all associated Consumers and Producers? Then later
>> restart
>> them after connection has been regained?
>
> Yeah you can for example use Camel error handling and stop a route in
> case of an ConnectionException or what you like.
>
>
> In Camel 2.1 you can use RoutePolicy to that as well. However you must
> implement some logic yourself that monitors that particular connection
> and can start/stop the route depending on its state. Camel does not do
> that for you.
>
>
>> - Is there a general example/recommendation on how to handle connections
>> to
>> unreliable external entities?
>>
>
> No not 2 business are the same. Some wants to self heal and just try
> at next interval. Others want to try 5 times with X delays and then
> alert someone if still a problem.
>
>
>> I've been using the IRC component as somewhat of a model, but my glance
>> through the code doesn't really indicate how the component deals with
>> connection loss or net-splits and other networking pit-falls.
>>
>
> Again connectivity is not that easy to encompass into a single model
> for dealing with lost connections.
> Often you only got the wacky java Exceptions that may or may not
> indicate a io/connection problem.
>
> However when you have a Camel consumer = input to a route then you got
> a chicken egg problem as Camel does not take over
> before the consumer has successfully got a message and created an
> Exchange to be routed in Camel. At this point Camel take over and you
> can use Camel error handling. But before that all error handling etc.
> is component specify.
>
>
> We love contributions so dig into it and help out.
> http://camel.apache.org/contributing.html
>
>
>> Here's a short outline of the wire protocol I have to implement if it
>> helps:
>>
>> 1. Open TWO TCP connections with Server
>> 2. Register Myself with Server, twice, once as a SENDER, once as a
>> RECEIVER.
>> Both connections are bi-directional, but the functions I can do on each
>> connection is limited by how I register myself.
>> 3. At this point I will receive data packets (messages) for some length
>> of
>> time over the Receiver connection.
>> 4. At some point in the future the Server will send me an "ALL DONE"
>> message. At which point I must un-register my sender and receiver, but
>> the
>> TCP connections remail open.
>> 5. After the unregister I need to notify Consumers that there is no more
>> messages coming and stop Producers from attempting to send more messages.
>> 6. After waiting X seconds (X in most cases is 180) I will attempt to
>> re-register myself and if successful need to re-activate Consumers and
>> Producers.
>> 7. I also need to account for the fact that at any time, I may lose a
>> network connection to the Server, and will need to stop all Consumers and
>> Producers until connection is restored.
>>
>> Thanks for any help
>> -p
>> --
>> View this message in context:
>> http://old.nabble.com/Lifecycle-of-a-Camel-Component-tp26522808p26522808.html
>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>
>>
>
>
>
> --
> Claus Ibsen
> Apache Camel Committer
>
> Author of Camel in Action: http://www.manning.com/ibsen/
> Open Source Integration: http://fusesource.com
> Blog: http://davsclaus.blogspot.com/
> Twitter: http://twitter.com/davsclaus
>
>
--
View this message in context: http://old.nabble.com/Lifecycle-of-a-Camel-Component-tp26522808p26535877.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Lifecycle of a Camel Component
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
A lot of questions. Let me give a start and try answering a couple of those.
On Thu, Nov 26, 2009 at 1:42 AM, Pete Mueller <pe...@routecloud.com> wrote:
>
> Does any one have any documentation on the lifecycle of a camel component and
> it's associated entities (Endpoint,
> Consumer, Producer). I am attempting to write a component for a
> event-driving wireline protocol. I understand the basics of:
> 0. Component is created.
> 1. URI in DSL gets passed to to component.
> 2. Component creates Endpoint for each unique URI (if using a singleton)
> 3. Endpoint creates Consumers and Producers.
>
> But I'm looking for a little more detail as to what triggers the creation
> and removal of say Consumers. Specifically,
> - Is a single Consumer around for the life of the CamelContext?
Consumers and Producers are in reality more dynamic as you can have
create a consumer, use it once and then discard it.
CamelContext does not hold a repository for all created
consumers/producers etc. As you can create then without involvement
from CamelContext.
On the other hand when a consumer is used as a input to a route, then
that consumer is around for the life of that route.
> - Does an Endpoint get notified if a Consumer it created is stop()'d?
No there is no such callback. Camel have a very liberal and light API
for Endpoint, Consumer, Producer so its easy to work with an implement
custom components etc.
There is a event API added in Camel 2.1 which have notifications when
routes are started/stopped etc. This can be used in case you need to
do some custom stuff.
> - What is Consumer.stop() used for? Only during shutdown of the
> CamelContext? Or is it more like a "pause" than a stop?
Both. You can stop a route at runtime. But when Camel shutdown it will
go through all routes, components, endpoints, services etc. Basically
all what it has and stop them one by one.
> - Is there a way (say in the case of a loss of network connection) to stop()
> an Endpoint and all associated Consumers and Producers? Then later restart
> them after connection has been regained?
Yeah you can for example use Camel error handling and stop a route in
case of an ConnectionException or what you like.
In Camel 2.1 you can use RoutePolicy to that as well. However you must
implement some logic yourself that monitors that particular connection
and can start/stop the route depending on its state. Camel does not do
that for you.
> - Is there a general example/recommendation on how to handle connections to
> unreliable external entities?
>
No not 2 business are the same. Some wants to self heal and just try
at next interval. Others want to try 5 times with X delays and then
alert someone if still a problem.
> I've been using the IRC component as somewhat of a model, but my glance
> through the code doesn't really indicate how the component deals with
> connection loss or net-splits and other networking pit-falls.
>
Again connectivity is not that easy to encompass into a single model
for dealing with lost connections.
Often you only got the wacky java Exceptions that may or may not
indicate a io/connection problem.
However when you have a Camel consumer = input to a route then you got
a chicken egg problem as Camel does not take over
before the consumer has successfully got a message and created an
Exchange to be routed in Camel. At this point Camel take over and you
can use Camel error handling. But before that all error handling etc.
is component specify.
We love contributions so dig into it and help out.
http://camel.apache.org/contributing.html
> Here's a short outline of the wire protocol I have to implement if it helps:
>
> 1. Open TWO TCP connections with Server
> 2. Register Myself with Server, twice, once as a SENDER, once as a RECEIVER.
> Both connections are bi-directional, but the functions I can do on each
> connection is limited by how I register myself.
> 3. At this point I will receive data packets (messages) for some length of
> time over the Receiver connection.
> 4. At some point in the future the Server will send me an "ALL DONE"
> message. At which point I must un-register my sender and receiver, but the
> TCP connections remail open.
> 5. After the unregister I need to notify Consumers that there is no more
> messages coming and stop Producers from attempting to send more messages.
> 6. After waiting X seconds (X in most cases is 180) I will attempt to
> re-register myself and if successful need to re-activate Consumers and
> Producers.
> 7. I also need to account for the fact that at any time, I may lose a
> network connection to the Server, and will need to stop all Consumers and
> Producers until connection is restored.
>
> Thanks for any help
> -p
> --
> View this message in context: http://old.nabble.com/Lifecycle-of-a-Camel-Component-tp26522808p26522808.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>
--
Claus Ibsen
Apache Camel Committer
Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus