You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@camel.apache.org by Christian Ohr <ch...@gmail.com> on 2012/10/23 13:18:23 UTC
[DISCUSS] Message Store
Hi everyone,
in the past I came several times across situations that required the
one or other kind of Message Store. I noticed that in Camel this
doesn't seem to be a "first class citizen" in the sense of a primary
architectural concept, which can be applied consistently whereever
needed.
In short, it might make sense to have a unified, generic, pluggable
Message Store (probably more of an "Exchange store") in Camel that
consolidates the different approaches and allows to similarly
parameterize persistence to various EIP patterns, and can be used
independently of EIP patterns as well. Implementations would handle
the mapping to the underlying database or file system or NoSQL or
whatever.
It seems that Spring Integration provides something corresponding
(http://static.springsource.org/spring-integration/reference/htmlsingle/#message-store).
Message Store implementations are already used by Camel in various
places, although using different approaches :
* Stream Caching (only file system)
* AggregationRepository (used for splitters, multicasts etc., but the
interface is not specific at all to aggregation use cases)
* IdempotentRepository
Message Store is requested for in other places:
* Reliable stream resequencing (CAMEL-949)
* Persistent Dead Letter Queue (CAMEL-4575)
And there might be other areas (seda, bam) that might benefit as well.
Maybe generalizing the AggregationRepository is a way to go forward.
Side note: The OeHF IPF platform (which is built on top of Camel and
partly extends it to the health care domain) has something called a
"Flow Manager" (http://www.openehealth.org/display/ipf2/Flow+management)
that is used for tracking exchanges while they are processed or after
processing is done, thereby being able to re-insert them into the
route. Not that I consider this being a shining example, but it shows
that message stores make sense outside their implicit use in EIP
processors as well.
Looking forward to your opinions!
regards
Christian
Re: [DISCUSS] Message Store
Posted by aedwards <ad...@middleware360.com>.
+1 for this feature. It is very much needed. Agree with Bengt's comments.
--
View this message in context: http://camel.465427.n5.nabble.com/DISCUSS-Message-Store-tp5721454p5721503.html
Sent from the Camel Development mailing list archive at Nabble.com.
Re: [DISCUSS] Message Store
Posted by Christian Müller <ch...@gmail.com>.
We can discuss some ideas at ApacheCon EU (I will be there). But we have to
make sure we share the content of our discussion afterwards here at this
list.
Looking forward to meeting you there...
Best,
Christian
Sent from a mobile device
Am 25.10.2012 15:55 schrieb "Christian Ohr" <ch...@gmail.com>:
> I would never dare to subvert the Camel community - in fact that's why
> I started this thread here.
>
> Christian
>
>
> 2012/10/25 Rob Davies <ra...@gmail.com>:
> > +1
> >
> >
> >
> > On 25 Oct 2012, at 12:44, Hadrian Zbarcea <hz...@gmail.com> wrote:
> >
> >> There will be quite a few Camel committers at ACEU. Technical
> discussions must be inclusive of the community and take place on this dev@list though.
> >>
> >> Hadrian
> >>
> >>
> >> On 10/25/2012 03:47 AM, Christian Ohr wrote:
> >>> Let me quickly summarize -
> >>>
> >>> there appears to be quite some interest from the community (and
> >>> apparently a number of homegrown solutions to fill the gap) and some
> >>> plans around a Message History EIP (at the bottom of
> >>> http://camel.apache.org/camel-30-roadmap.html) for a yet unscheduled
> >>> future Camel release.
> >>>
> >>> Personally, on this issue I would prefer quality (consistency,
> >>> sustainability) over time, but just moving this into the future may be
> >>> not sufficient either. If someone of the PMC plans to attend the
> >>> ApacheCon EU in November, maybe we can brainstorm a bit or exchange
> >>> further ideas...
> >>>
> >>> regards
> >>> Christian
> >>>
>
Re: [DISCUSS] Message Store
Posted by Christian Ohr <ch...@gmail.com>.
I would never dare to subvert the Camel community - in fact that's why
I started this thread here.
Christian
2012/10/25 Rob Davies <ra...@gmail.com>:
> +1
>
>
>
> On 25 Oct 2012, at 12:44, Hadrian Zbarcea <hz...@gmail.com> wrote:
>
>> There will be quite a few Camel committers at ACEU. Technical discussions must be inclusive of the community and take place on this dev@ list though.
>>
>> Hadrian
>>
>>
>> On 10/25/2012 03:47 AM, Christian Ohr wrote:
>>> Let me quickly summarize -
>>>
>>> there appears to be quite some interest from the community (and
>>> apparently a number of homegrown solutions to fill the gap) and some
>>> plans around a Message History EIP (at the bottom of
>>> http://camel.apache.org/camel-30-roadmap.html) for a yet unscheduled
>>> future Camel release.
>>>
>>> Personally, on this issue I would prefer quality (consistency,
>>> sustainability) over time, but just moving this into the future may be
>>> not sufficient either. If someone of the PMC plans to attend the
>>> ApacheCon EU in November, maybe we can brainstorm a bit or exchange
>>> further ideas...
>>>
>>> regards
>>> Christian
>>>
Re: [DISCUSS] Message Store
Posted by Rob Davies <ra...@gmail.com>.
+1
On 25 Oct 2012, at 12:44, Hadrian Zbarcea <hz...@gmail.com> wrote:
> There will be quite a few Camel committers at ACEU. Technical discussions must be inclusive of the community and take place on this dev@ list though.
>
> Hadrian
>
>
> On 10/25/2012 03:47 AM, Christian Ohr wrote:
>> Let me quickly summarize -
>>
>> there appears to be quite some interest from the community (and
>> apparently a number of homegrown solutions to fill the gap) and some
>> plans around a Message History EIP (at the bottom of
>> http://camel.apache.org/camel-30-roadmap.html) for a yet unscheduled
>> future Camel release.
>>
>> Personally, on this issue I would prefer quality (consistency,
>> sustainability) over time, but just moving this into the future may be
>> not sufficient either. If someone of the PMC plans to attend the
>> ApacheCon EU in November, maybe we can brainstorm a bit or exchange
>> further ideas...
>>
>> regards
>> Christian
>>
Re: [DISCUSS] Message Store
Posted by Hadrian Zbarcea <hz...@gmail.com>.
There will be quite a few Camel committers at ACEU. Technical
discussions must be inclusive of the community and take place on this
dev@ list though.
Hadrian
On 10/25/2012 03:47 AM, Christian Ohr wrote:
> Let me quickly summarize -
>
> there appears to be quite some interest from the community (and
> apparently a number of homegrown solutions to fill the gap) and some
> plans around a Message History EIP (at the bottom of
> http://camel.apache.org/camel-30-roadmap.html) for a yet unscheduled
> future Camel release.
>
> Personally, on this issue I would prefer quality (consistency,
> sustainability) over time, but just moving this into the future may be
> not sufficient either. If someone of the PMC plans to attend the
> ApacheCon EU in November, maybe we can brainstorm a bit or exchange
> further ideas...
>
> regards
> Christian
>
Re: [DISCUSS] Message Store
Posted by Christian Ohr <ch...@gmail.com>.
Let me quickly summarize -
there appears to be quite some interest from the community (and
apparently a number of homegrown solutions to fill the gap) and some
plans around a Message History EIP (at the bottom of
http://camel.apache.org/camel-30-roadmap.html) for a yet unscheduled
future Camel release.
Personally, on this issue I would prefer quality (consistency,
sustainability) over time, but just moving this into the future may be
not sufficient either. If someone of the PMC plans to attend the
ApacheCon EU in November, maybe we can brainstorm a bit or exchange
further ideas...
regards
Christian
Re: [DISCUSS] Message Store
Posted by James Carman <ja...@carmanconsulting.com>.
I'm +1 for this feature and wouldn't mind helping out. We've done
something very similar to this at work and would very much like for it
to be "baked in" already for us.
On Wed, Oct 24, 2012 at 9:50 AM, Claus Ibsen <cl...@gmail.com> wrote:
> On Wed, Oct 24, 2012 at 8:46 AM, Bengt Rodehav <be...@rodehav.com> wrote:
>> I think this is an excellent idea. We're using Camel as the basis for our
>> integration platform and a message store is a mandatory component. We need:
>>
>> - To store all message traffic in order to have an audit trail. Both
>> successful and failed exchanges.
>> - To provide a certain level of manual retry. That is to get the original
>> message from the store and feed it back in the originating route.
>> - Flexibility to specify what should be stored (e g what exchange- and
>> message properties) and also filter what exchanges should be stored.
>>
>> Currently we have implemented our own message store but it requires us to
>> adapt the routes to make this happen. A more generic way of handling this
>> would be terrific.
>>
>
> The Message History EIP is scheduled for overhaul/improvement in Camel 3.0.
> And in this light we will look at a message store API as well.
>
>
>
>> /Bengt
>>
>> 2012/10/24 Willem jiang <wi...@gmail.com>
>>
>>> There is a lots of way to implement to Message store.
>>> Option1. If we treat is a first class of Camel, we could build up a set of
>>> API to let aggregator or tracer to use. As it is common usage that we need
>>> to store the exchange persistently, It make sense to provides such kind of
>>> API in Camel as a first class. +1 to provide this kind of API in Camel 3.x.
>>>
>>> Option2. We can also leverage the already exits camel components such as
>>> polling consumer or camel-JPA components to implement a such Message store.
>>> I think IPF shows us a good way to go. In this way we don't need to wait
>>> for the New API from Camel, it could be good solution for Camel 2.x.
>>>
>>> Maybe it is a good time for us to think about Camel 3.0 again.
>>>
>>> Any thoughts?
>>>
>>> Willem
>>>
>>> --
>>> Willem Jiang
>>>
>>> Red Hat, Inc.
>>> FuseSource is now part of Red Hat
>>> Web: http://www.fusesource.com | http://www.redhat.com
>>> Blog: http://willemjiang.blogspot.com (http://willemjiang.blogspot.com/)
>>> (English)
>>> http://jnn.javaeye.com (http://jnn.javaeye.com/) (Chinese)
>>> Twitter: willemjiang
>>> Weibo: willemjiang
>>>
>>>
>>>
>>>
>>>
>>> On Wednesday, October 24, 2012 at 9:43 AM, Hadrian Zbarcea wrote:
>>>
>>> > Hi Christian,
>>> >
>>> > You are correct about the fact that a Message Store is missing (kinda).
>>> > There are a few components that support a message store implementation
>>> > of sorts, as you mentioned. You are also correct that it should be a
>>> > first class citizen.
>>> >
>>> > Practically, an endpoint that supports a polling consumer is a message
>>> > store. What is actually missing is random access via a polling consumer.
>>> > A message store would then work atop any endpoint that supports random
>>> > retrieval (polling). That would create the opportunity to cleanup a bit
>>> > the pollEnrich implementation as well.
>>> >
>>> > My $0.02,
>>> > Hadrian
>>> >
>>> >
>>> > On 10/23/2012 07:18 AM, Christian Ohr wrote:
>>> > > Hi everyone,
>>> > >
>>> > > in the past I came several times across situations that required the
>>> > > one or other kind of Message Store. I noticed that in Camel this
>>> > > doesn't seem to be a "first class citizen" in the sense of a primary
>>> > > architectural concept, which can be applied consistently whereever
>>> > > needed.
>>> > >
>>> > > In short, it might make sense to have a unified, generic, pluggable
>>> > > Message Store (probably more of an "Exchange store") in Camel that
>>> > > consolidates the different approaches and allows to similarly
>>> > > parameterize persistence to various EIP patterns, and can be used
>>> > > independently of EIP patterns as well. Implementations would handle
>>> > > the mapping to the underlying database or file system or NoSQL or
>>> > > whatever.
>>> > > It seems that Spring Integration provides something corresponding
>>> > > (
>>> http://static.springsource.org/spring-integration/reference/htmlsingle/#message-store
>>> ).
>>> > >
>>> > > Message Store implementations are already used by Camel in various
>>> > > places, although using different approaches :
>>> > > * Stream Caching (only file system)
>>> > > * AggregationRepository (used for splitters, multicasts etc., but the
>>> > > interface is not specific at all to aggregation use cases)
>>> > > * IdempotentRepository
>>> > >
>>> > > Message Store is requested for in other places:
>>> > > * Reliable stream resequencing (CAMEL-949)
>>> > > * Persistent Dead Letter Queue (CAMEL-4575)
>>> > >
>>> > > And there might be other areas (seda, bam) that might benefit as well.
>>> > > Maybe generalizing the AggregationRepository is a way to go forward.
>>> > >
>>> > > Side note: The OeHF IPF platform (which is built on top of Camel and
>>> > > partly extends it to the health care domain) has something called a
>>> > > "Flow Manager" (
>>> http://www.openehealth.org/display/ipf2/Flow+management)
>>> > > that is used for tracking exchanges while they are processed or after
>>> > > processing is done, thereby being able to re-insert them into the
>>> > > route. Not that I consider this being a shining example, but it shows
>>> > > that message stores make sense outside their implicit use in EIP
>>> > > processors as well.
>>> > >
>>> > > Looking forward to your opinions!
>>> > >
>>> > > regards
>>> > > Christian
>>> >
>>>
>>>
>>>
>>>
>
>
>
> --
> Claus Ibsen
> -----------------
> Red Hat, Inc.
> FuseSource is now part of Red Hat
> Email: cibsen@redhat.com
> Web: http://fusesource.com
> Twitter: davsclaus
> Blog: http://davsclaus.com
> Author of Camel in Action: http://www.manning.com/ibsen
Re: [DISCUSS] Message Store
Posted by Björn Bength <bj...@gmail.com>.
Yeah
We also need this kind of message store right now.
Our need is claim-check, history, polling consumer and with random access.
I'd thought we'd try with MongoDB, because it fits our current use
case, if no camel component fits the bill.
But it would be nice to be able to leverage an "official" solution .
/Björn
On Wed, Oct 24, 2012 at 3:50 PM, Claus Ibsen <cl...@gmail.com> wrote:
> On Wed, Oct 24, 2012 at 8:46 AM, Bengt Rodehav <be...@rodehav.com> wrote:
>> I think this is an excellent idea. We're using Camel as the basis for our
>> integration platform and a message store is a mandatory component. We need:
>>
>> - To store all message traffic in order to have an audit trail. Both
>> successful and failed exchanges.
>> - To provide a certain level of manual retry. That is to get the original
>> message from the store and feed it back in the originating route.
>> - Flexibility to specify what should be stored (e g what exchange- and
>> message properties) and also filter what exchanges should be stored.
>>
>> Currently we have implemented our own message store but it requires us to
>> adapt the routes to make this happen. A more generic way of handling this
>> would be terrific.
>>
>
> The Message History EIP is scheduled for overhaul/improvement in Camel 3.0.
> And in this light we will look at a message store API as well.
>
>
>
>> /Bengt
>>
>> 2012/10/24 Willem jiang <wi...@gmail.com>
>>
>>> There is a lots of way to implement to Message store.
>>> Option1. If we treat is a first class of Camel, we could build up a set of
>>> API to let aggregator or tracer to use. As it is common usage that we need
>>> to store the exchange persistently, It make sense to provides such kind of
>>> API in Camel as a first class. +1 to provide this kind of API in Camel 3.x.
>>>
>>> Option2. We can also leverage the already exits camel components such as
>>> polling consumer or camel-JPA components to implement a such Message store.
>>> I think IPF shows us a good way to go. In this way we don't need to wait
>>> for the New API from Camel, it could be good solution for Camel 2.x.
>>>
>>> Maybe it is a good time for us to think about Camel 3.0 again.
>>>
>>> Any thoughts?
>>>
>>> Willem
>>>
>>> --
>>> Willem Jiang
>>>
>>> Red Hat, Inc.
>>> FuseSource is now part of Red Hat
>>> Web: http://www.fusesource.com | http://www.redhat.com
>>> Blog: http://willemjiang.blogspot.com (http://willemjiang.blogspot.com/)
>>> (English)
>>> http://jnn.javaeye.com (http://jnn.javaeye.com/) (Chinese)
>>> Twitter: willemjiang
>>> Weibo: willemjiang
>>>
>>>
>>>
>>>
>>>
>>> On Wednesday, October 24, 2012 at 9:43 AM, Hadrian Zbarcea wrote:
>>>
>>> > Hi Christian,
>>> >
>>> > You are correct about the fact that a Message Store is missing (kinda).
>>> > There are a few components that support a message store implementation
>>> > of sorts, as you mentioned. You are also correct that it should be a
>>> > first class citizen.
>>> >
>>> > Practically, an endpoint that supports a polling consumer is a message
>>> > store. What is actually missing is random access via a polling consumer.
>>> > A message store would then work atop any endpoint that supports random
>>> > retrieval (polling). That would create the opportunity to cleanup a bit
>>> > the pollEnrich implementation as well.
>>> >
>>> > My $0.02,
>>> > Hadrian
>>> >
>>> >
>>> > On 10/23/2012 07:18 AM, Christian Ohr wrote:
>>> > > Hi everyone,
>>> > >
>>> > > in the past I came several times across situations that required the
>>> > > one or other kind of Message Store. I noticed that in Camel this
>>> > > doesn't seem to be a "first class citizen" in the sense of a primary
>>> > > architectural concept, which can be applied consistently whereever
>>> > > needed.
>>> > >
>>> > > In short, it might make sense to have a unified, generic, pluggable
>>> > > Message Store (probably more of an "Exchange store") in Camel that
>>> > > consolidates the different approaches and allows to similarly
>>> > > parameterize persistence to various EIP patterns, and can be used
>>> > > independently of EIP patterns as well. Implementations would handle
>>> > > the mapping to the underlying database or file system or NoSQL or
>>> > > whatever.
>>> > > It seems that Spring Integration provides something corresponding
>>> > > (
>>> http://static.springsource.org/spring-integration/reference/htmlsingle/#message-store
>>> ).
>>> > >
>>> > > Message Store implementations are already used by Camel in various
>>> > > places, although using different approaches :
>>> > > * Stream Caching (only file system)
>>> > > * AggregationRepository (used for splitters, multicasts etc., but the
>>> > > interface is not specific at all to aggregation use cases)
>>> > > * IdempotentRepository
>>> > >
>>> > > Message Store is requested for in other places:
>>> > > * Reliable stream resequencing (CAMEL-949)
>>> > > * Persistent Dead Letter Queue (CAMEL-4575)
>>> > >
>>> > > And there might be other areas (seda, bam) that might benefit as well.
>>> > > Maybe generalizing the AggregationRepository is a way to go forward.
>>> > >
>>> > > Side note: The OeHF IPF platform (which is built on top of Camel and
>>> > > partly extends it to the health care domain) has something called a
>>> > > "Flow Manager" (
>>> http://www.openehealth.org/display/ipf2/Flow+management)
>>> > > that is used for tracking exchanges while they are processed or after
>>> > > processing is done, thereby being able to re-insert them into the
>>> > > route. Not that I consider this being a shining example, but it shows
>>> > > that message stores make sense outside their implicit use in EIP
>>> > > processors as well.
>>> > >
>>> > > Looking forward to your opinions!
>>> > >
>>> > > regards
>>> > > Christian
>>> >
>>>
>>>
>>>
>>>
>
>
>
> --
> Claus Ibsen
> -----------------
> Red Hat, Inc.
> FuseSource is now part of Red Hat
> Email: cibsen@redhat.com
> Web: http://fusesource.com
> Twitter: davsclaus
> Blog: http://davsclaus.com
> Author of Camel in Action: http://www.manning.com/ibsen
Re: [DISCUSS] Message Store
Posted by Claus Ibsen <cl...@gmail.com>.
On Wed, Oct 24, 2012 at 8:46 AM, Bengt Rodehav <be...@rodehav.com> wrote:
> I think this is an excellent idea. We're using Camel as the basis for our
> integration platform and a message store is a mandatory component. We need:
>
> - To store all message traffic in order to have an audit trail. Both
> successful and failed exchanges.
> - To provide a certain level of manual retry. That is to get the original
> message from the store and feed it back in the originating route.
> - Flexibility to specify what should be stored (e g what exchange- and
> message properties) and also filter what exchanges should be stored.
>
> Currently we have implemented our own message store but it requires us to
> adapt the routes to make this happen. A more generic way of handling this
> would be terrific.
>
The Message History EIP is scheduled for overhaul/improvement in Camel 3.0.
And in this light we will look at a message store API as well.
> /Bengt
>
> 2012/10/24 Willem jiang <wi...@gmail.com>
>
>> There is a lots of way to implement to Message store.
>> Option1. If we treat is a first class of Camel, we could build up a set of
>> API to let aggregator or tracer to use. As it is common usage that we need
>> to store the exchange persistently, It make sense to provides such kind of
>> API in Camel as a first class. +1 to provide this kind of API in Camel 3.x.
>>
>> Option2. We can also leverage the already exits camel components such as
>> polling consumer or camel-JPA components to implement a such Message store.
>> I think IPF shows us a good way to go. In this way we don't need to wait
>> for the New API from Camel, it could be good solution for Camel 2.x.
>>
>> Maybe it is a good time for us to think about Camel 3.0 again.
>>
>> Any thoughts?
>>
>> Willem
>>
>> --
>> Willem Jiang
>>
>> Red Hat, Inc.
>> FuseSource is now part of Red Hat
>> Web: http://www.fusesource.com | http://www.redhat.com
>> Blog: http://willemjiang.blogspot.com (http://willemjiang.blogspot.com/)
>> (English)
>> http://jnn.javaeye.com (http://jnn.javaeye.com/) (Chinese)
>> Twitter: willemjiang
>> Weibo: willemjiang
>>
>>
>>
>>
>>
>> On Wednesday, October 24, 2012 at 9:43 AM, Hadrian Zbarcea wrote:
>>
>> > Hi Christian,
>> >
>> > You are correct about the fact that a Message Store is missing (kinda).
>> > There are a few components that support a message store implementation
>> > of sorts, as you mentioned. You are also correct that it should be a
>> > first class citizen.
>> >
>> > Practically, an endpoint that supports a polling consumer is a message
>> > store. What is actually missing is random access via a polling consumer.
>> > A message store would then work atop any endpoint that supports random
>> > retrieval (polling). That would create the opportunity to cleanup a bit
>> > the pollEnrich implementation as well.
>> >
>> > My $0.02,
>> > Hadrian
>> >
>> >
>> > On 10/23/2012 07:18 AM, Christian Ohr wrote:
>> > > Hi everyone,
>> > >
>> > > in the past I came several times across situations that required the
>> > > one or other kind of Message Store. I noticed that in Camel this
>> > > doesn't seem to be a "first class citizen" in the sense of a primary
>> > > architectural concept, which can be applied consistently whereever
>> > > needed.
>> > >
>> > > In short, it might make sense to have a unified, generic, pluggable
>> > > Message Store (probably more of an "Exchange store") in Camel that
>> > > consolidates the different approaches and allows to similarly
>> > > parameterize persistence to various EIP patterns, and can be used
>> > > independently of EIP patterns as well. Implementations would handle
>> > > the mapping to the underlying database or file system or NoSQL or
>> > > whatever.
>> > > It seems that Spring Integration provides something corresponding
>> > > (
>> http://static.springsource.org/spring-integration/reference/htmlsingle/#message-store
>> ).
>> > >
>> > > Message Store implementations are already used by Camel in various
>> > > places, although using different approaches :
>> > > * Stream Caching (only file system)
>> > > * AggregationRepository (used for splitters, multicasts etc., but the
>> > > interface is not specific at all to aggregation use cases)
>> > > * IdempotentRepository
>> > >
>> > > Message Store is requested for in other places:
>> > > * Reliable stream resequencing (CAMEL-949)
>> > > * Persistent Dead Letter Queue (CAMEL-4575)
>> > >
>> > > And there might be other areas (seda, bam) that might benefit as well.
>> > > Maybe generalizing the AggregationRepository is a way to go forward.
>> > >
>> > > Side note: The OeHF IPF platform (which is built on top of Camel and
>> > > partly extends it to the health care domain) has something called a
>> > > "Flow Manager" (
>> http://www.openehealth.org/display/ipf2/Flow+management)
>> > > that is used for tracking exchanges while they are processed or after
>> > > processing is done, thereby being able to re-insert them into the
>> > > route. Not that I consider this being a shining example, but it shows
>> > > that message stores make sense outside their implicit use in EIP
>> > > processors as well.
>> > >
>> > > Looking forward to your opinions!
>> > >
>> > > regards
>> > > Christian
>> >
>>
>>
>>
>>
--
Claus Ibsen
-----------------
Red Hat, Inc.
FuseSource is now part of Red Hat
Email: cibsen@redhat.com
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Re: [DISCUSS] Message Store
Posted by Bengt Rodehav <be...@rodehav.com>.
I think this is an excellent idea. We're using Camel as the basis for our
integration platform and a message store is a mandatory component. We need:
- To store all message traffic in order to have an audit trail. Both
successful and failed exchanges.
- To provide a certain level of manual retry. That is to get the original
message from the store and feed it back in the originating route.
- Flexibility to specify what should be stored (e g what exchange- and
message properties) and also filter what exchanges should be stored.
Currently we have implemented our own message store but it requires us to
adapt the routes to make this happen. A more generic way of handling this
would be terrific.
/Bengt
2012/10/24 Willem jiang <wi...@gmail.com>
> There is a lots of way to implement to Message store.
> Option1. If we treat is a first class of Camel, we could build up a set of
> API to let aggregator or tracer to use. As it is common usage that we need
> to store the exchange persistently, It make sense to provides such kind of
> API in Camel as a first class. +1 to provide this kind of API in Camel 3.x.
>
> Option2. We can also leverage the already exits camel components such as
> polling consumer or camel-JPA components to implement a such Message store.
> I think IPF shows us a good way to go. In this way we don't need to wait
> for the New API from Camel, it could be good solution for Camel 2.x.
>
> Maybe it is a good time for us to think about Camel 3.0 again.
>
> Any thoughts?
>
> Willem
>
> --
> Willem Jiang
>
> Red Hat, Inc.
> FuseSource is now part of Red Hat
> Web: http://www.fusesource.com | http://www.redhat.com
> Blog: http://willemjiang.blogspot.com (http://willemjiang.blogspot.com/)
> (English)
> http://jnn.javaeye.com (http://jnn.javaeye.com/) (Chinese)
> Twitter: willemjiang
> Weibo: willemjiang
>
>
>
>
>
> On Wednesday, October 24, 2012 at 9:43 AM, Hadrian Zbarcea wrote:
>
> > Hi Christian,
> >
> > You are correct about the fact that a Message Store is missing (kinda).
> > There are a few components that support a message store implementation
> > of sorts, as you mentioned. You are also correct that it should be a
> > first class citizen.
> >
> > Practically, an endpoint that supports a polling consumer is a message
> > store. What is actually missing is random access via a polling consumer.
> > A message store would then work atop any endpoint that supports random
> > retrieval (polling). That would create the opportunity to cleanup a bit
> > the pollEnrich implementation as well.
> >
> > My $0.02,
> > Hadrian
> >
> >
> > On 10/23/2012 07:18 AM, Christian Ohr wrote:
> > > Hi everyone,
> > >
> > > in the past I came several times across situations that required the
> > > one or other kind of Message Store. I noticed that in Camel this
> > > doesn't seem to be a "first class citizen" in the sense of a primary
> > > architectural concept, which can be applied consistently whereever
> > > needed.
> > >
> > > In short, it might make sense to have a unified, generic, pluggable
> > > Message Store (probably more of an "Exchange store") in Camel that
> > > consolidates the different approaches and allows to similarly
> > > parameterize persistence to various EIP patterns, and can be used
> > > independently of EIP patterns as well. Implementations would handle
> > > the mapping to the underlying database or file system or NoSQL or
> > > whatever.
> > > It seems that Spring Integration provides something corresponding
> > > (
> http://static.springsource.org/spring-integration/reference/htmlsingle/#message-store
> ).
> > >
> > > Message Store implementations are already used by Camel in various
> > > places, although using different approaches :
> > > * Stream Caching (only file system)
> > > * AggregationRepository (used for splitters, multicasts etc., but the
> > > interface is not specific at all to aggregation use cases)
> > > * IdempotentRepository
> > >
> > > Message Store is requested for in other places:
> > > * Reliable stream resequencing (CAMEL-949)
> > > * Persistent Dead Letter Queue (CAMEL-4575)
> > >
> > > And there might be other areas (seda, bam) that might benefit as well.
> > > Maybe generalizing the AggregationRepository is a way to go forward.
> > >
> > > Side note: The OeHF IPF platform (which is built on top of Camel and
> > > partly extends it to the health care domain) has something called a
> > > "Flow Manager" (
> http://www.openehealth.org/display/ipf2/Flow+management)
> > > that is used for tracking exchanges while they are processed or after
> > > processing is done, thereby being able to re-insert them into the
> > > route. Not that I consider this being a shining example, but it shows
> > > that message stores make sense outside their implicit use in EIP
> > > processors as well.
> > >
> > > Looking forward to your opinions!
> > >
> > > regards
> > > Christian
> >
>
>
>
>
Re: [DISCUSS] Message Store
Posted by Willem jiang <wi...@gmail.com>.
There is a lots of way to implement to Message store.
Option1. If we treat is a first class of Camel, we could build up a set of API to let aggregator or tracer to use. As it is common usage that we need to store the exchange persistently, It make sense to provides such kind of API in Camel as a first class. +1 to provide this kind of API in Camel 3.x.
Option2. We can also leverage the already exits camel components such as polling consumer or camel-JPA components to implement a such Message store. I think IPF shows us a good way to go. In this way we don't need to wait for the New API from Camel, it could be good solution for Camel 2.x.
Maybe it is a good time for us to think about Camel 3.0 again.
Any thoughts?
Willem
--
Willem Jiang
Red Hat, Inc.
FuseSource is now part of Red Hat
Web: http://www.fusesource.com | http://www.redhat.com
Blog: http://willemjiang.blogspot.com (http://willemjiang.blogspot.com/) (English)
http://jnn.javaeye.com (http://jnn.javaeye.com/) (Chinese)
Twitter: willemjiang
Weibo: willemjiang
On Wednesday, October 24, 2012 at 9:43 AM, Hadrian Zbarcea wrote:
> Hi Christian,
>
> You are correct about the fact that a Message Store is missing (kinda).
> There are a few components that support a message store implementation
> of sorts, as you mentioned. You are also correct that it should be a
> first class citizen.
>
> Practically, an endpoint that supports a polling consumer is a message
> store. What is actually missing is random access via a polling consumer.
> A message store would then work atop any endpoint that supports random
> retrieval (polling). That would create the opportunity to cleanup a bit
> the pollEnrich implementation as well.
>
> My $0.02,
> Hadrian
>
>
> On 10/23/2012 07:18 AM, Christian Ohr wrote:
> > Hi everyone,
> >
> > in the past I came several times across situations that required the
> > one or other kind of Message Store. I noticed that in Camel this
> > doesn't seem to be a "first class citizen" in the sense of a primary
> > architectural concept, which can be applied consistently whereever
> > needed.
> >
> > In short, it might make sense to have a unified, generic, pluggable
> > Message Store (probably more of an "Exchange store") in Camel that
> > consolidates the different approaches and allows to similarly
> > parameterize persistence to various EIP patterns, and can be used
> > independently of EIP patterns as well. Implementations would handle
> > the mapping to the underlying database or file system or NoSQL or
> > whatever.
> > It seems that Spring Integration provides something corresponding
> > (http://static.springsource.org/spring-integration/reference/htmlsingle/#message-store).
> >
> > Message Store implementations are already used by Camel in various
> > places, although using different approaches :
> > * Stream Caching (only file system)
> > * AggregationRepository (used for splitters, multicasts etc., but the
> > interface is not specific at all to aggregation use cases)
> > * IdempotentRepository
> >
> > Message Store is requested for in other places:
> > * Reliable stream resequencing (CAMEL-949)
> > * Persistent Dead Letter Queue (CAMEL-4575)
> >
> > And there might be other areas (seda, bam) that might benefit as well.
> > Maybe generalizing the AggregationRepository is a way to go forward.
> >
> > Side note: The OeHF IPF platform (which is built on top of Camel and
> > partly extends it to the health care domain) has something called a
> > "Flow Manager" (http://www.openehealth.org/display/ipf2/Flow+management)
> > that is used for tracking exchanges while they are processed or after
> > processing is done, thereby being able to re-insert them into the
> > route. Not that I consider this being a shining example, but it shows
> > that message stores make sense outside their implicit use in EIP
> > processors as well.
> >
> > Looking forward to your opinions!
> >
> > regards
> > Christian
>
Re: [DISCUSS] Message Store
Posted by Hadrian Zbarcea <hz...@gmail.com>.
Hi Christian,
You are correct about the fact that a Message Store is missing (kinda).
There are a few components that support a message store implementation
of sorts, as you mentioned. You are also correct that it should be a
first class citizen.
Practically, an endpoint that supports a polling consumer is a message
store. What is actually missing is random access via a polling consumer.
A message store would then work atop any endpoint that supports random
retrieval (polling). That would create the opportunity to cleanup a bit
the pollEnrich implementation as well.
My $0.02,
Hadrian
On 10/23/2012 07:18 AM, Christian Ohr wrote:
> Hi everyone,
>
> in the past I came several times across situations that required the
> one or other kind of Message Store. I noticed that in Camel this
> doesn't seem to be a "first class citizen" in the sense of a primary
> architectural concept, which can be applied consistently whereever
> needed.
>
> In short, it might make sense to have a unified, generic, pluggable
> Message Store (probably more of an "Exchange store") in Camel that
> consolidates the different approaches and allows to similarly
> parameterize persistence to various EIP patterns, and can be used
> independently of EIP patterns as well. Implementations would handle
> the mapping to the underlying database or file system or NoSQL or
> whatever.
> It seems that Spring Integration provides something corresponding
> (http://static.springsource.org/spring-integration/reference/htmlsingle/#message-store).
>
> Message Store implementations are already used by Camel in various
> places, although using different approaches :
> * Stream Caching (only file system)
> * AggregationRepository (used for splitters, multicasts etc., but the
> interface is not specific at all to aggregation use cases)
> * IdempotentRepository
>
> Message Store is requested for in other places:
> * Reliable stream resequencing (CAMEL-949)
> * Persistent Dead Letter Queue (CAMEL-4575)
>
> And there might be other areas (seda, bam) that might benefit as well.
> Maybe generalizing the AggregationRepository is a way to go forward.
>
> Side note: The OeHF IPF platform (which is built on top of Camel and
> partly extends it to the health care domain) has something called a
> "Flow Manager" (http://www.openehealth.org/display/ipf2/Flow+management)
> that is used for tracking exchanges while they are processed or after
> processing is done, thereby being able to re-insert them into the
> route. Not that I consider this being a shining example, but it shows
> that message stores make sense outside their implicit use in EIP
> processors as well.
>
> Looking forward to your opinions!
>
> regards
> Christian
>