You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by Tom Fornoville <to...@roots.be> on 2014/01/08 16:14:07 UTC
Distributed AggregationRepository
Hi camel-users,
Since we're going to run our system distributed (multiple containers in
Fuse Fabric) we need an alternative to the default
MemoryAggregationRepository.
Although we found it awkward that the Hazelcast component only offers an
IdempotentRepository and no AggregationRepository we found an
implementation here: https://github.com/smecsia/camel-hazelcast. We tried
that but sometimes we get strange TimeoutExceptions and we're not
experienced enough with Hazelcast to determine the root-cause.
Some extra searching on the Camel site led me to HawtDB (
http://camel.apache.org/hawtdb.html).
So I now have 3 questions:
1) does HawtDB work out-of-the-box in a distributed environment (provided
that you can place the file on a shared location), and can it recover
wihtout extra manual steps?
2) can you place multiple repositories in a single file or do you need a
separate file per repository (we have 5 aggregators on our system, each one
needing its own repository)
3) are there any other proven and documented implementations of
AggregationRepository that work in distributed environments?
Thanks in advance,
Tom
Re: Distributed AggregationRepository
Posted by Raul Kripalani <ra...@evosent.com>.
Hi Claus,
On Thu, Jan 9, 2014 at 8:42 AM, Claus Ibsen <cl...@gmail.com> wrote:
> Hmm I am not aware of such.
>
> camel-levedb is using same principle as the leveldb store in AMQ
> http://activemq.apache.org/leveldb-store.html
>
> And allows concurrent and distributed reads/writes to the shared
> store. eg use a shared LAN for the file.
>
LevelDb doesn't, per-se, as expressed on their page [1]:
"Only a single process (possibly multi-threaded) can access a particular
database at a time."
To my understanding, the "Replicated LevelDB Store" developed in ActiveMQ
is simply that: a mechanism to replicate a given DB to remote nodes. But it
doesn't mean that different clients/nodes are opening to the same DB all at
once. That's not even possible with LevelDB because it is an embedded DB,
not exposed on the network directly. You'd have to implement your own layer
to provide this connectivity, or use something like multilevel [2].
[1] https://code.google.com/p/leveldb/
[2] https://github.com/juliangruber/multilevel
*Raúl Kripalani*
Apache Camel PMC Member & Committer | Enterprise Architect, Open Source
Integration specialist
http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani
http://blog.raulkr.net | twitter: @raulvk
Re: Distributed AggregationRepository
Posted by Claus Ibsen <cl...@gmail.com>.
On Wed, Jan 8, 2014 at 5:41 PM, Raul Kripalani <ra...@evosent.com> wrote:
> Careful, HawtDB is deprecated, as advertised in the camel-hawtdb component
> page. Unfortunately, the warning doesn't render in a shaded box any longer
> (due to some recent Confluence migration), so it's easy to miss it. Sorry
> about that.
>
> LevelDB is recommended instead, but the technology has two limitation: it
> runs embedded and only one physical process can open the data files
> simultaneously. In other words, it does not support distributed access OOTB.
>
Hmm I am not aware of such.
camel-levedb is using same principle as the leveldb store in AMQ
http://activemq.apache.org/leveldb-store.html
And allows concurrent and distributed reads/writes to the shared
store. eg use a shared LAN for the file.
> The old-school advice would be to use the JDBC Aggregation Repository along
> with either a SQL DB or a NoSQL DB from whom JDBC support exists. For
> example, there are some experimental projects providing JDBC drivers for
> MongoDB, or OrientDB has some support for JDBC too.
> Alternatively, we would be extremely happy if you contributed a new NoSQL
> Aggregation Repository. I can see much benefit in incorporating a MongoDB
> Aggregation Repository to the camel-mongodb component which uses the native
> Java driver.
>
> Regards,
>
> *Raúl Kripalani*
> Apache Camel PMC Member & Committer | Enterprise Architect, Open Source
> Integration specialist
> http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani
> http://blog.raulkr.net | twitter: @raulvk
>
> On Wed, Jan 8, 2014 at 3:14 PM, Tom Fornoville <to...@roots.be>wrote:
>
>> Hi camel-users,
>>
>> Since we're going to run our system distributed (multiple containers in
>> Fuse Fabric) we need an alternative to the default
>> MemoryAggregationRepository.
>>
>> Although we found it awkward that the Hazelcast component only offers an
>> IdempotentRepository and no AggregationRepository we found an
>> implementation here: https://github.com/smecsia/camel-hazelcast. We tried
>> that but sometimes we get strange TimeoutExceptions and we're not
>> experienced enough with Hazelcast to determine the root-cause.
>>
>> Some extra searching on the Camel site led me to HawtDB (
>> http://camel.apache.org/hawtdb.html).
>> So I now have 3 questions:
>> 1) does HawtDB work out-of-the-box in a distributed environment (provided
>> that you can place the file on a shared location), and can it recover
>> wihtout extra manual steps?
>>
>> 2) can you place multiple repositories in a single file or do you need a
>> separate file per repository (we have 5 aggregators on our system, each one
>> needing its own repository)
>>
>> 3) are there any other proven and documented implementations of
>> AggregationRepository that work in distributed environments?
>>
>> Thanks in advance,
>> Tom
>>
--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: cibsen@redhat.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Make your Camel applications look hawt, try: http://hawt.io
Re: Distributed AggregationRepository
Posted by Raul Kripalani <ra...@evosent.com>.
Careful, HawtDB is deprecated, as advertised in the camel-hawtdb component
page. Unfortunately, the warning doesn't render in a shaded box any longer
(due to some recent Confluence migration), so it's easy to miss it. Sorry
about that.
LevelDB is recommended instead, but the technology has two limitation: it
runs embedded and only one physical process can open the data files
simultaneously. In other words, it does not support distributed access OOTB.
The old-school advice would be to use the JDBC Aggregation Repository along
with either a SQL DB or a NoSQL DB from whom JDBC support exists. For
example, there are some experimental projects providing JDBC drivers for
MongoDB, or OrientDB has some support for JDBC too.
Alternatively, we would be extremely happy if you contributed a new NoSQL
Aggregation Repository. I can see much benefit in incorporating a MongoDB
Aggregation Repository to the camel-mongodb component which uses the native
Java driver.
Regards,
*Raúl Kripalani*
Apache Camel PMC Member & Committer | Enterprise Architect, Open Source
Integration specialist
http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani
http://blog.raulkr.net | twitter: @raulvk
On Wed, Jan 8, 2014 at 3:14 PM, Tom Fornoville <to...@roots.be>wrote:
> Hi camel-users,
>
> Since we're going to run our system distributed (multiple containers in
> Fuse Fabric) we need an alternative to the default
> MemoryAggregationRepository.
>
> Although we found it awkward that the Hazelcast component only offers an
> IdempotentRepository and no AggregationRepository we found an
> implementation here: https://github.com/smecsia/camel-hazelcast. We tried
> that but sometimes we get strange TimeoutExceptions and we're not
> experienced enough with Hazelcast to determine the root-cause.
>
> Some extra searching on the Camel site led me to HawtDB (
> http://camel.apache.org/hawtdb.html).
> So I now have 3 questions:
> 1) does HawtDB work out-of-the-box in a distributed environment (provided
> that you can place the file on a shared location), and can it recover
> wihtout extra manual steps?
>
> 2) can you place multiple repositories in a single file or do you need a
> separate file per repository (we have 5 aggregators on our system, each one
> needing its own repository)
>
> 3) are there any other proven and documented implementations of
> AggregationRepository that work in distributed environments?
>
> Thanks in advance,
> Tom
>
Re: Distributed AggregationRepository
Posted by Henryk Konsek <he...@gmail.com>.
Hi Tom,
> Since we're going to run our system distributed (multiple containers in
> Fuse Fabric) we need an alternative to the default
> MemoryAggregationRepository.
As Raul said, JDBC is the only real distributed option for a while.
Keep in mind however that even if SQL performance might not be the
best option, you can start with JDBC repository and relatively easy
switch to the other implementation later. Life time of data kept in
aggregation repositories is very short, so there is no migration
burden associated with the repository replacement. This might be an
option to start with something and buy some time to create
NoSQL/DistributedMemoryGrid/JGroups based solution that will scale
better.
Cheers.
--
Henryk Konsek
http://henryk-konsek.blogspot.com