Using Camel as backend in a new Apache "spinoff" project?

[Christofer Dutz <ch...@c-ware.de>]

Hi,

we’re currently discussing potentially using Apache Camel for building a product based on Apache PLC4X, Apache IoTDB to build a Historian solution for industrial use-cases.
We’re planning on making this less a framework, but more a product, based on open-source frameworks and as soon as we have something, to bring it back into Apache as a new project.

Now I brought up the Idea of using Apache Camel as the communication layer between all.

I am admittedly a bit hesitant to introduce Kafka into the game as we aim at building something we can run as installable product, adding Kafka would complicate this, so I’d be happy to use something like Camel for this usecase.

What are your thoughts on this? How do camel routes perform when we’re talking hundreds of thousands to millions of events a minute?

And … anyone interested into joining this initiative? If yes, please ping me off-list and I’ll add you.


Chris

Re: Using Camel as backend in a new Apache "spinoff" project?

[Claus Ibsen <cl...@gmail.com>]

Hi Christofer

Thanks for sharing this with the Camel community.
Good luck and let us know how it goes.



On Wed, Dec 7, 2022 at 1:51 PM Christofer Dutz <ch...@c-ware.de>
wrote:

> Hi,
>
> we’re currently discussing potentially using Apache Camel for building a
> product based on Apache PLC4X, Apache IoTDB to build a Historian solution
> for industrial use-cases.
> We’re planning on making this less a framework, but more a product, based
> on open-source frameworks and as soon as we have something, to bring it
> back into Apache as a new project.
>
> Now I brought up the Idea of using Apache Camel as the communication layer
> between all.
>
> I am admittedly a bit hesitant to introduce Kafka into the game as we aim
> at building something we can run as installable product, adding Kafka would
> complicate this, so I’d be happy to use something like Camel for this
> usecase.
>
> What are your thoughts on this? How do camel routes perform when we’re
> talking hundreds of thousands to millions of events a minute?
>
> And … anyone interested into joining this initiative? If yes, please ping
> me off-list and I’ll add you.
>
>
> Chris
>
>
>

-- 
Claus Ibsen
-----------------
@davsclaus
Camel in Action 2: https://www.manning.com/ibsen2

Re: Using Camel as backend in a new Apache "spinoff" project?

[Christofer Dutz <ch...@c-ware.de>]

Hi Willem,

well I am working in the automation Industry for quite a few yers now.
Unfortunately they don’t seem to understand the concept of buying services, but more buying products.

So, I’ve been seeing everywhere, that the folks in the automation industry are using so-called Historians, which are nothing else than time-series storages for storing production data. Mostly they are doing, because they are required to for some regulations like the FDA rules.

Now besides these things being insanely expensive, I haven’t come across a single instance, that was a pain to access data as these systems usually are super-slow at that. Mainly because people are pumping a loads of data into glorified relational databases.

So, I tried promoting modern IT timeseries solutions like IoTDB to the industry, but they are completely ignorant to a concept, where they can’t install a product and simply run it.

So, my idea is, to start building some of these industry tools, based on our Apache Open Source stack and build drop in replacements. While providing companies to provide commercial support offerings around these solutions. A historian would just be the first of such tools, as I think it’s probably the one where we can have the biggest impact with the least amount of work to put in.

And we’re talking about collecting the production data of (in my case) thousands of PLCs … so it would be pretty high volume.

Hope that sets the stage a bit better.


Chris



From: Willem Jiang <wi...@gmail.com>
Date: Friday, 9. December 2022 at 15:21
To: dev@camel.apache.org <de...@camel.apache.org>
Subject: Re: Using Camel as backend in a new Apache "spinoff" project?
I used to address the camel performance issue 6 years ago, which led
me to find a use case for collecting millions of performance data as a
SaaS service.
In that case, camel leverages Kafka to persist received messages.  I
don't think the in-memory queue could address the problem.
@ Chris,  Can you share more information about your use case?


Willem Jiang

Twitter: willemjiang
Weibo: 姜宁willem


On Wed, Dec 7, 2022 at 11:16 PM Christofer Dutz
<ch...@c-ware.de> wrote:
>
> Hi Zheng,
>
> We’re currently discussing multiple scenarios. In the “all in one” if would definitely make sense to do that and to improve the Camel PLC4X component while at it.
>
> In general, we’re thinking of an application, that starts an IoTDB server embedded as well as something that pumps data into it and some sort of API frontend with which industry solutions can communicate with.
>
> But it’s not even decided IF we use Camel … just wanted to ask you folks here, if you think it’s a good idea and if it’s a good idea, if anyone wants to join in 😉
>
> Chris
>
>
> From: Zheng Feng <zf...@redhat.com>
> Date: Wednesday, 7. December 2022 at 14:31
> To: dev@camel.apache.org <de...@camel.apache.org>
> Subject: Re: Using Camel as backend in a new Apache "spinoff" project?
> It looks interesting and will it run a camel router with PLC4X component as
> an agent of IoTDB to collect data and send them directly to a server?
>
> On Wed, Dec 7, 2022 at 8:59 PM Christofer Dutz <ch...@c-ware.de>
> wrote:
>
> > Hi,
> >
> > we’re currently discussing potentially using Apache Camel for building a
> > product based on Apache PLC4X, Apache IoTDB to build a Historian solution
> > for industrial use-cases.
> > We’re planning on making this less a framework, but more a product, based
> > on open-source frameworks and as soon as we have something, to bring it
> > back into Apache as a new project.
> >
> > Now I brought up the Idea of using Apache Camel as the communication layer
> > between all.
> >
> > I am admittedly a bit hesitant to introduce Kafka into the game as we aim
> > at building something we can run as installable product, adding Kafka would
> > complicate this, so I’d be happy to use something like Camel for this
> > usecase.
> >
> > What are your thoughts on this? How do camel routes perform when we’re
> > talking hundreds of thousands to millions of events a minute?
> >
> > And … anyone interested into joining this initiative? If yes, please ping
> > me off-list and I’ll add you.
> >
> >
> > Chris
> >
> >
> >

Re: Using Camel as backend in a new Apache "spinoff" project?

[Willem Jiang <wi...@gmail.com>]

I used to address the camel performance issue 6 years ago, which led
me to find a use case for collecting millions of performance data as a
SaaS service.
In that case, camel leverages Kafka to persist received messages.  I
don't think the in-memory queue could address the problem.
@ Chris,  Can you share more information about your use case?


Willem Jiang

Twitter: willemjiang
Weibo: 姜宁willem


On Wed, Dec 7, 2022 at 11:16 PM Christofer Dutz
<ch...@c-ware.de> wrote:
>
> Hi Zheng,
>
> We’re currently discussing multiple scenarios. In the “all in one” if would definitely make sense to do that and to improve the Camel PLC4X component while at it.
>
> In general, we’re thinking of an application, that starts an IoTDB server embedded as well as something that pumps data into it and some sort of API frontend with which industry solutions can communicate with.
>
> But it’s not even decided IF we use Camel … just wanted to ask you folks here, if you think it’s a good idea and if it’s a good idea, if anyone wants to join in 😉
>
> Chris
>
>
> From: Zheng Feng <zf...@redhat.com>
> Date: Wednesday, 7. December 2022 at 14:31
> To: dev@camel.apache.org <de...@camel.apache.org>
> Subject: Re: Using Camel as backend in a new Apache "spinoff" project?
> It looks interesting and will it run a camel router with PLC4X component as
> an agent of IoTDB to collect data and send them directly to a server?
>
> On Wed, Dec 7, 2022 at 8:59 PM Christofer Dutz <ch...@c-ware.de>
> wrote:
>
> > Hi,
> >
> > we’re currently discussing potentially using Apache Camel for building a
> > product based on Apache PLC4X, Apache IoTDB to build a Historian solution
> > for industrial use-cases.
> > We’re planning on making this less a framework, but more a product, based
> > on open-source frameworks and as soon as we have something, to bring it
> > back into Apache as a new project.
> >
> > Now I brought up the Idea of using Apache Camel as the communication layer
> > between all.
> >
> > I am admittedly a bit hesitant to introduce Kafka into the game as we aim
> > at building something we can run as installable product, adding Kafka would
> > complicate this, so I’d be happy to use something like Camel for this
> > usecase.
> >
> > What are your thoughts on this? How do camel routes perform when we’re
> > talking hundreds of thousands to millions of events a minute?
> >
> > And … anyone interested into joining this initiative? If yes, please ping
> > me off-list and I’ll add you.
> >
> >
> > Chris
> >
> >
> >

Re: Using Camel as backend in a new Apache "spinoff" project?

[Christofer Dutz <ch...@c-ware.de>]

Hi Zheng,

We’re currently discussing multiple scenarios. In the “all in one” if would definitely make sense to do that and to improve the Camel PLC4X component while at it.

In general, we’re thinking of an application, that starts an IoTDB server embedded as well as something that pumps data into it and some sort of API frontend with which industry solutions can communicate with.

But it’s not even decided IF we use Camel … just wanted to ask you folks here, if you think it’s a good idea and if it’s a good idea, if anyone wants to join in 😉

Chris


From: Zheng Feng <zf...@redhat.com>
Date: Wednesday, 7. December 2022 at 14:31
To: dev@camel.apache.org <de...@camel.apache.org>
Subject: Re: Using Camel as backend in a new Apache "spinoff" project?
It looks interesting and will it run a camel router with PLC4X component as
an agent of IoTDB to collect data and send them directly to a server?

On Wed, Dec 7, 2022 at 8:59 PM Christofer Dutz <ch...@c-ware.de>
wrote:

> Hi,
>
> we’re currently discussing potentially using Apache Camel for building a
> product based on Apache PLC4X, Apache IoTDB to build a Historian solution
> for industrial use-cases.
> We’re planning on making this less a framework, but more a product, based
> on open-source frameworks and as soon as we have something, to bring it
> back into Apache as a new project.
>
> Now I brought up the Idea of using Apache Camel as the communication layer
> between all.
>
> I am admittedly a bit hesitant to introduce Kafka into the game as we aim
> at building something we can run as installable product, adding Kafka would
> complicate this, so I’d be happy to use something like Camel for this
> usecase.
>
> What are your thoughts on this? How do camel routes perform when we’re
> talking hundreds of thousands to millions of events a minute?
>
> And … anyone interested into joining this initiative? If yes, please ping
> me off-list and I’ll add you.
>
>
> Chris
>
>
>

Re: Using Camel as backend in a new Apache "spinoff" project?

[Zheng Feng <zf...@redhat.com>]

It looks interesting and will it run a camel router with PLC4X component as
an agent of IoTDB to collect data and send them directly to a server?

On Wed, Dec 7, 2022 at 8:59 PM Christofer Dutz <ch...@c-ware.de>
wrote:

> Hi,
>
> we’re currently discussing potentially using Apache Camel for building a
> product based on Apache PLC4X, Apache IoTDB to build a Historian solution
> for industrial use-cases.
> We’re planning on making this less a framework, but more a product, based
> on open-source frameworks and as soon as we have something, to bring it
> back into Apache as a new project.
>
> Now I brought up the Idea of using Apache Camel as the communication layer
> between all.
>
> I am admittedly a bit hesitant to introduce Kafka into the game as we aim
> at building something we can run as installable product, adding Kafka would
> complicate this, so I’d be happy to use something like Camel for this
> usecase.
>
> What are your thoughts on this? How do camel routes perform when we’re
> talking hundreds of thousands to millions of events a minute?
>
> And … anyone interested into joining this initiative? If yes, please ping
> me off-list and I’ll add you.
>
>
> Chris
>
>
>