You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@camel.apache.org by Nicola Ferraro <ni...@gmail.com> on 2018/07/12 23:30:11 UTC
A "Kamel" crazy idea
Hi Cameleers,
it's now passed some time since I started thinking about a new project that
we can begin here at Apache Camel, and I'd like to have your opinion.
We've already been targeting cloud-native applications with Camel,
especially on top of Kubernetes, that is becoming "the standard" cloud
platform. But writing a Camel integration and running it on Kubernetes
requires some effort: choosing the base platform (spring-boot, karaf,
simple main?), adding health checks (actuator?), packaging a docker image
and creating the Kubernetes resources (fabric8-maven-plugin, helm?),
publishing the image on a docker registry, then finally deploying the
resources on a Kubernetes cluster.
The resulting integration container is then far from being optimal from a
resource consumption point of view: it is likely that a Camel Spring-Boot
application will require at least 200MB of RAM and also some CPU shares
because of polling threads used by many components.
In case people use a CI/CD pipeline, it will take also a long time to get
from a code update to having a Kubernetes POD up and running.
Apart from compilation and image push/pull time, also startup time is often
~10 seconds for Camel + Spring-Boot in a container with standard limits on
resources, making it difficult to propose this combination for "serverless
integration" (this term is becoming increasingly more popular).
So, my proposal is to start to investigate a "more cloud-native" approach
to integration: *making Camel integrations first-class citizens in
Kubernetes, and making them super fast and lightweight.*
We can base the project on Kubernetes Custom Resource Definitions (CRD)
<https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/>,
for example a Integration CRD and have a Kubernetes "operator"
<https://coreos.com/operators/> taking care of:
- Optimizing the integration that we want to run
- Packaging in a container
- Running it on Kubernetes
- Managing its entire lifecycle
A Kubernetes-native integration may look like:
-------------------
kind: "Integration"
apiVersion: "camel.apache.org/v1alpha1"
metadata:
name: "example"
spec:
replicas: 1
routes:
- id: timer
route:
- type: endpoint
uri: timer:tick
- type: endpoint
uri: log:info
-------------------
For those who are not familiar with Kubernetes resources, this kind of
YAML/JSON resource definitions are really common.
The example route is embedded in the Kubernetes resource declaration and
follows a basic "flow DSL". We may start from a basic one and evolve it as
new requirements arrive from the community.
I've made a very simple (but working) POC here:
https://github.com/nicolaferraro/integration-operator.
This idea of a "Cloud-Native Camel" on Kubernetes (project codename can be "
*Kamel*", if you like it :D), will be an enabler for a lot of nice features.
For example, we can propose "Kamel" as "ideal" platform for "serverless
integration" (I see many people reinventing the wheel out there): the
operator can reduce resource consumption of a single integration by
optimizing the runtime and also pause/resume integrations when they are not
used, that is the basic idea behind "serverless" (e.g. think to
HTTP-triggered integrations, but not only).
Focusing on serverless will bring more emphasis on push-based notifications
(webhooks, cloud events <https://cloudevents.io/>), that are rarely used in
Camel components, that prefer a poll based approach being it simpler to use
in classic deployments, but not so good in the cloud, where more resources
become higher direct costs for the users.
The presence of the simplified DSL enables also experimenting on "*reduced*
subsets of Camel" implemented in languages other than Java, for example one
language that has a reactive approach on thread scheduling and a really low
memory footprint, like Go.
But apart from this kind of experiments (that are valid IMO), the "Kamel"
optimizer will have free room to choose the right platform for the
integration that the user wants to run, including, in the future, doing AOT
compilation using Graal/VM (less memory, faster startup) if the features
(components) used in the integration are supporting it (maybe we can add
AOT compilation in the roadmap for Camel 3).
A silly optimization: integrations starting from "timer:..." may be
scheduled directly with Kubernetes CronJobs, so they will consume resources
only when actually running.
Being the final integrations lightweight and being the DSL
language-independent, we may see a increased adoption of Camel also as
agile integration layer for not-only-java applications (both "cloud" and
"serverless" applications).
I'm the first one that would like to work on a project ilke this. I've
worked on many Kubernetes/Openshift based applications and frameworks in
the past years, also on operators and CRDs, and I think this way of
redesigning integrations has a lot of potential.
Integrations will not be necessarily limited to the simplified DSL, but we
can add extension points for scripting and even custom libraries (although
limiting the freedom of the optimizer).
The most important thing: it may become a great project, since it's driven
by a great community.
So, what do you think? Is it crazy enough?
Nicola
Re: A "Kamel" crazy idea
Posted by Zoran Regvart <zo...@regvart.com>.
Hi Nicola,
+1 on the idea and the great write-up, seems to me like something that
we should definitely explore further.
I also like the "Kamel" name :)
zoran
On Fri, Jul 13, 2018 at 1:30 AM, Nicola Ferraro <ni...@gmail.com> wrote:
> Hi Cameleers,
> it's now passed some time since I started thinking about a new project that
> we can begin here at Apache Camel, and I'd like to have your opinion.
>
> We've already been targeting cloud-native applications with Camel,
> especially on top of Kubernetes, that is becoming "the standard" cloud
> platform. But writing a Camel integration and running it on Kubernetes
> requires some effort: choosing the base platform (spring-boot, karaf,
> simple main?), adding health checks (actuator?), packaging a docker image
> and creating the Kubernetes resources (fabric8-maven-plugin, helm?),
> publishing the image on a docker registry, then finally deploying the
> resources on a Kubernetes cluster.
>
> The resulting integration container is then far from being optimal from a
> resource consumption point of view: it is likely that a Camel Spring-Boot
> application will require at least 200MB of RAM and also some CPU shares
> because of polling threads used by many components.
>
> In case people use a CI/CD pipeline, it will take also a long time to get
> from a code update to having a Kubernetes POD up and running.
> Apart from compilation and image push/pull time, also startup time is often
> ~10 seconds for Camel + Spring-Boot in a container with standard limits on
> resources, making it difficult to propose this combination for "serverless
> integration" (this term is becoming increasingly more popular).
>
> So, my proposal is to start to investigate a "more cloud-native" approach
> to integration: *making Camel integrations first-class citizens in
> Kubernetes, and making them super fast and lightweight.*
>
> We can base the project on Kubernetes Custom Resource Definitions (CRD)
> <https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/>,
> for example a Integration CRD and have a Kubernetes "operator"
> <https://coreos.com/operators/> taking care of:
> - Optimizing the integration that we want to run
> - Packaging in a container
> - Running it on Kubernetes
> - Managing its entire lifecycle
>
> A Kubernetes-native integration may look like:
>
> -------------------
> kind: "Integration"
> apiVersion: "camel.apache.org/v1alpha1"
> metadata:
> name: "example"
> spec:
> replicas: 1
> routes:
> - id: timer
> route:
> - type: endpoint
> uri: timer:tick
> - type: endpoint
> uri: log:info
> -------------------
>
> For those who are not familiar with Kubernetes resources, this kind of
> YAML/JSON resource definitions are really common.
> The example route is embedded in the Kubernetes resource declaration and
> follows a basic "flow DSL". We may start from a basic one and evolve it as
> new requirements arrive from the community.
>
> I've made a very simple (but working) POC here:
> https://github.com/nicolaferraro/integration-operator.
>
> This idea of a "Cloud-Native Camel" on Kubernetes (project codename can be "
> *Kamel*", if you like it :D), will be an enabler for a lot of nice features.
>
> For example, we can propose "Kamel" as "ideal" platform for "serverless
> integration" (I see many people reinventing the wheel out there): the
> operator can reduce resource consumption of a single integration by
> optimizing the runtime and also pause/resume integrations when they are not
> used, that is the basic idea behind "serverless" (e.g. think to
> HTTP-triggered integrations, but not only).
> Focusing on serverless will bring more emphasis on push-based notifications
> (webhooks, cloud events <https://cloudevents.io/>), that are rarely used in
> Camel components, that prefer a poll based approach being it simpler to use
> in classic deployments, but not so good in the cloud, where more resources
> become higher direct costs for the users.
>
> The presence of the simplified DSL enables also experimenting on "*reduced*
> subsets of Camel" implemented in languages other than Java, for example one
> language that has a reactive approach on thread scheduling and a really low
> memory footprint, like Go.
>
> But apart from this kind of experiments (that are valid IMO), the "Kamel"
> optimizer will have free room to choose the right platform for the
> integration that the user wants to run, including, in the future, doing AOT
> compilation using Graal/VM (less memory, faster startup) if the features
> (components) used in the integration are supporting it (maybe we can add
> AOT compilation in the roadmap for Camel 3).
> A silly optimization: integrations starting from "timer:..." may be
> scheduled directly with Kubernetes CronJobs, so they will consume resources
> only when actually running.
>
> Being the final integrations lightweight and being the DSL
> language-independent, we may see a increased adoption of Camel also as
> agile integration layer for not-only-java applications (both "cloud" and
> "serverless" applications).
>
> I'm the first one that would like to work on a project ilke this. I've
> worked on many Kubernetes/Openshift based applications and frameworks in
> the past years, also on operators and CRDs, and I think this way of
> redesigning integrations has a lot of potential.
>
> Integrations will not be necessarily limited to the simplified DSL, but we
> can add extension points for scripting and even custom libraries (although
> limiting the freedom of the optimizer).
>
> The most important thing: it may become a great project, since it's driven
> by a great community.
>
> So, what do you think? Is it crazy enough?
>
> Nicola
--
Zoran Regvart
R: Re: A "Kamel" crazy idea
Posted by Andrea Cosentino <an...@yahoo.com.INVALID>.
+1
Great idea!
Il ven, 13 lug, 2018 alle 7:53, Claus Ibsen<cl...@gmail.com> ha scritto: +1
Great idea and love the Kamel name.
On Fri, Jul 13, 2018 at 1:30 AM, Nicola Ferraro <ni...@gmail.com> wrote:
> Hi Cameleers,
> it's now passed some time since I started thinking about a new project that
> we can begin here at Apache Camel, and I'd like to have your opinion.
>
> We've already been targeting cloud-native applications with Camel,
> especially on top of Kubernetes, that is becoming "the standard" cloud
> platform. But writing a Camel integration and running it on Kubernetes
> requires some effort: choosing the base platform (spring-boot, karaf,
> simple main?), adding health checks (actuator?), packaging a docker image
> and creating the Kubernetes resources (fabric8-maven-plugin, helm?),
> publishing the image on a docker registry, then finally deploying the
> resources on a Kubernetes cluster.
>
> The resulting integration container is then far from being optimal from a
> resource consumption point of view: it is likely that a Camel Spring-Boot
> application will require at least 200MB of RAM and also some CPU shares
> because of polling threads used by many components.
>
> In case people use a CI/CD pipeline, it will take also a long time to get
> from a code update to having a Kubernetes POD up and running.
> Apart from compilation and image push/pull time, also startup time is often
> ~10 seconds for Camel + Spring-Boot in a container with standard limits on
> resources, making it difficult to propose this combination for "serverless
> integration" (this term is becoming increasingly more popular).
>
> So, my proposal is to start to investigate a "more cloud-native" approach
> to integration: *making Camel integrations first-class citizens in
> Kubernetes, and making them super fast and lightweight.*
>
> We can base the project on Kubernetes Custom Resource Definitions (CRD)
> <https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/>,
> for example a Integration CRD and have a Kubernetes "operator"
> <https://coreos.com/operators/> taking care of:
> - Optimizing the integration that we want to run
> - Packaging in a container
> - Running it on Kubernetes
> - Managing its entire lifecycle
>
> A Kubernetes-native integration may look like:
>
> -------------------
> kind: "Integration"
> apiVersion: "camel.apache.org/v1alpha1"
> metadata:
> name: "example"
> spec:
> replicas: 1
> routes:
> - id: timer
> route:
> - type: endpoint
> uri: timer:tick
> - type: endpoint
> uri: log:info
> -------------------
>
> For those who are not familiar with Kubernetes resources, this kind of
> YAML/JSON resource definitions are really common.
> The example route is embedded in the Kubernetes resource declaration and
> follows a basic "flow DSL". We may start from a basic one and evolve it as
> new requirements arrive from the community.
>
> I've made a very simple (but working) POC here:
> https://github.com/nicolaferraro/integration-operator.
>
> This idea of a "Cloud-Native Camel" on Kubernetes (project codename can be "
> *Kamel*", if you like it :D), will be an enabler for a lot of nice features.
>
> For example, we can propose "Kamel" as "ideal" platform for "serverless
> integration" (I see many people reinventing the wheel out there): the
> operator can reduce resource consumption of a single integration by
> optimizing the runtime and also pause/resume integrations when they are not
> used, that is the basic idea behind "serverless" (e.g. think to
> HTTP-triggered integrations, but not only).
> Focusing on serverless will bring more emphasis on push-based notifications
> (webhooks, cloud events <https://cloudevents.io/>), that are rarely used in
> Camel components, that prefer a poll based approach being it simpler to use
> in classic deployments, but not so good in the cloud, where more resources
> become higher direct costs for the users.
>
> The presence of the simplified DSL enables also experimenting on "*reduced*
> subsets of Camel" implemented in languages other than Java, for example one
> language that has a reactive approach on thread scheduling and a really low
> memory footprint, like Go.
>
> But apart from this kind of experiments (that are valid IMO), the "Kamel"
> optimizer will have free room to choose the right platform for the
> integration that the user wants to run, including, in the future, doing AOT
> compilation using Graal/VM (less memory, faster startup) if the features
> (components) used in the integration are supporting it (maybe we can add
> AOT compilation in the roadmap for Camel 3).
> A silly optimization: integrations starting from "timer:..." may be
> scheduled directly with Kubernetes CronJobs, so they will consume resources
> only when actually running.
>
> Being the final integrations lightweight and being the DSL
> language-independent, we may see a increased adoption of Camel also as
> agile integration layer for not-only-java applications (both "cloud" and
> "serverless" applications).
>
> I'm the first one that would like to work on a project ilke this. I've
> worked on many Kubernetes/Openshift based applications and frameworks in
> the past years, also on operators and CRDs, and I think this way of
> redesigning integrations has a lot of potential.
>
> Integrations will not be necessarily limited to the simplified DSL, but we
> can add extension points for scripting and even custom libraries (although
> limiting the freedom of the optimizer).
>
> The most important thing: it may become a great project, since it's driven
> by a great community.
>
> So, what do you think? Is it crazy enough?
>
> Nicola
--
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2
Re: A "Kamel" crazy idea
Posted by Claus Ibsen <cl...@gmail.com>.
+1
Great idea and love the Kamel name.
On Fri, Jul 13, 2018 at 1:30 AM, Nicola Ferraro <ni...@gmail.com> wrote:
> Hi Cameleers,
> it's now passed some time since I started thinking about a new project that
> we can begin here at Apache Camel, and I'd like to have your opinion.
>
> We've already been targeting cloud-native applications with Camel,
> especially on top of Kubernetes, that is becoming "the standard" cloud
> platform. But writing a Camel integration and running it on Kubernetes
> requires some effort: choosing the base platform (spring-boot, karaf,
> simple main?), adding health checks (actuator?), packaging a docker image
> and creating the Kubernetes resources (fabric8-maven-plugin, helm?),
> publishing the image on a docker registry, then finally deploying the
> resources on a Kubernetes cluster.
>
> The resulting integration container is then far from being optimal from a
> resource consumption point of view: it is likely that a Camel Spring-Boot
> application will require at least 200MB of RAM and also some CPU shares
> because of polling threads used by many components.
>
> In case people use a CI/CD pipeline, it will take also a long time to get
> from a code update to having a Kubernetes POD up and running.
> Apart from compilation and image push/pull time, also startup time is often
> ~10 seconds for Camel + Spring-Boot in a container with standard limits on
> resources, making it difficult to propose this combination for "serverless
> integration" (this term is becoming increasingly more popular).
>
> So, my proposal is to start to investigate a "more cloud-native" approach
> to integration: *making Camel integrations first-class citizens in
> Kubernetes, and making them super fast and lightweight.*
>
> We can base the project on Kubernetes Custom Resource Definitions (CRD)
> <https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/>,
> for example a Integration CRD and have a Kubernetes "operator"
> <https://coreos.com/operators/> taking care of:
> - Optimizing the integration that we want to run
> - Packaging in a container
> - Running it on Kubernetes
> - Managing its entire lifecycle
>
> A Kubernetes-native integration may look like:
>
> -------------------
> kind: "Integration"
> apiVersion: "camel.apache.org/v1alpha1"
> metadata:
> name: "example"
> spec:
> replicas: 1
> routes:
> - id: timer
> route:
> - type: endpoint
> uri: timer:tick
> - type: endpoint
> uri: log:info
> -------------------
>
> For those who are not familiar with Kubernetes resources, this kind of
> YAML/JSON resource definitions are really common.
> The example route is embedded in the Kubernetes resource declaration and
> follows a basic "flow DSL". We may start from a basic one and evolve it as
> new requirements arrive from the community.
>
> I've made a very simple (but working) POC here:
> https://github.com/nicolaferraro/integration-operator.
>
> This idea of a "Cloud-Native Camel" on Kubernetes (project codename can be "
> *Kamel*", if you like it :D), will be an enabler for a lot of nice features.
>
> For example, we can propose "Kamel" as "ideal" platform for "serverless
> integration" (I see many people reinventing the wheel out there): the
> operator can reduce resource consumption of a single integration by
> optimizing the runtime and also pause/resume integrations when they are not
> used, that is the basic idea behind "serverless" (e.g. think to
> HTTP-triggered integrations, but not only).
> Focusing on serverless will bring more emphasis on push-based notifications
> (webhooks, cloud events <https://cloudevents.io/>), that are rarely used in
> Camel components, that prefer a poll based approach being it simpler to use
> in classic deployments, but not so good in the cloud, where more resources
> become higher direct costs for the users.
>
> The presence of the simplified DSL enables also experimenting on "*reduced*
> subsets of Camel" implemented in languages other than Java, for example one
> language that has a reactive approach on thread scheduling and a really low
> memory footprint, like Go.
>
> But apart from this kind of experiments (that are valid IMO), the "Kamel"
> optimizer will have free room to choose the right platform for the
> integration that the user wants to run, including, in the future, doing AOT
> compilation using Graal/VM (less memory, faster startup) if the features
> (components) used in the integration are supporting it (maybe we can add
> AOT compilation in the roadmap for Camel 3).
> A silly optimization: integrations starting from "timer:..." may be
> scheduled directly with Kubernetes CronJobs, so they will consume resources
> only when actually running.
>
> Being the final integrations lightweight and being the DSL
> language-independent, we may see a increased adoption of Camel also as
> agile integration layer for not-only-java applications (both "cloud" and
> "serverless" applications).
>
> I'm the first one that would like to work on a project ilke this. I've
> worked on many Kubernetes/Openshift based applications and frameworks in
> the past years, also on operators and CRDs, and I think this way of
> redesigning integrations has a lot of potential.
>
> Integrations will not be necessarily limited to the simplified DSL, but we
> can add extension points for scripting and even custom libraries (although
> limiting the freedom of the optimizer).
>
> The most important thing: it may become a great project, since it's driven
> by a great community.
>
> So, what do you think? Is it crazy enough?
>
> Nicola
--
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2
Re: A "Kamel" crazy idea
Posted by Luca Burgazzoli <lb...@gmail.com>.
Of course you need to port them to go lang, the project is not intended to
be a drop in replacement for camel but a companion for integration that do
not require all the power and flexibility the jvm implementation provide
On Fri, 13 Jul 2018 at 08:55, Willem Jiang <wi...@gmail.com> wrote:
> Hi Luca,
>
> You are building Go version of Camel by implement the DSL in Go.
> Now we can start the camel context by using camel-go to load the route,
> but here are some missing points that I want fill such as how to reuse the
> Camel components?
>
>
>
>
>
> Willem Jiang
>
> Twitter: willemjiang
> Weibo: 姜宁willem
>
> On Fri, Jul 13, 2018 at 2:29 PM, Luca Burgazzoli <lb...@gmail.com>
> wrote:
>
> > Love it so ++1
> >
> > Btw, I did start some (early) experiments around alternative canel
> runtime
> > in my spare time:
> >
> > - https://github.com/lburgazzoli/camel-go
> > - https://github.com/lburgazzoli/camel-go-examples/
> > tree/master/example-yaml
> >
> > Happy to make it part of the kamel initiative if you find it interesting.
> >
> > On Fri, 13 Jul 2018 at 08:14, Sascha Dirbach <
> > sascha.dirbach@endless-webservices.de> wrote:
> >
> > > Hi Nicola,
> > >
> > > +1
> > >
> > > Great idea, do you already have a concept/idea/sketch on how to deal
> > > with complex configurations? i.e. SSL certs for outgoing calls, XSLT
> > > transformations, etc.
> > >
> > > When I think about it, you could probably use configmaps/secrets to
> > > mount these config in the container an then reference them in the
> route.
> > >
> > > Best regards,
> > >
> > > Sascha
> > >
> > > Am 13.07.2018 um 01:30 schrieb Nicola Ferraro:
> > > > Hi Cameleers,
> > > > it's now passed some time since I started thinking about a new
> project
> > > that
> > > > we can begin here at Apache Camel, and I'd like to have your opinion.
> > > >
> > > > We've already been targeting cloud-native applications with Camel,
> > > > especially on top of Kubernetes, that is becoming "the standard"
> cloud
> > > > platform. But writing a Camel integration and running it on
> Kubernetes
> > > > requires some effort: choosing the base platform (spring-boot, karaf,
> > > > simple main?), adding health checks (actuator?), packaging a docker
> > image
> > > > and creating the Kubernetes resources (fabric8-maven-plugin, helm?),
> > > > publishing the image on a docker registry, then finally deploying the
> > > > resources on a Kubernetes cluster.
> > > >
> > > > The resulting integration container is then far from being optimal
> > from a
> > > > resource consumption point of view: it is likely that a Camel
> > Spring-Boot
> > > > application will require at least 200MB of RAM and also some CPU
> shares
> > > > because of polling threads used by many components.
> > > >
> > > > In case people use a CI/CD pipeline, it will take also a long time to
> > get
> > > > from a code update to having a Kubernetes POD up and running.
> > > > Apart from compilation and image push/pull time, also startup time is
> > > often
> > > > ~10 seconds for Camel + Spring-Boot in a container with standard
> limits
> > > on
> > > > resources, making it difficult to propose this combination for
> > > "serverless
> > > > integration" (this term is becoming increasingly more popular).
> > > >
> > > > So, my proposal is to start to investigate a "more cloud-native"
> > approach
> > > > to integration: *making Camel integrations first-class citizens in
> > > > Kubernetes, and making them super fast and lightweight.*
> > > >
> > > > We can base the project on Kubernetes Custom Resource Definitions
> (CRD)
> > > > <
> > > https://kubernetes.io/docs/concepts/extend-kubernetes/
> > api-extension/custom-resources/
> > > >,
> > > > for example a Integration CRD and have a Kubernetes "operator"
> > > > <https://coreos.com/operators/> taking care of:
> > > > - Optimizing the integration that we want to run
> > > > - Packaging in a container
> > > > - Running it on Kubernetes
> > > > - Managing its entire lifecycle
> > > >
> > > > A Kubernetes-native integration may look like:
> > > >
> > > > -------------------
> > > > kind: "Integration"
> > > > apiVersion: "camel.apache.org/v1alpha1"
> > > > metadata:
> > > > name: "example"
> > > > spec:
> > > > replicas: 1
> > > > routes:
> > > > - id: timer
> > > > route:
> > > > - type: endpoint
> > > > uri: timer:tick
> > > > - type: endpoint
> > > > uri: log:info
> > > > -------------------
> > > >
> > > > For those who are not familiar with Kubernetes resources, this kind
> of
> > > > YAML/JSON resource definitions are really common.
> > > > The example route is embedded in the Kubernetes resource declaration
> > and
> > > > follows a basic "flow DSL". We may start from a basic one and evolve
> it
> > > as
> > > > new requirements arrive from the community.
> > > >
> > > > I've made a very simple (but working) POC here:
> > > > https://github.com/nicolaferraro/integration-operator.
> > > >
> > > > This idea of a "Cloud-Native Camel" on Kubernetes (project codename
> can
> > > be "
> > > > *Kamel*", if you like it :D), will be an enabler for a lot of nice
> > > features.
> > > >
> > > > For example, we can propose "Kamel" as "ideal" platform for
> "serverless
> > > > integration" (I see many people reinventing the wheel out there): the
> > > > operator can reduce resource consumption of a single integration by
> > > > optimizing the runtime and also pause/resume integrations when they
> are
> > > not
> > > > used, that is the basic idea behind "serverless" (e.g. think to
> > > > HTTP-triggered integrations, but not only).
> > > > Focusing on serverless will bring more emphasis on push-based
> > > notifications
> > > > (webhooks, cloud events <https://cloudevents.io/>), that are rarely
> > > used in
> > > > Camel components, that prefer a poll based approach being it simpler
> to
> > > use
> > > > in classic deployments, but not so good in the cloud, where more
> > > resources
> > > > become higher direct costs for the users.
> > > >
> > > > The presence of the simplified DSL enables also experimenting on
> > > "*reduced*
> > > > subsets of Camel" implemented in languages other than Java, for
> example
> > > one
> > > > language that has a reactive approach on thread scheduling and a
> really
> > > low
> > > > memory footprint, like Go.
> > > >
> > > > But apart from this kind of experiments (that are valid IMO), the
> > "Kamel"
> > > > optimizer will have free room to choose the right platform for the
> > > > integration that the user wants to run, including, in the future,
> doing
> > > AOT
> > > > compilation using Graal/VM (less memory, faster startup) if the
> > features
> > > > (components) used in the integration are supporting it (maybe we can
> > add
> > > > AOT compilation in the roadmap for Camel 3).
> > > > A silly optimization: integrations starting from "timer:..." may be
> > > > scheduled directly with Kubernetes CronJobs, so they will consume
> > > resources
> > > > only when actually running.
> > > >
> > > > Being the final integrations lightweight and being the DSL
> > > > language-independent, we may see a increased adoption of Camel also
> as
> > > > agile integration layer for not-only-java applications (both "cloud"
> > and
> > > > "serverless" applications).
> > > >
> > > > I'm the first one that would like to work on a project ilke this.
> I've
> > > > worked on many Kubernetes/Openshift based applications and frameworks
> > in
> > > > the past years, also on operators and CRDs, and I think this way of
> > > > redesigning integrations has a lot of potential.
> > > >
> > > > Integrations will not be necessarily limited to the simplified DSL,
> but
> > > we
> > > > can add extension points for scripting and even custom libraries
> > > (although
> > > > limiting the freedom of the optimizer).
> > > >
> > > > The most important thing: it may become a great project, since it's
> > > driven
> > > > by a great community.
> > > >
> > > > So, what do you think? Is it crazy enough?
> > > >
> > > > Nicola
> > > >
> > >
> > > --
> > > Sascha Dirbach
> > >
> > > Inhaber
> > >
> > > endless webservices
> > > Marco Paetschke & Sascha Dirbach GbR
> > >
> > > Kirchweg 113
> > > 28201 Bremen
> > >
> > > Mobil: +49 (0)160-94182103
> > > Mail: sascha.dirbach@endless-webservices.de
> > > Web: www.endless-webservices.de
> > >
> > > USt-IdNr.: DE310969215
> > >
> > >
> > > --
> > --
> > Luca Burgazzoli
> >
>
--
--
Luca Burgazzoli
Re: A "Kamel" crazy idea
Posted by Willem Jiang <wi...@gmail.com>.
Hi Luca,
You are building Go version of Camel by implement the DSL in Go.
Now we can start the camel context by using camel-go to load the route,
but here are some missing points that I want fill such as how to reuse the
Camel components?
Willem Jiang
Twitter: willemjiang
Weibo: 姜宁willem
On Fri, Jul 13, 2018 at 2:29 PM, Luca Burgazzoli <lb...@gmail.com>
wrote:
> Love it so ++1
>
> Btw, I did start some (early) experiments around alternative canel runtime
> in my spare time:
>
> - https://github.com/lburgazzoli/camel-go
> - https://github.com/lburgazzoli/camel-go-examples/
> tree/master/example-yaml
>
> Happy to make it part of the kamel initiative if you find it interesting.
>
> On Fri, 13 Jul 2018 at 08:14, Sascha Dirbach <
> sascha.dirbach@endless-webservices.de> wrote:
>
> > Hi Nicola,
> >
> > +1
> >
> > Great idea, do you already have a concept/idea/sketch on how to deal
> > with complex configurations? i.e. SSL certs for outgoing calls, XSLT
> > transformations, etc.
> >
> > When I think about it, you could probably use configmaps/secrets to
> > mount these config in the container an then reference them in the route.
> >
> > Best regards,
> >
> > Sascha
> >
> > Am 13.07.2018 um 01:30 schrieb Nicola Ferraro:
> > > Hi Cameleers,
> > > it's now passed some time since I started thinking about a new project
> > that
> > > we can begin here at Apache Camel, and I'd like to have your opinion.
> > >
> > > We've already been targeting cloud-native applications with Camel,
> > > especially on top of Kubernetes, that is becoming "the standard" cloud
> > > platform. But writing a Camel integration and running it on Kubernetes
> > > requires some effort: choosing the base platform (spring-boot, karaf,
> > > simple main?), adding health checks (actuator?), packaging a docker
> image
> > > and creating the Kubernetes resources (fabric8-maven-plugin, helm?),
> > > publishing the image on a docker registry, then finally deploying the
> > > resources on a Kubernetes cluster.
> > >
> > > The resulting integration container is then far from being optimal
> from a
> > > resource consumption point of view: it is likely that a Camel
> Spring-Boot
> > > application will require at least 200MB of RAM and also some CPU shares
> > > because of polling threads used by many components.
> > >
> > > In case people use a CI/CD pipeline, it will take also a long time to
> get
> > > from a code update to having a Kubernetes POD up and running.
> > > Apart from compilation and image push/pull time, also startup time is
> > often
> > > ~10 seconds for Camel + Spring-Boot in a container with standard limits
> > on
> > > resources, making it difficult to propose this combination for
> > "serverless
> > > integration" (this term is becoming increasingly more popular).
> > >
> > > So, my proposal is to start to investigate a "more cloud-native"
> approach
> > > to integration: *making Camel integrations first-class citizens in
> > > Kubernetes, and making them super fast and lightweight.*
> > >
> > > We can base the project on Kubernetes Custom Resource Definitions (CRD)
> > > <
> > https://kubernetes.io/docs/concepts/extend-kubernetes/
> api-extension/custom-resources/
> > >,
> > > for example a Integration CRD and have a Kubernetes "operator"
> > > <https://coreos.com/operators/> taking care of:
> > > - Optimizing the integration that we want to run
> > > - Packaging in a container
> > > - Running it on Kubernetes
> > > - Managing its entire lifecycle
> > >
> > > A Kubernetes-native integration may look like:
> > >
> > > -------------------
> > > kind: "Integration"
> > > apiVersion: "camel.apache.org/v1alpha1"
> > > metadata:
> > > name: "example"
> > > spec:
> > > replicas: 1
> > > routes:
> > > - id: timer
> > > route:
> > > - type: endpoint
> > > uri: timer:tick
> > > - type: endpoint
> > > uri: log:info
> > > -------------------
> > >
> > > For those who are not familiar with Kubernetes resources, this kind of
> > > YAML/JSON resource definitions are really common.
> > > The example route is embedded in the Kubernetes resource declaration
> and
> > > follows a basic "flow DSL". We may start from a basic one and evolve it
> > as
> > > new requirements arrive from the community.
> > >
> > > I've made a very simple (but working) POC here:
> > > https://github.com/nicolaferraro/integration-operator.
> > >
> > > This idea of a "Cloud-Native Camel" on Kubernetes (project codename can
> > be "
> > > *Kamel*", if you like it :D), will be an enabler for a lot of nice
> > features.
> > >
> > > For example, we can propose "Kamel" as "ideal" platform for "serverless
> > > integration" (I see many people reinventing the wheel out there): the
> > > operator can reduce resource consumption of a single integration by
> > > optimizing the runtime and also pause/resume integrations when they are
> > not
> > > used, that is the basic idea behind "serverless" (e.g. think to
> > > HTTP-triggered integrations, but not only).
> > > Focusing on serverless will bring more emphasis on push-based
> > notifications
> > > (webhooks, cloud events <https://cloudevents.io/>), that are rarely
> > used in
> > > Camel components, that prefer a poll based approach being it simpler to
> > use
> > > in classic deployments, but not so good in the cloud, where more
> > resources
> > > become higher direct costs for the users.
> > >
> > > The presence of the simplified DSL enables also experimenting on
> > "*reduced*
> > > subsets of Camel" implemented in languages other than Java, for example
> > one
> > > language that has a reactive approach on thread scheduling and a really
> > low
> > > memory footprint, like Go.
> > >
> > > But apart from this kind of experiments (that are valid IMO), the
> "Kamel"
> > > optimizer will have free room to choose the right platform for the
> > > integration that the user wants to run, including, in the future, doing
> > AOT
> > > compilation using Graal/VM (less memory, faster startup) if the
> features
> > > (components) used in the integration are supporting it (maybe we can
> add
> > > AOT compilation in the roadmap for Camel 3).
> > > A silly optimization: integrations starting from "timer:..." may be
> > > scheduled directly with Kubernetes CronJobs, so they will consume
> > resources
> > > only when actually running.
> > >
> > > Being the final integrations lightweight and being the DSL
> > > language-independent, we may see a increased adoption of Camel also as
> > > agile integration layer for not-only-java applications (both "cloud"
> and
> > > "serverless" applications).
> > >
> > > I'm the first one that would like to work on a project ilke this. I've
> > > worked on many Kubernetes/Openshift based applications and frameworks
> in
> > > the past years, also on operators and CRDs, and I think this way of
> > > redesigning integrations has a lot of potential.
> > >
> > > Integrations will not be necessarily limited to the simplified DSL, but
> > we
> > > can add extension points for scripting and even custom libraries
> > (although
> > > limiting the freedom of the optimizer).
> > >
> > > The most important thing: it may become a great project, since it's
> > driven
> > > by a great community.
> > >
> > > So, what do you think? Is it crazy enough?
> > >
> > > Nicola
> > >
> >
> > --
> > Sascha Dirbach
> >
> > Inhaber
> >
> > endless webservices
> > Marco Paetschke & Sascha Dirbach GbR
> >
> > Kirchweg 113
> > 28201 Bremen
> >
> > Mobil: +49 (0)160-94182103
> > Mail: sascha.dirbach@endless-webservices.de
> > Web: www.endless-webservices.de
> >
> > USt-IdNr.: DE310969215
> >
> >
> > --
> --
> Luca Burgazzoli
>
Re: A "Kamel" crazy idea
Posted by Luca Burgazzoli <lb...@gmail.com>.
Love it so ++1
Btw, I did start some (early) experiments around alternative canel runtime
in my spare time:
- https://github.com/lburgazzoli/camel-go
- https://github.com/lburgazzoli/camel-go-examples/tree/master/example-yaml
Happy to make it part of the kamel initiative if you find it interesting.
On Fri, 13 Jul 2018 at 08:14, Sascha Dirbach <
sascha.dirbach@endless-webservices.de> wrote:
> Hi Nicola,
>
> +1
>
> Great idea, do you already have a concept/idea/sketch on how to deal
> with complex configurations? i.e. SSL certs for outgoing calls, XSLT
> transformations, etc.
>
> When I think about it, you could probably use configmaps/secrets to
> mount these config in the container an then reference them in the route.
>
> Best regards,
>
> Sascha
>
> Am 13.07.2018 um 01:30 schrieb Nicola Ferraro:
> > Hi Cameleers,
> > it's now passed some time since I started thinking about a new project
> that
> > we can begin here at Apache Camel, and I'd like to have your opinion.
> >
> > We've already been targeting cloud-native applications with Camel,
> > especially on top of Kubernetes, that is becoming "the standard" cloud
> > platform. But writing a Camel integration and running it on Kubernetes
> > requires some effort: choosing the base platform (spring-boot, karaf,
> > simple main?), adding health checks (actuator?), packaging a docker image
> > and creating the Kubernetes resources (fabric8-maven-plugin, helm?),
> > publishing the image on a docker registry, then finally deploying the
> > resources on a Kubernetes cluster.
> >
> > The resulting integration container is then far from being optimal from a
> > resource consumption point of view: it is likely that a Camel Spring-Boot
> > application will require at least 200MB of RAM and also some CPU shares
> > because of polling threads used by many components.
> >
> > In case people use a CI/CD pipeline, it will take also a long time to get
> > from a code update to having a Kubernetes POD up and running.
> > Apart from compilation and image push/pull time, also startup time is
> often
> > ~10 seconds for Camel + Spring-Boot in a container with standard limits
> on
> > resources, making it difficult to propose this combination for
> "serverless
> > integration" (this term is becoming increasingly more popular).
> >
> > So, my proposal is to start to investigate a "more cloud-native" approach
> > to integration: *making Camel integrations first-class citizens in
> > Kubernetes, and making them super fast and lightweight.*
> >
> > We can base the project on Kubernetes Custom Resource Definitions (CRD)
> > <
> https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/
> >,
> > for example a Integration CRD and have a Kubernetes "operator"
> > <https://coreos.com/operators/> taking care of:
> > - Optimizing the integration that we want to run
> > - Packaging in a container
> > - Running it on Kubernetes
> > - Managing its entire lifecycle
> >
> > A Kubernetes-native integration may look like:
> >
> > -------------------
> > kind: "Integration"
> > apiVersion: "camel.apache.org/v1alpha1"
> > metadata:
> > name: "example"
> > spec:
> > replicas: 1
> > routes:
> > - id: timer
> > route:
> > - type: endpoint
> > uri: timer:tick
> > - type: endpoint
> > uri: log:info
> > -------------------
> >
> > For those who are not familiar with Kubernetes resources, this kind of
> > YAML/JSON resource definitions are really common.
> > The example route is embedded in the Kubernetes resource declaration and
> > follows a basic "flow DSL". We may start from a basic one and evolve it
> as
> > new requirements arrive from the community.
> >
> > I've made a very simple (but working) POC here:
> > https://github.com/nicolaferraro/integration-operator.
> >
> > This idea of a "Cloud-Native Camel" on Kubernetes (project codename can
> be "
> > *Kamel*", if you like it :D), will be an enabler for a lot of nice
> features.
> >
> > For example, we can propose "Kamel" as "ideal" platform for "serverless
> > integration" (I see many people reinventing the wheel out there): the
> > operator can reduce resource consumption of a single integration by
> > optimizing the runtime and also pause/resume integrations when they are
> not
> > used, that is the basic idea behind "serverless" (e.g. think to
> > HTTP-triggered integrations, but not only).
> > Focusing on serverless will bring more emphasis on push-based
> notifications
> > (webhooks, cloud events <https://cloudevents.io/>), that are rarely
> used in
> > Camel components, that prefer a poll based approach being it simpler to
> use
> > in classic deployments, but not so good in the cloud, where more
> resources
> > become higher direct costs for the users.
> >
> > The presence of the simplified DSL enables also experimenting on
> "*reduced*
> > subsets of Camel" implemented in languages other than Java, for example
> one
> > language that has a reactive approach on thread scheduling and a really
> low
> > memory footprint, like Go.
> >
> > But apart from this kind of experiments (that are valid IMO), the "Kamel"
> > optimizer will have free room to choose the right platform for the
> > integration that the user wants to run, including, in the future, doing
> AOT
> > compilation using Graal/VM (less memory, faster startup) if the features
> > (components) used in the integration are supporting it (maybe we can add
> > AOT compilation in the roadmap for Camel 3).
> > A silly optimization: integrations starting from "timer:..." may be
> > scheduled directly with Kubernetes CronJobs, so they will consume
> resources
> > only when actually running.
> >
> > Being the final integrations lightweight and being the DSL
> > language-independent, we may see a increased adoption of Camel also as
> > agile integration layer for not-only-java applications (both "cloud" and
> > "serverless" applications).
> >
> > I'm the first one that would like to work on a project ilke this. I've
> > worked on many Kubernetes/Openshift based applications and frameworks in
> > the past years, also on operators and CRDs, and I think this way of
> > redesigning integrations has a lot of potential.
> >
> > Integrations will not be necessarily limited to the simplified DSL, but
> we
> > can add extension points for scripting and even custom libraries
> (although
> > limiting the freedom of the optimizer).
> >
> > The most important thing: it may become a great project, since it's
> driven
> > by a great community.
> >
> > So, what do you think? Is it crazy enough?
> >
> > Nicola
> >
>
> --
> Sascha Dirbach
>
> Inhaber
>
> endless webservices
> Marco Paetschke & Sascha Dirbach GbR
>
> Kirchweg 113
> 28201 Bremen
>
> Mobil: +49 (0)160-94182103
> Mail: sascha.dirbach@endless-webservices.de
> Web: www.endless-webservices.de
>
> USt-IdNr.: DE310969215
>
>
> --
--
Luca Burgazzoli
Re: A "Kamel" crazy idea
Posted by Sascha Dirbach <sa...@endless-webservices.de>.
Hi Nicola,
+1
Great idea, do you already have a concept/idea/sketch on how to deal
with complex configurations? i.e. SSL certs for outgoing calls, XSLT
transformations, etc.
When I think about it, you could probably use configmaps/secrets to
mount these config in the container an then reference them in the route.
Best regards,
Sascha
Am 13.07.2018 um 01:30 schrieb Nicola Ferraro:
> Hi Cameleers,
> it's now passed some time since I started thinking about a new project that
> we can begin here at Apache Camel, and I'd like to have your opinion.
>
> We've already been targeting cloud-native applications with Camel,
> especially on top of Kubernetes, that is becoming "the standard" cloud
> platform. But writing a Camel integration and running it on Kubernetes
> requires some effort: choosing the base platform (spring-boot, karaf,
> simple main?), adding health checks (actuator?), packaging a docker image
> and creating the Kubernetes resources (fabric8-maven-plugin, helm?),
> publishing the image on a docker registry, then finally deploying the
> resources on a Kubernetes cluster.
>
> The resulting integration container is then far from being optimal from a
> resource consumption point of view: it is likely that a Camel Spring-Boot
> application will require at least 200MB of RAM and also some CPU shares
> because of polling threads used by many components.
>
> In case people use a CI/CD pipeline, it will take also a long time to get
> from a code update to having a Kubernetes POD up and running.
> Apart from compilation and image push/pull time, also startup time is often
> ~10 seconds for Camel + Spring-Boot in a container with standard limits on
> resources, making it difficult to propose this combination for "serverless
> integration" (this term is becoming increasingly more popular).
>
> So, my proposal is to start to investigate a "more cloud-native" approach
> to integration: *making Camel integrations first-class citizens in
> Kubernetes, and making them super fast and lightweight.*
>
> We can base the project on Kubernetes Custom Resource Definitions (CRD)
> <https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/>,
> for example a Integration CRD and have a Kubernetes "operator"
> <https://coreos.com/operators/> taking care of:
> - Optimizing the integration that we want to run
> - Packaging in a container
> - Running it on Kubernetes
> - Managing its entire lifecycle
>
> A Kubernetes-native integration may look like:
>
> -------------------
> kind: "Integration"
> apiVersion: "camel.apache.org/v1alpha1"
> metadata:
> name: "example"
> spec:
> replicas: 1
> routes:
> - id: timer
> route:
> - type: endpoint
> uri: timer:tick
> - type: endpoint
> uri: log:info
> -------------------
>
> For those who are not familiar with Kubernetes resources, this kind of
> YAML/JSON resource definitions are really common.
> The example route is embedded in the Kubernetes resource declaration and
> follows a basic "flow DSL". We may start from a basic one and evolve it as
> new requirements arrive from the community.
>
> I've made a very simple (but working) POC here:
> https://github.com/nicolaferraro/integration-operator.
>
> This idea of a "Cloud-Native Camel" on Kubernetes (project codename can be "
> *Kamel*", if you like it :D), will be an enabler for a lot of nice features.
>
> For example, we can propose "Kamel" as "ideal" platform for "serverless
> integration" (I see many people reinventing the wheel out there): the
> operator can reduce resource consumption of a single integration by
> optimizing the runtime and also pause/resume integrations when they are not
> used, that is the basic idea behind "serverless" (e.g. think to
> HTTP-triggered integrations, but not only).
> Focusing on serverless will bring more emphasis on push-based notifications
> (webhooks, cloud events <https://cloudevents.io/>), that are rarely used in
> Camel components, that prefer a poll based approach being it simpler to use
> in classic deployments, but not so good in the cloud, where more resources
> become higher direct costs for the users.
>
> The presence of the simplified DSL enables also experimenting on "*reduced*
> subsets of Camel" implemented in languages other than Java, for example one
> language that has a reactive approach on thread scheduling and a really low
> memory footprint, like Go.
>
> But apart from this kind of experiments (that are valid IMO), the "Kamel"
> optimizer will have free room to choose the right platform for the
> integration that the user wants to run, including, in the future, doing AOT
> compilation using Graal/VM (less memory, faster startup) if the features
> (components) used in the integration are supporting it (maybe we can add
> AOT compilation in the roadmap for Camel 3).
> A silly optimization: integrations starting from "timer:..." may be
> scheduled directly with Kubernetes CronJobs, so they will consume resources
> only when actually running.
>
> Being the final integrations lightweight and being the DSL
> language-independent, we may see a increased adoption of Camel also as
> agile integration layer for not-only-java applications (both "cloud" and
> "serverless" applications).
>
> I'm the first one that would like to work on a project ilke this. I've
> worked on many Kubernetes/Openshift based applications and frameworks in
> the past years, also on operators and CRDs, and I think this way of
> redesigning integrations has a lot of potential.
>
> Integrations will not be necessarily limited to the simplified DSL, but we
> can add extension points for scripting and even custom libraries (although
> limiting the freedom of the optimizer).
>
> The most important thing: it may become a great project, since it's driven
> by a great community.
>
> So, what do you think? Is it crazy enough?
>
> Nicola
>
--
Sascha Dirbach
Inhaber
endless webservices
Marco Paetschke & Sascha Dirbach GbR
Kirchweg 113
28201 Bremen
Mobil: +49 (0)160-94182103
Mail: sascha.dirbach@endless-webservices.de
Web: www.endless-webservices.de
USt-IdNr.: DE310969215
Re: A "Kamel" crazy idea
Posted by Luca Burgazzoli <lb...@gmail.com>.
++1
---
Luca Burgazzoli
On Tue, Jul 31, 2018 at 9:47 AM, Andrea Cosentino
<an...@yahoo.com.invalid> wrote:
> This is a wonderful idea and I believe we can improve our community with a subproject like this.
>
> So +1 for me.
>
> --
> Andrea Cosentino
> ----------------------------------
> Apache Camel PMC Chair
> Apache Karaf Committer
> Apache Servicemix PMC Member
> Email: ancosen1985@yahoo.com
> Twitter: @oscerd2
> Github: oscerd
>
>
>
>
>
>
> On Monday, July 30, 2018, 5:49:50 PM GMT+2, Nicola Ferraro <ni...@gmail.com> wrote:
>
>
>
>
>
> Hi Cameleers,
> it seems from the comments that this "Kamel" subproject is something we
> want to start and I think that also the main camel core will benefit from
> the new features it will bring.
>
> I would like to donate the current "Kamel" code to Apache Camel, in order
> to have a initial brick to start from.
>
> From your reactions, the name "Kamel" seems cool and I'd like to keep it.
> However if you talk to people about the "Kamel" project, they won't
> understand if you're talking about "Kamel with the K" or "Camel with the C".
> For this reasons I propose to keep "Kamel", but also use "Camel K" [ˈkæməl
> keɪ] as friendly name when talking about it. This is in line with the
> repository we should create for the subproject, that needs to start with
> "camel-" (it will be "apache/camel-k" in that case).
>
> These days I've been experimenting different ideas with Luca Burgazzoli. In
> particular, there have been some concerns here on the power of a
> declarative DSL (a new one or the existing XML one) and we've found a
> strategy that will allow us to use also the Java DSL in Kamel integrations.
> Without adding too many details here, at integration build time we can run
> the user code in a build container and inspect the produced routes to get
> metadata for the optimizer. This is one of the first things we should do
> next.
>
> This possibility of running code that produces integrations opened a lot of
> unexpected paths, that I've written down in the project roadmap (
> https://github.com/nicolaferraro/integration-operator) and summarized below.
>
> One nice feature (kudos to Luca) is that we can simplify life for Kamel
> users up to the point that they'll just need to write their routes on a
> Java file and run them with e.g.:
> "kamel run Routes.java"
>
> Where "kamel" is a binary we release within the project. Kubernetes custom
> resources will be used under the hood, but the "kamel" binary is a utility
> that will provide a user experience comparable to that of current
> serverless platforms (or even better).
>
> But there's not just that.
>
> Luca wrote this weekend a prototype for having a polyglot Camel (
> https://github.com/lburgazzoli/camel-routes-loader).
>
> With that, you will be able to e.g. write integrations in groovy and use
> the same Kamel engine:
> "kamel run routes.groovy"
>
> But also JavaScript:
> https://github.com/lburgazzoli/camel-routes-loader/blob/ca986541f7c422ee02c21727cdfe4293d64a364e/src/test/resources/ext/camel/routes.js#L2
>
> This is a prototype right now, but a similar approach based on GraalVM has
> a lot of potential, because users can not only use their preferred language
> to write routes and processors, but also bind them to functionalities
> available in their preferred libraries.
>
> How much this will be feasible depends on the adoption of GraalVM, but I'm
> seeing many frameworks adding metadata to make GraalVM work with
> reflection. We've also done some work in Camel, there are some Jiras for it
> and first tests made by Guillaume signal that it's something feasible, at
> least for the Camel core and a subset of components. GraalVM would be
> important also to reduce memory footprint and improve startup time, as
> already said.
>
> If the GraalVM approach works (it is working for other frameworks), instead
> of rewriting a subset of Camel in Go (as the original proposal mentions),
> we can just sanitize and recompile our existing codebase: this way we fully
> leverage the strength of Apache Camel.
>
> Another important thing in the roadmap is actually how we'll integrate with
> current FaaS platforms. We will evaluate together if it's better to have a
> tighter integration with some of them or to leverage knative for some
> aspects of Kamel...
>
> But, let's get started!
>
> Il 19 lug 2018 10:45, "Nicola Ferraro" <ni...@gmail.com> ha scritto:
>
> It's clear to me that we need to add support for our existing XML DSL, that
> is powerful. But there are multiple reasons why I'd like to also "add" a
> limited yaml/json notation to "Kamel".
>
> The first one (and simplest) is that json/yaml is the primary encoding for
> all resources exchanged with the Kubernetes API server. I'm not saying that
> for this reason the Integration resource cannot contain a XML
> (/Java/Ballerina) section, but from a pure presentational point of view,
> having the possibility to write simple use cases (even a "hello world") in
> pure json/yaml is better than always requiring a mix of json and XML.
>
> The second one is simplicity. Writing a optimizer (the module that
> materializes the integration into running code, choosing a specific runtime
> and configuration) for a turing-complete language like Java is not easy:
> even if you manage to create a good parser, it may be able to do
> optimizations only if you write routes in a particular way, without complex
> constructs.
> XML is ok from this point of view. The thing is that both json/yaml and XML
> are just two different ways to serialize object trees, that can be then
> statically analyzed.
> The point is not XML vs json/yaml, it's more about tailoring a new
> minimalistic DSL to the emerging use cases vs proposing "only" our classic
> way of writing integrations. I think XML can be the "advanced" way. We can
> experiment optimizations easily with the new DSL, and enable them also on
> XML if it's worth.
>
> Scripting should be part of the spec, but I'd try to use programming
> languages only for processing/transformation, not for the route definition.
>
> Third one is performance. Apart from the fact that json parsers are in
> general said to be faster than XML parsers... Given the "fast startup"
> target that we want to reach, we may think e.g. to translate the new DSL
> into Java or Go code, then compiling it. This allows doing parsing at build
> time in order to avoid it on startup. This kind of improvements are much
> easier with a limited DSL but much more difficult with a existing
> fully-fledged DSL..
>
> Nicola
>
>
>
> On Wed, Jul 18, 2018 at 11:51 PM Pontus Ullgren <ul...@gmail.com> wrote:
>
>> I also like the idea but with some comments.
>>
>> As Hiram Chirino I'm not sure YAML/JSON is the best language for this.
>> Perhaps a more fluent DSL like the Camel Java DSL or perhaps something like
>> Ballerina language would be better suited ?
>> Also in my experience even simple integrations, that is simple real world
>> integration and not just hello world demos, requires
>> you to add one or more Java class or scripting in addition to the core
>> components. So for it to be useful there must be some way to add custom
>> code in some way for aggregation/enrichment strategies.
>>
>> If you go with a GO implementation I would also like some fallback for
>> using Java since I find it unlikely that the existing base of components
>> will be ported any time soon.
>>
>> As for the name as other I like the "Kamel" name however a being a newborn
>> kotlin fan I know of one pre-existing project with that name
>> that is close enough to allow for potential confusion.
>> https://github.com/topicusoverheid/kamel
>>
>> Looking forward to see the result.
>>
>> // Pontus
>>
>>
>>
>>
>>
>>
>> On Mon, 16 Jul 2018 at 15:15 Hiram Chirino <hi...@hiramchirino.com> wrote:
>>
>> > Love the idea.
>> >
>> > Personally, I'd keep using the existing Camel XML DSL if possible. You
>> can
>> > still embed it in the CRD. The operator that deploys it could inspect it
>> > and figure whats the most optimized runtime that can support the DSL.
>> > Perhaps if it's only using the restricted set of camel components
>> supported
>> > by camel-go (https://github.com/lburgazzoli/camel-go) then is uses that.
>> > Otherwise it falls back to using camel spring boot.
>> >
>> > For that to work I think we need the different runtime implementations to
>> > provide a way to ask them: 'hey do you support running this camel route?'
>> > Not a trivial thing to respond to, it might require a build step in there
>> > for traditional Camel runtimes.
>> >
>> >
>> > On Fri, Jul 13, 2018 at 4:32 AM Antonin Stefanutti <
>> antonin@stefanutti.fr>
>> > wrote:
>> >
>> > > Hi Nicola,
>> > >
>> > > I love the idea.
>> > >
>> > > I just wonder whether YAML/JSON is an expressive enough format in the
>> > long
>> > > term. But as you’ve mentioned, starting simple would enable
>> experimenting
>> > > some very interesting / promising optimisations. So it seems worth
>> taking
>> > > that path, instead of trying to embed a complex DSL or the existing XML
>> > DSL
>> > > into the CRD.
>> > >
>> > > Definitely +1
>> > >
>> > > > On 13 Jul 2018, at 01:30, Nicola Ferraro <ni...@gmail.com>
>> wrote:
>> > > >
>> > > > Hi Cameleers,
>> > > > it's now passed some time since I started thinking about a new
>> project
>> > > that
>> > > > we can begin here at Apache Camel, and I'd like to have your opinion.
>> > > >
>> > > > We've already been targeting cloud-native applications with Camel,
>> > > > especially on top of Kubernetes, that is becoming "the standard"
>> cloud
>> > > > platform. But writing a Camel integration and running it on
>> Kubernetes
>> > > > requires some effort: choosing the base platform (spring-boot, karaf,
>> > > > simple main?), adding health checks (actuator?), packaging a docker
>> > image
>> > > > and creating the Kubernetes resources (fabric8-maven-plugin, helm?),
>> > > > publishing the image on a docker registry, then finally deploying the
>> > > > resources on a Kubernetes cluster.
>> > > >
>> > > > The resulting integration container is then far from being optimal
>> > from a
>> > > > resource consumption point of view: it is likely that a Camel
>> > Spring-Boot
>> > > > application will require at least 200MB of RAM and also some CPU
>> shares
>> > > > because of polling threads used by many components.
>> > > >
>> > > > In case people use a CI/CD pipeline, it will take also a long time to
>> > get
>> > > > from a code update to having a Kubernetes POD up and running.
>> > > > Apart from compilation and image push/pull time, also startup time is
>> > > often
>> > > > ~10 seconds for Camel + Spring-Boot in a container with standard
>> limits
>> > > on
>> > > > resources, making it difficult to propose this combination for
>> > > "serverless
>> > > > integration" (this term is becoming increasingly more popular).
>> > > >
>> > > > So, my proposal is to start to investigate a "more cloud-native"
>> > approach
>> > > > to integration: *making Camel integrations first-class citizens in
>> > > > Kubernetes, and making them super fast and lightweight.*
>> > > >
>> > > > We can base the project on Kubernetes Custom Resource Definitions
>> (CRD)
>> > > > <
>> > >
>> >
>> https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/
>> > > >,
>> > > > for example a Integration CRD and have a Kubernetes "operator"
>> > > > <https://coreos.com/operators/> taking care of:
>> > > > - Optimizing the integration that we want to run
>> > > > - Packaging in a container
>> > > > - Running it on Kubernetes
>> > > > - Managing its entire lifecycle
>> > > >
>> > > > A Kubernetes-native integration may look like:
>> > > >
>> > > > -------------------
>> > > > kind: "Integration"
>> > > > apiVersion: "camel.apache.org/v1alpha1"
>> > > > metadata:
>> > > > name: "example"
>> > > > spec:
>> > > > replicas: 1
>> > > > routes:
>> > > > - id: timer
>> > > > route:
>> > > > - type: endpoint
>> > > > uri: timer:tick
>> > > > - type: endpoint
>> > > > uri: log:info
>> > > > -------------------
>> > > >
>> > > > For those who are not familiar with Kubernetes resources, this kind
>> of
>> > > > YAML/JSON resource definitions are really common.
>> > > > The example route is embedded in the Kubernetes resource declaration
>> > and
>> > > > follows a basic "flow DSL". We may start from a basic one and evolve
>> it
>> > > as
>> > > > new requirements arrive from the community.
>> > > >
>> > > > I've made a very simple (but working) POC here:
>> > > > https://github.com/nicolaferraro/integration-operator.
>> > > >
>> > > > This idea of a "Cloud-Native Camel" on Kubernetes (project codename
>> can
>> > > be "
>> > > > *Kamel*", if you like it :D), will be an enabler for a lot of nice
>> > > features.
>> > > >
>> > > > For example, we can propose "Kamel" as "ideal" platform for
>> "serverless
>> > > > integration" (I see many people reinventing the wheel out there): the
>> > > > operator can reduce resource consumption of a single integration by
>> > > > optimizing the runtime and also pause/resume integrations when they
>> are
>> > > not
>> > > > used, that is the basic idea behind "serverless" (e.g. think to
>> > > > HTTP-triggered integrations, but not only).
>> > > > Focusing on serverless will bring more emphasis on push-based
>> > > notifications
>> > > > (webhooks, cloud events <https://cloudevents.io/>), that are rarely
>> > > used in
>> > > > Camel components, that prefer a poll based approach being it simpler
>> to
>> > > use
>> > > > in classic deployments, but not so good in the cloud, where more
>> > > resources
>> > > > become higher direct costs for the users.
>> > > >
>> > > > The presence of the simplified DSL enables also experimenting on
>> > > "*reduced*
>> > > > subsets of Camel" implemented in languages other than Java, for
>> example
>> > > one
>> > > > language that has a reactive approach on thread scheduling and a
>> really
>> > > low
>> > > > memory footprint, like Go.
>> > > >
>> > > > But apart from this kind of experiments (that are valid IMO), the
>> > "Kamel"
>> > > > optimizer will have free room to choose the right platform for the
>> > > > integration that the user wants to run, including, in the future,
>> doing
>> > > AOT
>> > > > compilation using Graal/VM (less memory, faster startup) if the
>> > features
>> > > > (components) used in the integration are supporting it (maybe we can
>> > add
>> > > > AOT compilation in the roadmap for Camel 3).
>> > > > A silly optimization: integrations starting from "timer:..." may be
>> > > > scheduled directly with Kubernetes CronJobs, so they will consume
>> > > resources
>> > > > only when actually running.
>> > > >
>> > > > Being the final integrations lightweight and being the DSL
>> > > > language-independent, we may see a increased adoption of Camel also
>> as
>> > > > agile integration layer for not-only-java applications (both "cloud"
>> > and
>> > > > "serverless" applications).
>> > > >
>> > > > I'm the first one that would like to work on a project ilke this.
>> I've
>> > > > worked on many Kubernetes/Openshift based applications and frameworks
>> > in
>> > > > the past years, also on operators and CRDs, and I think this way of
>> > > > redesigning integrations has a lot of potential.
>> > > >
>> > > > Integrations will not be necessarily limited to the simplified DSL,
>> but
>> > > we
>> > > > can add extension points for scripting and even custom libraries
>> > > (although
>> > > > limiting the freedom of the optimizer).
>> > > >
>> > > > The most important thing: it may become a great project, since it's
>> > > driven
>> > > > by a great community.
>> > > >
>> > > > So, what do you think? Is it crazy enough?
>> > > >
>> > > > Nicola
>> > >
>> > >
>> >
>> > --
>> > Hiram Chirino
>> > Engineering | Red Hat, Inc.
>> > hchirino@redhat.com | fusesource.com | redhat.com
>> > skype: hiramchirino | twitter: @hiramchirino
>> >
>>
Re: A "Kamel" crazy idea
Posted by Andrea Cosentino <an...@yahoo.com.INVALID>.
This is a wonderful idea and I believe we can improve our community with a subproject like this.
So +1 for me.
--
Andrea Cosentino
----------------------------------
Apache Camel PMC Chair
Apache Karaf Committer
Apache Servicemix PMC Member
Email: ancosen1985@yahoo.com
Twitter: @oscerd2
Github: oscerd
On Monday, July 30, 2018, 5:49:50 PM GMT+2, Nicola Ferraro <ni...@gmail.com> wrote:
Hi Cameleers,
it seems from the comments that this "Kamel" subproject is something we
want to start and I think that also the main camel core will benefit from
the new features it will bring.
I would like to donate the current "Kamel" code to Apache Camel, in order
to have a initial brick to start from.
From your reactions, the name "Kamel" seems cool and I'd like to keep it.
However if you talk to people about the "Kamel" project, they won't
understand if you're talking about "Kamel with the K" or "Camel with the C".
For this reasons I propose to keep "Kamel", but also use "Camel K" [ˈkæməl
keɪ] as friendly name when talking about it. This is in line with the
repository we should create for the subproject, that needs to start with
"camel-" (it will be "apache/camel-k" in that case).
These days I've been experimenting different ideas with Luca Burgazzoli. In
particular, there have been some concerns here on the power of a
declarative DSL (a new one or the existing XML one) and we've found a
strategy that will allow us to use also the Java DSL in Kamel integrations.
Without adding too many details here, at integration build time we can run
the user code in a build container and inspect the produced routes to get
metadata for the optimizer. This is one of the first things we should do
next.
This possibility of running code that produces integrations opened a lot of
unexpected paths, that I've written down in the project roadmap (
https://github.com/nicolaferraro/integration-operator) and summarized below.
One nice feature (kudos to Luca) is that we can simplify life for Kamel
users up to the point that they'll just need to write their routes on a
Java file and run them with e.g.:
"kamel run Routes.java"
Where "kamel" is a binary we release within the project. Kubernetes custom
resources will be used under the hood, but the "kamel" binary is a utility
that will provide a user experience comparable to that of current
serverless platforms (or even better).
But there's not just that.
Luca wrote this weekend a prototype for having a polyglot Camel (
https://github.com/lburgazzoli/camel-routes-loader).
With that, you will be able to e.g. write integrations in groovy and use
the same Kamel engine:
"kamel run routes.groovy"
But also JavaScript:
https://github.com/lburgazzoli/camel-routes-loader/blob/ca986541f7c422ee02c21727cdfe4293d64a364e/src/test/resources/ext/camel/routes.js#L2
This is a prototype right now, but a similar approach based on GraalVM has
a lot of potential, because users can not only use their preferred language
to write routes and processors, but also bind them to functionalities
available in their preferred libraries.
How much this will be feasible depends on the adoption of GraalVM, but I'm
seeing many frameworks adding metadata to make GraalVM work with
reflection. We've also done some work in Camel, there are some Jiras for it
and first tests made by Guillaume signal that it's something feasible, at
least for the Camel core and a subset of components. GraalVM would be
important also to reduce memory footprint and improve startup time, as
already said.
If the GraalVM approach works (it is working for other frameworks), instead
of rewriting a subset of Camel in Go (as the original proposal mentions),
we can just sanitize and recompile our existing codebase: this way we fully
leverage the strength of Apache Camel.
Another important thing in the roadmap is actually how we'll integrate with
current FaaS platforms. We will evaluate together if it's better to have a
tighter integration with some of them or to leverage knative for some
aspects of Kamel...
But, let's get started!
Il 19 lug 2018 10:45, "Nicola Ferraro" <ni...@gmail.com> ha scritto:
It's clear to me that we need to add support for our existing XML DSL, that
is powerful. But there are multiple reasons why I'd like to also "add" a
limited yaml/json notation to "Kamel".
The first one (and simplest) is that json/yaml is the primary encoding for
all resources exchanged with the Kubernetes API server. I'm not saying that
for this reason the Integration resource cannot contain a XML
(/Java/Ballerina) section, but from a pure presentational point of view,
having the possibility to write simple use cases (even a "hello world") in
pure json/yaml is better than always requiring a mix of json and XML.
The second one is simplicity. Writing a optimizer (the module that
materializes the integration into running code, choosing a specific runtime
and configuration) for a turing-complete language like Java is not easy:
even if you manage to create a good parser, it may be able to do
optimizations only if you write routes in a particular way, without complex
constructs.
XML is ok from this point of view. The thing is that both json/yaml and XML
are just two different ways to serialize object trees, that can be then
statically analyzed.
The point is not XML vs json/yaml, it's more about tailoring a new
minimalistic DSL to the emerging use cases vs proposing "only" our classic
way of writing integrations. I think XML can be the "advanced" way. We can
experiment optimizations easily with the new DSL, and enable them also on
XML if it's worth.
Scripting should be part of the spec, but I'd try to use programming
languages only for processing/transformation, not for the route definition.
Third one is performance. Apart from the fact that json parsers are in
general said to be faster than XML parsers... Given the "fast startup"
target that we want to reach, we may think e.g. to translate the new DSL
into Java or Go code, then compiling it. This allows doing parsing at build
time in order to avoid it on startup. This kind of improvements are much
easier with a limited DSL but much more difficult with a existing
fully-fledged DSL..
Nicola
On Wed, Jul 18, 2018 at 11:51 PM Pontus Ullgren <ul...@gmail.com> wrote:
> I also like the idea but with some comments.
>
> As Hiram Chirino I'm not sure YAML/JSON is the best language for this.
> Perhaps a more fluent DSL like the Camel Java DSL or perhaps something like
> Ballerina language would be better suited ?
> Also in my experience even simple integrations, that is simple real world
> integration and not just hello world demos, requires
> you to add one or more Java class or scripting in addition to the core
> components. So for it to be useful there must be some way to add custom
> code in some way for aggregation/enrichment strategies.
>
> If you go with a GO implementation I would also like some fallback for
> using Java since I find it unlikely that the existing base of components
> will be ported any time soon.
>
> As for the name as other I like the "Kamel" name however a being a newborn
> kotlin fan I know of one pre-existing project with that name
> that is close enough to allow for potential confusion.
> https://github.com/topicusoverheid/kamel
>
> Looking forward to see the result.
>
> // Pontus
>
>
>
>
>
>
> On Mon, 16 Jul 2018 at 15:15 Hiram Chirino <hi...@hiramchirino.com> wrote:
>
> > Love the idea.
> >
> > Personally, I'd keep using the existing Camel XML DSL if possible. You
> can
> > still embed it in the CRD. The operator that deploys it could inspect it
> > and figure whats the most optimized runtime that can support the DSL.
> > Perhaps if it's only using the restricted set of camel components
> supported
> > by camel-go (https://github.com/lburgazzoli/camel-go) then is uses that.
> > Otherwise it falls back to using camel spring boot.
> >
> > For that to work I think we need the different runtime implementations to
> > provide a way to ask them: 'hey do you support running this camel route?'
> > Not a trivial thing to respond to, it might require a build step in there
> > for traditional Camel runtimes.
> >
> >
> > On Fri, Jul 13, 2018 at 4:32 AM Antonin Stefanutti <
> antonin@stefanutti.fr>
> > wrote:
> >
> > > Hi Nicola,
> > >
> > > I love the idea.
> > >
> > > I just wonder whether YAML/JSON is an expressive enough format in the
> > long
> > > term. But as you’ve mentioned, starting simple would enable
> experimenting
> > > some very interesting / promising optimisations. So it seems worth
> taking
> > > that path, instead of trying to embed a complex DSL or the existing XML
> > DSL
> > > into the CRD.
> > >
> > > Definitely +1
> > >
> > > > On 13 Jul 2018, at 01:30, Nicola Ferraro <ni...@gmail.com>
> wrote:
> > > >
> > > > Hi Cameleers,
> > > > it's now passed some time since I started thinking about a new
> project
> > > that
> > > > we can begin here at Apache Camel, and I'd like to have your opinion.
> > > >
> > > > We've already been targeting cloud-native applications with Camel,
> > > > especially on top of Kubernetes, that is becoming "the standard"
> cloud
> > > > platform. But writing a Camel integration and running it on
> Kubernetes
> > > > requires some effort: choosing the base platform (spring-boot, karaf,
> > > > simple main?), adding health checks (actuator?), packaging a docker
> > image
> > > > and creating the Kubernetes resources (fabric8-maven-plugin, helm?),
> > > > publishing the image on a docker registry, then finally deploying the
> > > > resources on a Kubernetes cluster.
> > > >
> > > > The resulting integration container is then far from being optimal
> > from a
> > > > resource consumption point of view: it is likely that a Camel
> > Spring-Boot
> > > > application will require at least 200MB of RAM and also some CPU
> shares
> > > > because of polling threads used by many components.
> > > >
> > > > In case people use a CI/CD pipeline, it will take also a long time to
> > get
> > > > from a code update to having a Kubernetes POD up and running.
> > > > Apart from compilation and image push/pull time, also startup time is
> > > often
> > > > ~10 seconds for Camel + Spring-Boot in a container with standard
> limits
> > > on
> > > > resources, making it difficult to propose this combination for
> > > "serverless
> > > > integration" (this term is becoming increasingly more popular).
> > > >
> > > > So, my proposal is to start to investigate a "more cloud-native"
> > approach
> > > > to integration: *making Camel integrations first-class citizens in
> > > > Kubernetes, and making them super fast and lightweight.*
> > > >
> > > > We can base the project on Kubernetes Custom Resource Definitions
> (CRD)
> > > > <
> > >
> >
> https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/
> > > >,
> > > > for example a Integration CRD and have a Kubernetes "operator"
> > > > <https://coreos.com/operators/> taking care of:
> > > > - Optimizing the integration that we want to run
> > > > - Packaging in a container
> > > > - Running it on Kubernetes
> > > > - Managing its entire lifecycle
> > > >
> > > > A Kubernetes-native integration may look like:
> > > >
> > > > -------------------
> > > > kind: "Integration"
> > > > apiVersion: "camel.apache.org/v1alpha1"
> > > > metadata:
> > > > name: "example"
> > > > spec:
> > > > replicas: 1
> > > > routes:
> > > > - id: timer
> > > > route:
> > > > - type: endpoint
> > > > uri: timer:tick
> > > > - type: endpoint
> > > > uri: log:info
> > > > -------------------
> > > >
> > > > For those who are not familiar with Kubernetes resources, this kind
> of
> > > > YAML/JSON resource definitions are really common.
> > > > The example route is embedded in the Kubernetes resource declaration
> > and
> > > > follows a basic "flow DSL". We may start from a basic one and evolve
> it
> > > as
> > > > new requirements arrive from the community.
> > > >
> > > > I've made a very simple (but working) POC here:
> > > > https://github.com/nicolaferraro/integration-operator.
> > > >
> > > > This idea of a "Cloud-Native Camel" on Kubernetes (project codename
> can
> > > be "
> > > > *Kamel*", if you like it :D), will be an enabler for a lot of nice
> > > features.
> > > >
> > > > For example, we can propose "Kamel" as "ideal" platform for
> "serverless
> > > > integration" (I see many people reinventing the wheel out there): the
> > > > operator can reduce resource consumption of a single integration by
> > > > optimizing the runtime and also pause/resume integrations when they
> are
> > > not
> > > > used, that is the basic idea behind "serverless" (e.g. think to
> > > > HTTP-triggered integrations, but not only).
> > > > Focusing on serverless will bring more emphasis on push-based
> > > notifications
> > > > (webhooks, cloud events <https://cloudevents.io/>), that are rarely
> > > used in
> > > > Camel components, that prefer a poll based approach being it simpler
> to
> > > use
> > > > in classic deployments, but not so good in the cloud, where more
> > > resources
> > > > become higher direct costs for the users.
> > > >
> > > > The presence of the simplified DSL enables also experimenting on
> > > "*reduced*
> > > > subsets of Camel" implemented in languages other than Java, for
> example
> > > one
> > > > language that has a reactive approach on thread scheduling and a
> really
> > > low
> > > > memory footprint, like Go.
> > > >
> > > > But apart from this kind of experiments (that are valid IMO), the
> > "Kamel"
> > > > optimizer will have free room to choose the right platform for the
> > > > integration that the user wants to run, including, in the future,
> doing
> > > AOT
> > > > compilation using Graal/VM (less memory, faster startup) if the
> > features
> > > > (components) used in the integration are supporting it (maybe we can
> > add
> > > > AOT compilation in the roadmap for Camel 3).
> > > > A silly optimization: integrations starting from "timer:..." may be
> > > > scheduled directly with Kubernetes CronJobs, so they will consume
> > > resources
> > > > only when actually running.
> > > >
> > > > Being the final integrations lightweight and being the DSL
> > > > language-independent, we may see a increased adoption of Camel also
> as
> > > > agile integration layer for not-only-java applications (both "cloud"
> > and
> > > > "serverless" applications).
> > > >
> > > > I'm the first one that would like to work on a project ilke this.
> I've
> > > > worked on many Kubernetes/Openshift based applications and frameworks
> > in
> > > > the past years, also on operators and CRDs, and I think this way of
> > > > redesigning integrations has a lot of potential.
> > > >
> > > > Integrations will not be necessarily limited to the simplified DSL,
> but
> > > we
> > > > can add extension points for scripting and even custom libraries
> > > (although
> > > > limiting the freedom of the optimizer).
> > > >
> > > > The most important thing: it may become a great project, since it's
> > > driven
> > > > by a great community.
> > > >
> > > > So, what do you think? Is it crazy enough?
> > > >
> > > > Nicola
> > >
> > >
> >
> > --
> > Hiram Chirino
> > Engineering | Red Hat, Inc.
> > hchirino@redhat.com | fusesource.com | redhat.com
> > skype: hiramchirino | twitter: @hiramchirino
> >
>
Re: A "Kamel" crazy idea
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
About GraalVM then I just spotted this tweet about Spring Framework
5.1 being compatible
https://twitter.com/java/status/1023632246325035008
--
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2
Re: A "Kamel" crazy idea
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
+1 for me
On Mon, Jul 30, 2018 at 5:49 PM, Nicola Ferraro <ni...@gmail.com> wrote:
> Hi Cameleers,
> it seems from the comments that this "Kamel" subproject is something we
> want to start and I think that also the main camel core will benefit from
> the new features it will bring.
>
> I would like to donate the current "Kamel" code to Apache Camel, in order
> to have a initial brick to start from.
>
> From your reactions, the name "Kamel" seems cool and I'd like to keep it.
> However if you talk to people about the "Kamel" project, they won't
> understand if you're talking about "Kamel with the K" or "Camel with the C".
> For this reasons I propose to keep "Kamel", but also use "Camel K" [ˈkæməl
> keɪ] as friendly name when talking about it. This is in line with the
> repository we should create for the subproject, that needs to start with
> "camel-" (it will be "apache/camel-k" in that case).
>
Yeah that is really a good idea. I like the camel-k name. That brands
it to Apache Camel.
> These days I've been experimenting different ideas with Luca Burgazzoli. In
> particular, there have been some concerns here on the power of a
> declarative DSL (a new one or the existing XML one) and we've found a
> strategy that will allow us to use also the Java DSL in Kamel integrations.
> Without adding too many details here, at integration build time we can run
> the user code in a build container and inspect the produced routes to get
> metadata for the optimizer. This is one of the first things we should do
> next.
>
> This possibility of running code that produces integrations opened a lot of
> unexpected paths, that I've written down in the project roadmap (
> https://github.com/nicolaferraro/integration-operator) and summarized below.
>
Thanks that is a good summary.
> One nice feature (kudos to Luca) is that we can simplify life for Kamel
> users up to the point that they'll just need to write their routes on a
> Java file and run them with e.g.:
> "kamel run Routes.java"
>
> Where "kamel" is a binary we release within the project. Kubernetes custom
> resources will be used under the hood, but the "kamel" binary is a utility
> that will provide a user experience comparable to that of current
> serverless platforms (or even better).
>
Yeah I love the CLI - Other function and serverless projects have that too.
> But there's not just that.
>
> Luca wrote this weekend a prototype for having a polyglot Camel (
> https://github.com/lburgazzoli/camel-routes-loader).
>
> With that, you will be able to e.g. write integrations in groovy and use
> the same Kamel engine:
> "kamel run routes.groovy"
>
That will make James and the groovy guys happy, to have more love for
Groovy in Camel ;)
> But also JavaScript:
> https://github.com/lburgazzoli/camel-routes-loader/blob/ca986541f7c422ee02c21727cdfe4293d64a364e/src/test/resources/ext/camel/routes.js#L2
>
> This is a prototype right now, but a similar approach based on GraalVM has
> a lot of potential, because users can not only use their preferred language
> to write routes and processors, but also bind them to functionalities
> available in their preferred libraries.
>
> How much this will be feasible depends on the adoption of GraalVM, but I'm
> seeing many frameworks adding metadata to make GraalVM work with
> reflection. We've also done some work in Camel, there are some Jiras for it
> and first tests made by Guillaume signal that it's something feasible, at
> least for the Camel core and a subset of components. GraalVM would be
> important also to reduce memory footprint and improve startup time, as
> already said.
>
Yeah GraalVM is very interesting.
> If the GraalVM approach works (it is working for other frameworks), instead
> of rewriting a subset of Camel in Go (as the original proposal mentions),
> we can just sanitize and recompile our existing codebase: this way we fully
> leverage the strength of Apache Camel.
>
+1 a rewrite in go of Camel will be hard as all the components are Java based.
So if can keep it as Java that would be great IMHO.
> Another important thing in the roadmap is actually how we'll integrate with
> current FaaS platforms. We will evaluate together if it's better to have a
> tighter integration with some of them or to leverage knative for some
> aspects of Kamel...
>
Yeah what happens in CFCN and around Kubernetes moves so fast, that it sometimes
seems more crazy than the last web-framework for node or build tools.
> But, let's get started!
>
Yep sure, and this work helps drives the need for getting Camel 3.0 started too.
We need both legs to stand and have Camel's keep running yet another 10 years.
> Il 19 lug 2018 10:45, "Nicola Ferraro" <ni...@gmail.com> ha scritto:
>
> It's clear to me that we need to add support for our existing XML DSL, that
> is powerful. But there are multiple reasons why I'd like to also "add" a
> limited yaml/json notation to "Kamel".
>
> The first one (and simplest) is that json/yaml is the primary encoding for
> all resources exchanged with the Kubernetes API server. I'm not saying that
> for this reason the Integration resource cannot contain a XML
> (/Java/Ballerina) section, but from a pure presentational point of view,
> having the possibility to write simple use cases (even a "hello world") in
> pure json/yaml is better than always requiring a mix of json and XML.
>
> The second one is simplicity. Writing a optimizer (the module that
> materializes the integration into running code, choosing a specific runtime
> and configuration) for a turing-complete language like Java is not easy:
> even if you manage to create a good parser, it may be able to do
> optimizations only if you write routes in a particular way, without complex
> constructs.
> XML is ok from this point of view. The thing is that both json/yaml and XML
> are just two different ways to serialize object trees, that can be then
> statically analyzed.
> The point is not XML vs json/yaml, it's more about tailoring a new
> minimalistic DSL to the emerging use cases vs proposing "only" our classic
> way of writing integrations. I think XML can be the "advanced" way. We can
> experiment optimizations easily with the new DSL, and enable them also on
> XML if it's worth.
>
> Scripting should be part of the spec, but I'd try to use programming
> languages only for processing/transformation, not for the route definition.
>
> Third one is performance. Apart from the fact that json parsers are in
> general said to be faster than XML parsers... Given the "fast startup"
> target that we want to reach, we may think e.g. to translate the new DSL
> into Java or Go code, then compiling it. This allows doing parsing at build
> time in order to avoid it on startup. This kind of improvements are much
> easier with a limited DSL but much more difficult with a existing
> fully-fledged DSL..
>
> Nicola
>
>
>
> On Wed, Jul 18, 2018 at 11:51 PM Pontus Ullgren <ul...@gmail.com> wrote:
>
>> I also like the idea but with some comments.
>>
>> As Hiram Chirino I'm not sure YAML/JSON is the best language for this.
>> Perhaps a more fluent DSL like the Camel Java DSL or perhaps something like
>> Ballerina language would be better suited ?
>> Also in my experience even simple integrations, that is simple real world
>> integration and not just hello world demos, requires
>> you to add one or more Java class or scripting in addition to the core
>> components. So for it to be useful there must be some way to add custom
>> code in some way for aggregation/enrichment strategies.
>>
>> If you go with a GO implementation I would also like some fallback for
>> using Java since I find it unlikely that the existing base of components
>> will be ported any time soon.
>>
>> As for the name as other I like the "Kamel" name however a being a newborn
>> kotlin fan I know of one pre-existing project with that name
>> that is close enough to allow for potential confusion.
>> https://github.com/topicusoverheid/kamel
>>
>> Looking forward to see the result.
>>
>> // Pontus
>>
>>
>>
>>
>>
>>
>> On Mon, 16 Jul 2018 at 15:15 Hiram Chirino <hi...@hiramchirino.com> wrote:
>>
>> > Love the idea.
>> >
>> > Personally, I'd keep using the existing Camel XML DSL if possible. You
>> can
>> > still embed it in the CRD. The operator that deploys it could inspect it
>> > and figure whats the most optimized runtime that can support the DSL.
>> > Perhaps if it's only using the restricted set of camel components
>> supported
>> > by camel-go (https://github.com/lburgazzoli/camel-go) then is uses that.
>> > Otherwise it falls back to using camel spring boot.
>> >
>> > For that to work I think we need the different runtime implementations to
>> > provide a way to ask them: 'hey do you support running this camel route?'
>> > Not a trivial thing to respond to, it might require a build step in there
>> > for traditional Camel runtimes.
>> >
>> >
>> > On Fri, Jul 13, 2018 at 4:32 AM Antonin Stefanutti <
>> antonin@stefanutti.fr>
>> > wrote:
>> >
>> > > Hi Nicola,
>> > >
>> > > I love the idea.
>> > >
>> > > I just wonder whether YAML/JSON is an expressive enough format in the
>> > long
>> > > term. But as you’ve mentioned, starting simple would enable
>> experimenting
>> > > some very interesting / promising optimisations. So it seems worth
>> taking
>> > > that path, instead of trying to embed a complex DSL or the existing XML
>> > DSL
>> > > into the CRD.
>> > >
>> > > Definitely +1
>> > >
>> > > > On 13 Jul 2018, at 01:30, Nicola Ferraro <ni...@gmail.com>
>> wrote:
>> > > >
>> > > > Hi Cameleers,
>> > > > it's now passed some time since I started thinking about a new
>> project
>> > > that
>> > > > we can begin here at Apache Camel, and I'd like to have your opinion.
>> > > >
>> > > > We've already been targeting cloud-native applications with Camel,
>> > > > especially on top of Kubernetes, that is becoming "the standard"
>> cloud
>> > > > platform. But writing a Camel integration and running it on
>> Kubernetes
>> > > > requires some effort: choosing the base platform (spring-boot, karaf,
>> > > > simple main?), adding health checks (actuator?), packaging a docker
>> > image
>> > > > and creating the Kubernetes resources (fabric8-maven-plugin, helm?),
>> > > > publishing the image on a docker registry, then finally deploying the
>> > > > resources on a Kubernetes cluster.
>> > > >
>> > > > The resulting integration container is then far from being optimal
>> > from a
>> > > > resource consumption point of view: it is likely that a Camel
>> > Spring-Boot
>> > > > application will require at least 200MB of RAM and also some CPU
>> shares
>> > > > because of polling threads used by many components.
>> > > >
>> > > > In case people use a CI/CD pipeline, it will take also a long time to
>> > get
>> > > > from a code update to having a Kubernetes POD up and running.
>> > > > Apart from compilation and image push/pull time, also startup time is
>> > > often
>> > > > ~10 seconds for Camel + Spring-Boot in a container with standard
>> limits
>> > > on
>> > > > resources, making it difficult to propose this combination for
>> > > "serverless
>> > > > integration" (this term is becoming increasingly more popular).
>> > > >
>> > > > So, my proposal is to start to investigate a "more cloud-native"
>> > approach
>> > > > to integration: *making Camel integrations first-class citizens in
>> > > > Kubernetes, and making them super fast and lightweight.*
>> > > >
>> > > > We can base the project on Kubernetes Custom Resource Definitions
>> (CRD)
>> > > > <
>> > >
>> >
>> https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/
>> > > >,
>> > > > for example a Integration CRD and have a Kubernetes "operator"
>> > > > <https://coreos.com/operators/> taking care of:
>> > > > - Optimizing the integration that we want to run
>> > > > - Packaging in a container
>> > > > - Running it on Kubernetes
>> > > > - Managing its entire lifecycle
>> > > >
>> > > > A Kubernetes-native integration may look like:
>> > > >
>> > > > -------------------
>> > > > kind: "Integration"
>> > > > apiVersion: "camel.apache.org/v1alpha1"
>> > > > metadata:
>> > > > name: "example"
>> > > > spec:
>> > > > replicas: 1
>> > > > routes:
>> > > > - id: timer
>> > > > route:
>> > > > - type: endpoint
>> > > > uri: timer:tick
>> > > > - type: endpoint
>> > > > uri: log:info
>> > > > -------------------
>> > > >
>> > > > For those who are not familiar with Kubernetes resources, this kind
>> of
>> > > > YAML/JSON resource definitions are really common.
>> > > > The example route is embedded in the Kubernetes resource declaration
>> > and
>> > > > follows a basic "flow DSL". We may start from a basic one and evolve
>> it
>> > > as
>> > > > new requirements arrive from the community.
>> > > >
>> > > > I've made a very simple (but working) POC here:
>> > > > https://github.com/nicolaferraro/integration-operator.
>> > > >
>> > > > This idea of a "Cloud-Native Camel" on Kubernetes (project codename
>> can
>> > > be "
>> > > > *Kamel*", if you like it :D), will be an enabler for a lot of nice
>> > > features.
>> > > >
>> > > > For example, we can propose "Kamel" as "ideal" platform for
>> "serverless
>> > > > integration" (I see many people reinventing the wheel out there): the
>> > > > operator can reduce resource consumption of a single integration by
>> > > > optimizing the runtime and also pause/resume integrations when they
>> are
>> > > not
>> > > > used, that is the basic idea behind "serverless" (e.g. think to
>> > > > HTTP-triggered integrations, but not only).
>> > > > Focusing on serverless will bring more emphasis on push-based
>> > > notifications
>> > > > (webhooks, cloud events <https://cloudevents.io/>), that are rarely
>> > > used in
>> > > > Camel components, that prefer a poll based approach being it simpler
>> to
>> > > use
>> > > > in classic deployments, but not so good in the cloud, where more
>> > > resources
>> > > > become higher direct costs for the users.
>> > > >
>> > > > The presence of the simplified DSL enables also experimenting on
>> > > "*reduced*
>> > > > subsets of Camel" implemented in languages other than Java, for
>> example
>> > > one
>> > > > language that has a reactive approach on thread scheduling and a
>> really
>> > > low
>> > > > memory footprint, like Go.
>> > > >
>> > > > But apart from this kind of experiments (that are valid IMO), the
>> > "Kamel"
>> > > > optimizer will have free room to choose the right platform for the
>> > > > integration that the user wants to run, including, in the future,
>> doing
>> > > AOT
>> > > > compilation using Graal/VM (less memory, faster startup) if the
>> > features
>> > > > (components) used in the integration are supporting it (maybe we can
>> > add
>> > > > AOT compilation in the roadmap for Camel 3).
>> > > > A silly optimization: integrations starting from "timer:..." may be
>> > > > scheduled directly with Kubernetes CronJobs, so they will consume
>> > > resources
>> > > > only when actually running.
>> > > >
>> > > > Being the final integrations lightweight and being the DSL
>> > > > language-independent, we may see a increased adoption of Camel also
>> as
>> > > > agile integration layer for not-only-java applications (both "cloud"
>> > and
>> > > > "serverless" applications).
>> > > >
>> > > > I'm the first one that would like to work on a project ilke this.
>> I've
>> > > > worked on many Kubernetes/Openshift based applications and frameworks
>> > in
>> > > > the past years, also on operators and CRDs, and I think this way of
>> > > > redesigning integrations has a lot of potential.
>> > > >
>> > > > Integrations will not be necessarily limited to the simplified DSL,
>> but
>> > > we
>> > > > can add extension points for scripting and even custom libraries
>> > > (although
>> > > > limiting the freedom of the optimizer).
>> > > >
>> > > > The most important thing: it may become a great project, since it's
>> > > driven
>> > > > by a great community.
>> > > >
>> > > > So, what do you think? Is it crazy enough?
>> > > >
>> > > > Nicola
>> > >
>> > >
>> >
>> > --
>> > Hiram Chirino
>> > Engineering | Red Hat, Inc.
>> > hchirino@redhat.com | fusesource.com | redhat.com
>> > skype: hiramchirino | twitter: @hiramchirino
>> >
>>
--
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2
Re: A "Kamel" crazy idea
Posted by Luca Burgazzoli <lb...@gmail.com>.
---
Luca Burgazzoli
On Thu, Aug 2, 2018 at 8:09 AM, Guillaume Nodet <gn...@apache.org> wrote:
> In order to actually start the route, I forgot to add the required
> parameter:
>
> gnodet•camel/platforms/graalvm*(*graalvm*)*»
> ./example/target/org.apache.camel.graalvm.main -r
> org.apache.camel.graalvm.example.SimpleCamelRouteBuilder
>
> [main] INFO org.apache.camel.graalvm.FastCamelContext - Apache Camel
> 2.23.0-SNAPSHOT (CamelContext: camel-1) is starting
>
> [main] INFO org.apache.camel.graalvm.FastCamelContext - StreamCaching is
> not in use. If using streams then its recommended to enable stream caching.
> See more details at http://camel.apache.org/stream-caching.html
>
> [main] INFO org.apache.camel.graalvm.FastCamelContext - Route: route1
> started and consuming from: file://./target/orders
>
> [main] INFO org.apache.camel.graalvm.FastCamelContext - Total 1 routes, of
> which 1 are started
>
> [main] INFO org.apache.camel.graalvm.FastCamelContext - Apache Camel
> 2.23.0-SNAPSHOT (CamelContext: camel-1) started in 0.012 seconds
>
> Note that the binary is quite big: 26 Mb. One of the reason is that it
> contains the whole camel-core code. This is because the components /
> languages / dataformats are loaded dynamically, so the graalvm tool has to
> be informed of the implementation classes that can be discovered.
> Currently, all of them from camel-core are included, but I think a better
> way would be to discover them through the route model and only include the
> ones that are needed. I'll work on that.
>
I guess it can also be a job for a maven plugin to include only what
is listed as dependency. We need to move all the core components out
of camel-core of course but this is also a goal for camel 3 :)
> Le jeu. 2 août 2018 à 00:18, Luca Burgazzoli <lb...@gmail.com> a
> écrit :
>
>> That’s very nice start, thx Guillaume
>>
>> On Wed, 1 Aug 2018 at 20:15, Guillaume Nodet <gn...@apache.org> wrote:
>>
>> > I've pushed a branch with graalvm experiments.
>> > https://github.com/apache/camel/tree/graalvm
>> > In order to build the project, you need to use the GraalVM jdk and use
>> the
>> > following ~/.m2/toolchains.xml adapted to your path:
>> >
>> > <?xml version="1.0" encoding="UTF8"?>
>> >
>> > <toolchains>
>> >
>> > <toolchain>
>> >
>> > <type>jdk</type>
>> >
>> > <provides>
>> >
>> > <version>1.0.0-rc4</version>
>> >
>> > <vendor>oracle</vendor>
>> >
>> > <id>graalvm-ee-1.0.0-rc4</id>
>> >
>> > </provides>
>> >
>> > <configuration>
>> >
>> >
>> >
>> >
>> <jdkHome>/Users/gnodet/Downloads/graalvm-ee-1.0.0-rc4/Contents/Home/</jdkHome>
>> >
>> > </configuration>
>> >
>> > </toolchain>
>> >
>> > </toolchains>
>> >
>> > Build and run:
>> > gnodet•camel*(*graalvm*)*» cd platforms/graalvm
>> > gnodet•camel/platforms/graalvm*(*graalvm*)*» mvn install
>> > ...
>> > gnodet•camel/platforms/graalvm*(*graalvm*)*»
>> > ./example/target/org.apache.camel.graalvm.main
>> >
>> > [main] INFO org.apache.camel.graalvm.FastCamelContext - Apache Camel
>> > 2.23.0-SNAPSHOT (CamelContext: camel-1) is starting
>> >
>> > [main] INFO org.apache.camel.graalvm.FastCamelContext - StreamCaching is
>> > not in use. If using streams then its recommended to enable stream
>> caching.
>> > See more details at http://camel.apache.org/stream-caching.html
>> >
>> > [main] INFO org.apache.camel.graalvm.FastCamelContext - Total 0 routes,
>> of
>> > which 0 are started
>> >
>> > [main] INFO org.apache.camel.graalvm.FastCamelContext - Apache Camel
>> > 2.23.0-SNAPSHOT (CamelContext: camel-1) started in 0.006 seconds
>> >
>> > Guillaume
>> >
>> > Le mar. 31 juil. 2018 à 09:44, Guillaume Nodet <gn...@apache.org> a
>> > écrit :
>> >
>> > >
>> > >
>> > > Le lun. 30 juil. 2018 à 17:49, Nicola Ferraro <ni...@gmail.com> a
>> > > écrit :
>> > >
>> > >> Hi Cameleers,
>> > >> it seems from the comments that this "Kamel" subproject is something
>> we
>> > >> want to start and I think that also the main camel core will benefit
>> > from
>> > >> the new features it will bring.
>> > >>
>> > >> I would like to donate the current "Kamel" code to Apache Camel, in
>> > order
>> > >> to have a initial brick to start from.
>> > >>
>> > >> From your reactions, the name "Kamel" seems cool and I'd like to keep
>> > it.
>> > >> However if you talk to people about the "Kamel" project, they won't
>> > >> understand if you're talking about "Kamel with the K" or "Camel with
>> the
>> > >> C".
>> > >> For this reasons I propose to keep "Kamel", but also use "Camel K"
>> > [ˈkæməl
>> > >> keɪ] as friendly name when talking about it. This is in line with the
>> > >> repository we should create for the subproject, that needs to start
>> with
>> > >> "camel-" (it will be "apache/camel-k" in that case).
>> > >>
>> > >> These days I've been experimenting different ideas with Luca
>> Burgazzoli.
>> > >> In
>> > >> particular, there have been some concerns here on the power of a
>> > >> declarative DSL (a new one or the existing XML one) and we've found a
>> > >> strategy that will allow us to use also the Java DSL in Kamel
>> > >> integrations.
>> > >> Without adding too many details here, at integration build time we can
>> > run
>> > >> the user code in a build container and inspect the produced routes to
>> > get
>> > >> metadata for the optimizer. This is one of the first things we should
>> do
>> > >> next.
>> > >>
>> > >> This possibility of running code that produces integrations opened a
>> lot
>> > >> of
>> > >> unexpected paths, that I've written down in the project roadmap (
>> > >> https://github.com/nicolaferraro/integration-operator) and summarized
>> > >> below.
>> > >>
>> > >> One nice feature (kudos to Luca) is that we can simplify life for
>> Kamel
>> > >> users up to the point that they'll just need to write their routes on
>> a
>> > >> Java file and run them with e.g.:
>> > >> "kamel run Routes.java"
>> > >>
>> > >> Where "kamel" is a binary we release within the project. Kubernetes
>> > custom
>> > >> resources will be used under the hood, but the "kamel" binary is a
>> > utility
>> > >> that will provide a user experience comparable to that of current
>> > >> serverless platforms (or even better).
>> > >>
>> > >> But there's not just that.
>> > >>
>> > >> Luca wrote this weekend a prototype for having a polyglot Camel (
>> > >> https://github.com/lburgazzoli/camel-routes-loader).
>> > >>
>> > >> With that, you will be able to e.g. write integrations in groovy and
>> use
>> > >> the same Kamel engine:
>> > >> "kamel run routes.groovy"
>> > >>
>> > >> But also JavaScript:
>> > >>
>> > >>
>> >
>> https://github.com/lburgazzoli/camel-routes-loader/blob/ca986541f7c422ee02c21727cdfe4293d64a364e/src/test/resources/ext/camel/routes.js#L2
>> > >>
>> > >> This is a prototype right now, but a similar approach based on GraalVM
>> > has
>> > >> a lot of potential, because users can not only use their preferred
>> > >> language
>> > >> to write routes and processors, but also bind them to functionalities
>> > >> available in their preferred libraries.
>> > >>
>> > >> How much this will be feasible depends on the adoption of GraalVM, but
>> > I'm
>> > >> seeing many frameworks adding metadata to make GraalVM work with
>> > >> reflection. We've also done some work in Camel, there are some Jiras
>> for
>> > >> it
>> > >> and first tests made by Guillaume signal that it's something feasible,
>> > at
>> > >> least for the Camel core and a subset of components. GraalVM would be
>> > >> important also to reduce memory footprint and improve startup time, as
>> > >> already said.
>> > >>
>> > >
>> > > I'll continue experimenting and I'll report back.
>> > > I did some initial experiments leveraging some modifications I did to
>> > > speed up the start up time [1] and on a single route [2].
>> > > There may be some limitations down the road of course, but at least it
>> > > shows that it's feasible. Fwiw, the experiments lead to a startup time
>> of
>> > > 14 ms compared to a few hundreds milliseconds (roughly 680 with [1]).
>> > >
>> > >
>> > >>
>> > >> If the GraalVM approach works (it is working for other frameworks),
>> > >> instead
>> > >> of rewriting a subset of Camel in Go (as the original proposal
>> > mentions),
>> > >> we can just sanitize and recompile our existing codebase: this way we
>> > >> fully
>> > >> leverage the strength of Apache Camel.
>> > >>
>> > >
>> > > Right, if we can avoid rewriting Camel, that would be much better !
>> > >
>> > > [1] https://issues.apache.org/jira/browse/CAMEL-12688
>> > > [2]
>> > >
>> >
>> https://github.com/gnodet/openwhisk-runtime-camel/blob/master/camel-openwhisk-example/src/main/java/org/jboss/fuse/openwhisk/camel/example/SimpleCamelFunction.java
>> > >
>> > >
>> > >>
>> > >> Another important thing in the roadmap is actually how we'll integrate
>> > >> with
>> > >> current FaaS platforms. We will evaluate together if it's better to
>> > have a
>> > >> tighter integration with some of them or to leverage knative for some
>> > >> aspects of Kamel...
>> > >>
>> > >> But, let's get started!
>> > >>
>> > >> Il 19 lug 2018 10:45, "Nicola Ferraro" <ni...@gmail.com> ha
>> > scritto:
>> > >>
>> > >> It's clear to me that we need to add support for our existing XML DSL,
>> > >> that
>> > >> is powerful. But there are multiple reasons why I'd like to also
>> "add" a
>> > >> limited yaml/json notation to "Kamel".
>> > >>
>> > >> The first one (and simplest) is that json/yaml is the primary encoding
>> > for
>> > >> all resources exchanged with the Kubernetes API server. I'm not saying
>> > >> that
>> > >> for this reason the Integration resource cannot contain a XML
>> > >> (/Java/Ballerina) section, but from a pure presentational point of
>> view,
>> > >> having the possibility to write simple use cases (even a "hello
>> world")
>> > in
>> > >> pure json/yaml is better than always requiring a mix of json and XML.
>> > >>
>> > >> The second one is simplicity. Writing a optimizer (the module that
>> > >> materializes the integration into running code, choosing a specific
>> > >> runtime
>> > >> and configuration) for a turing-complete language like Java is not
>> easy:
>> > >> even if you manage to create a good parser, it may be able to do
>> > >> optimizations only if you write routes in a particular way, without
>> > >> complex
>> > >> constructs.
>> > >> XML is ok from this point of view. The thing is that both json/yaml
>> and
>> > >> XML
>> > >> are just two different ways to serialize object trees, that can be
>> then
>> > >> statically analyzed.
>> > >> The point is not XML vs json/yaml, it's more about tailoring a new
>> > >> minimalistic DSL to the emerging use cases vs proposing "only" our
>> > classic
>> > >> way of writing integrations. I think XML can be the "advanced" way. We
>> > can
>> > >> experiment optimizations easily with the new DSL, and enable them also
>> > on
>> > >> XML if it's worth.
>> > >>
>> > >> Scripting should be part of the spec, but I'd try to use programming
>> > >> languages only for processing/transformation, not for the route
>> > >> definition.
>> > >>
>> > >> Third one is performance. Apart from the fact that json parsers are in
>> > >> general said to be faster than XML parsers... Given the "fast startup"
>> > >> target that we want to reach, we may think e.g. to translate the new
>> DSL
>> > >> into Java or Go code, then compiling it. This allows doing parsing at
>> > >> build
>> > >> time in order to avoid it on startup. This kind of improvements are
>> much
>> > >> easier with a limited DSL but much more difficult with a existing
>> > >> fully-fledged DSL..
>> > >>
>> > >> Nicola
>> > >>
>> > >>
>> > >>
>> > >> On Wed, Jul 18, 2018 at 11:51 PM Pontus Ullgren <ul...@gmail.com>
>> > >> wrote:
>> > >>
>> > >> > I also like the idea but with some comments.
>> > >> >
>> > >> > As Hiram Chirino I'm not sure YAML/JSON is the best language for
>> this.
>> > >> > Perhaps a more fluent DSL like the Camel Java DSL or perhaps
>> something
>> > >> like
>> > >> > Ballerina language would be better suited ?
>> > >> > Also in my experience even simple integrations, that is simple real
>> > >> world
>> > >> > integration and not just hello world demos, requires
>> > >> > you to add one or more Java class or scripting in addition to the
>> core
>> > >> > components. So for it to be useful there must be some way to add
>> > custom
>> > >> > code in some way for aggregation/enrichment strategies.
>> > >> >
>> > >> > If you go with a GO implementation I would also like some fallback
>> for
>> > >> > using Java since I find it unlikely that the existing base of
>> > components
>> > >> > will be ported any time soon.
>> > >> >
>> > >> > As for the name as other I like the "Kamel" name however a being a
>> > >> newborn
>> > >> > kotlin fan I know of one pre-existing project with that name
>> > >> > that is close enough to allow for potential confusion.
>> > >> > https://github.com/topicusoverheid/kamel
>> > >> >
>> > >> > Looking forward to see the result.
>> > >> >
>> > >> > // Pontus
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> > On Mon, 16 Jul 2018 at 15:15 Hiram Chirino <hi...@hiramchirino.com>
>> > >> wrote:
>> > >> >
>> > >> > > Love the idea.
>> > >> > >
>> > >> > > Personally, I'd keep using the existing Camel XML DSL if possible.
>> > >> You
>> > >> > can
>> > >> > > still embed it in the CRD. The operator that deploys it could
>> > >> inspect it
>> > >> > > and figure whats the most optimized runtime that can support the
>> > DSL.
>> > >> > > Perhaps if it's only using the restricted set of camel components
>> > >> > supported
>> > >> > > by camel-go (https://github.com/lburgazzoli/camel-go) then is
>> uses
>> > >> that.
>> > >> > > Otherwise it falls back to using camel spring boot.
>> > >> > >
>> > >> > > For that to work I think we need the different runtime
>> > >> implementations to
>> > >> > > provide a way to ask them: 'hey do you support running this camel
>> > >> route?'
>> > >> > > Not a trivial thing to respond to, it might require a build step
>> in
>> > >> there
>> > >> > > for traditional Camel runtimes.
>> > >> > >
>> > >> > >
>> > >> > > On Fri, Jul 13, 2018 at 4:32 AM Antonin Stefanutti <
>> > >> > antonin@stefanutti.fr>
>> > >> > > wrote:
>> > >> > >
>> > >> > > > Hi Nicola,
>> > >> > > >
>> > >> > > > I love the idea.
>> > >> > > >
>> > >> > > > I just wonder whether YAML/JSON is an expressive enough format
>> in
>> > >> the
>> > >> > > long
>> > >> > > > term. But as you’ve mentioned, starting simple would enable
>> > >> > experimenting
>> > >> > > > some very interesting / promising optimisations. So it seems
>> worth
>> > >> > taking
>> > >> > > > that path, instead of trying to embed a complex DSL or the
>> > existing
>> > >> XML
>> > >> > > DSL
>> > >> > > > into the CRD.
>> > >> > > >
>> > >> > > > Definitely +1
>> > >> > > >
>> > >> > > > > On 13 Jul 2018, at 01:30, Nicola Ferraro <
>> ni.ferraro@gmail.com>
>> > >> > wrote:
>> > >> > > > >
>> > >> > > > > Hi Cameleers,
>> > >> > > > > it's now passed some time since I started thinking about a new
>> > >> > project
>> > >> > > > that
>> > >> > > > > we can begin here at Apache Camel, and I'd like to have your
>> > >> opinion.
>> > >> > > > >
>> > >> > > > > We've already been targeting cloud-native applications with
>> > Camel,
>> > >> > > > > especially on top of Kubernetes, that is becoming "the
>> standard"
>> > >> > cloud
>> > >> > > > > platform. But writing a Camel integration and running it on
>> > >> > Kubernetes
>> > >> > > > > requires some effort: choosing the base platform (spring-boot,
>> > >> karaf,
>> > >> > > > > simple main?), adding health checks (actuator?), packaging a
>> > >> docker
>> > >> > > image
>> > >> > > > > and creating the Kubernetes resources (fabric8-maven-plugin,
>> > >> helm?),
>> > >> > > > > publishing the image on a docker registry, then finally
>> > deploying
>> > >> the
>> > >> > > > > resources on a Kubernetes cluster.
>> > >> > > > >
>> > >> > > > > The resulting integration container is then far from being
>> > optimal
>> > >> > > from a
>> > >> > > > > resource consumption point of view: it is likely that a Camel
>> > >> > > Spring-Boot
>> > >> > > > > application will require at least 200MB of RAM and also some
>> CPU
>> > >> > shares
>> > >> > > > > because of polling threads used by many components.
>> > >> > > > >
>> > >> > > > > In case people use a CI/CD pipeline, it will take also a long
>> > >> time to
>> > >> > > get
>> > >> > > > > from a code update to having a Kubernetes POD up and running.
>> > >> > > > > Apart from compilation and image push/pull time, also startup
>> > >> time is
>> > >> > > > often
>> > >> > > > > ~10 seconds for Camel + Spring-Boot in a container with
>> standard
>> > >> > limits
>> > >> > > > on
>> > >> > > > > resources, making it difficult to propose this combination for
>> > >> > > > "serverless
>> > >> > > > > integration" (this term is becoming increasingly more
>> popular).
>> > >> > > > >
>> > >> > > > > So, my proposal is to start to investigate a "more
>> cloud-native"
>> > >> > > approach
>> > >> > > > > to integration: *making Camel integrations first-class
>> citizens
>> > in
>> > >> > > > > Kubernetes, and making them super fast and lightweight.*
>> > >> > > > >
>> > >> > > > > We can base the project on Kubernetes Custom Resource
>> > Definitions
>> > >> > (CRD)
>> > >> > > > > <
>> > >> > > >
>> > >> > >
>> > >> >
>> > >>
>> >
>> https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/
>> > >> > > > >,
>> > >> > > > > for example a Integration CRD and have a Kubernetes "operator"
>> > >> > > > > <https://coreos.com/operators/> taking care of:
>> > >> > > > > - Optimizing the integration that we want to run
>> > >> > > > > - Packaging in a container
>> > >> > > > > - Running it on Kubernetes
>> > >> > > > > - Managing its entire lifecycle
>> > >> > > > >
>> > >> > > > > A Kubernetes-native integration may look like:
>> > >> > > > >
>> > >> > > > > -------------------
>> > >> > > > > kind: "Integration"
>> > >> > > > > apiVersion: "camel.apache.org/v1alpha1"
>> > >> > > > > metadata:
>> > >> > > > > name: "example"
>> > >> > > > > spec:
>> > >> > > > > replicas: 1
>> > >> > > > > routes:
>> > >> > > > > - id: timer
>> > >> > > > > route:
>> > >> > > > > - type: endpoint
>> > >> > > > > uri: timer:tick
>> > >> > > > > - type: endpoint
>> > >> > > > > uri: log:info
>> > >> > > > > -------------------
>> > >> > > > >
>> > >> > > > > For those who are not familiar with Kubernetes resources, this
>> > >> kind
>> > >> > of
>> > >> > > > > YAML/JSON resource definitions are really common.
>> > >> > > > > The example route is embedded in the Kubernetes resource
>> > >> declaration
>> > >> > > and
>> > >> > > > > follows a basic "flow DSL". We may start from a basic one and
>> > >> evolve
>> > >> > it
>> > >> > > > as
>> > >> > > > > new requirements arrive from the community.
>> > >> > > > >
>> > >> > > > > I've made a very simple (but working) POC here:
>> > >> > > > > https://github.com/nicolaferraro/integration-operator.
>> > >> > > > >
>> > >> > > > > This idea of a "Cloud-Native Camel" on Kubernetes (project
>> > >> codename
>> > >> > can
>> > >> > > > be "
>> > >> > > > > *Kamel*", if you like it :D), will be an enabler for a lot of
>> > nice
>> > >> > > > features.
>> > >> > > > >
>> > >> > > > > For example, we can propose "Kamel" as "ideal" platform for
>> > >> > "serverless
>> > >> > > > > integration" (I see many people reinventing the wheel out
>> > there):
>> > >> the
>> > >> > > > > operator can reduce resource consumption of a single
>> integration
>> > >> by
>> > >> > > > > optimizing the runtime and also pause/resume integrations when
>> > >> they
>> > >> > are
>> > >> > > > not
>> > >> > > > > used, that is the basic idea behind "serverless" (e.g. think
>> to
>> > >> > > > > HTTP-triggered integrations, but not only).
>> > >> > > > > Focusing on serverless will bring more emphasis on push-based
>> > >> > > > notifications
>> > >> > > > > (webhooks, cloud events <https://cloudevents.io/>), that are
>> > >> rarely
>> > >> > > > used in
>> > >> > > > > Camel components, that prefer a poll based approach being it
>> > >> simpler
>> > >> > to
>> > >> > > > use
>> > >> > > > > in classic deployments, but not so good in the cloud, where
>> more
>> > >> > > > resources
>> > >> > > > > become higher direct costs for the users.
>> > >> > > > >
>> > >> > > > > The presence of the simplified DSL enables also experimenting
>> on
>> > >> > > > "*reduced*
>> > >> > > > > subsets of Camel" implemented in languages other than Java,
>> for
>> > >> > example
>> > >> > > > one
>> > >> > > > > language that has a reactive approach on thread scheduling
>> and a
>> > >> > really
>> > >> > > > low
>> > >> > > > > memory footprint, like Go.
>> > >> > > > >
>> > >> > > > > But apart from this kind of experiments (that are valid IMO),
>> > the
>> > >> > > "Kamel"
>> > >> > > > > optimizer will have free room to choose the right platform for
>> > the
>> > >> > > > > integration that the user wants to run, including, in the
>> > future,
>> > >> > doing
>> > >> > > > AOT
>> > >> > > > > compilation using Graal/VM (less memory, faster startup) if
>> the
>> > >> > > features
>> > >> > > > > (components) used in the integration are supporting it (maybe
>> we
>> > >> can
>> > >> > > add
>> > >> > > > > AOT compilation in the roadmap for Camel 3).
>> > >> > > > > A silly optimization: integrations starting from "timer:..."
>> may
>> > >> be
>> > >> > > > > scheduled directly with Kubernetes CronJobs, so they will
>> > consume
>> > >> > > > resources
>> > >> > > > > only when actually running.
>> > >> > > > >
>> > >> > > > > Being the final integrations lightweight and being the DSL
>> > >> > > > > language-independent, we may see a increased adoption of Camel
>> > >> also
>> > >> > as
>> > >> > > > > agile integration layer for not-only-java applications (both
>> > >> "cloud"
>> > >> > > and
>> > >> > > > > "serverless" applications).
>> > >> > > > >
>> > >> > > > > I'm the first one that would like to work on a project ilke
>> > this.
>> > >> > I've
>> > >> > > > > worked on many Kubernetes/Openshift based applications and
>> > >> frameworks
>> > >> > > in
>> > >> > > > > the past years, also on operators and CRDs, and I think this
>> way
>> > >> of
>> > >> > > > > redesigning integrations has a lot of potential.
>> > >> > > > >
>> > >> > > > > Integrations will not be necessarily limited to the simplified
>> > >> DSL,
>> > >> > but
>> > >> > > > we
>> > >> > > > > can add extension points for scripting and even custom
>> libraries
>> > >> > > > (although
>> > >> > > > > limiting the freedom of the optimizer).
>> > >> > > > >
>> > >> > > > > The most important thing: it may become a great project, since
>> > >> it's
>> > >> > > > driven
>> > >> > > > > by a great community.
>> > >> > > > >
>> > >> > > > > So, what do you think? Is it crazy enough?
>> > >> > > > >
>> > >> > > > > Nicola
>> > >> > > >
>> > >> > > >
>> > >> > >
>> > >> > > --
>> > >> > > Hiram Chirino
>> > >> > > Engineering | Red Hat, Inc.
>> > >> > > hchirino@redhat.com | fusesource.com | redhat.com
>> > >> > > skype: hiramchirino | twitter: @hiramchirino
>> > >> > >
>> > >> >
>> > >>
>> > >
>> > >
>> > > --
>> > > ------------------------
>> > > Guillaume Nodet
>> > >
>> > >
>> >
>> > --
>> > ------------------------
>> > Guillaume Nodet
>> >
>> --
>> --
>> Luca Burgazzoli
>>
>
>
> --
> ------------------------
> Guillaume Nodet
Re: A "Kamel" crazy idea
Posted by Guillaume Nodet <gn...@apache.org>.
In order to actually start the route, I forgot to add the required
parameter:
gnodet•camel/platforms/graalvm*(*graalvm*)*»
./example/target/org.apache.camel.graalvm.main -r
org.apache.camel.graalvm.example.SimpleCamelRouteBuilder
[main] INFO org.apache.camel.graalvm.FastCamelContext - Apache Camel
2.23.0-SNAPSHOT (CamelContext: camel-1) is starting
[main] INFO org.apache.camel.graalvm.FastCamelContext - StreamCaching is
not in use. If using streams then its recommended to enable stream caching.
See more details at http://camel.apache.org/stream-caching.html
[main] INFO org.apache.camel.graalvm.FastCamelContext - Route: route1
started and consuming from: file://./target/orders
[main] INFO org.apache.camel.graalvm.FastCamelContext - Total 1 routes, of
which 1 are started
[main] INFO org.apache.camel.graalvm.FastCamelContext - Apache Camel
2.23.0-SNAPSHOT (CamelContext: camel-1) started in 0.012 seconds
Note that the binary is quite big: 26 Mb. One of the reason is that it
contains the whole camel-core code. This is because the components /
languages / dataformats are loaded dynamically, so the graalvm tool has to
be informed of the implementation classes that can be discovered.
Currently, all of them from camel-core are included, but I think a better
way would be to discover them through the route model and only include the
ones that are needed. I'll work on that.
Le jeu. 2 août 2018 à 00:18, Luca Burgazzoli <lb...@gmail.com> a
écrit :
> That’s very nice start, thx Guillaume
>
> On Wed, 1 Aug 2018 at 20:15, Guillaume Nodet <gn...@apache.org> wrote:
>
> > I've pushed a branch with graalvm experiments.
> > https://github.com/apache/camel/tree/graalvm
> > In order to build the project, you need to use the GraalVM jdk and use
> the
> > following ~/.m2/toolchains.xml adapted to your path:
> >
> > <?xml version="1.0" encoding="UTF8"?>
> >
> > <toolchains>
> >
> > <toolchain>
> >
> > <type>jdk</type>
> >
> > <provides>
> >
> > <version>1.0.0-rc4</version>
> >
> > <vendor>oracle</vendor>
> >
> > <id>graalvm-ee-1.0.0-rc4</id>
> >
> > </provides>
> >
> > <configuration>
> >
> >
> >
> >
> <jdkHome>/Users/gnodet/Downloads/graalvm-ee-1.0.0-rc4/Contents/Home/</jdkHome>
> >
> > </configuration>
> >
> > </toolchain>
> >
> > </toolchains>
> >
> > Build and run:
> > gnodet•camel*(*graalvm*)*» cd platforms/graalvm
> > gnodet•camel/platforms/graalvm*(*graalvm*)*» mvn install
> > ...
> > gnodet•camel/platforms/graalvm*(*graalvm*)*»
> > ./example/target/org.apache.camel.graalvm.main
> >
> > [main] INFO org.apache.camel.graalvm.FastCamelContext - Apache Camel
> > 2.23.0-SNAPSHOT (CamelContext: camel-1) is starting
> >
> > [main] INFO org.apache.camel.graalvm.FastCamelContext - StreamCaching is
> > not in use. If using streams then its recommended to enable stream
> caching.
> > See more details at http://camel.apache.org/stream-caching.html
> >
> > [main] INFO org.apache.camel.graalvm.FastCamelContext - Total 0 routes,
> of
> > which 0 are started
> >
> > [main] INFO org.apache.camel.graalvm.FastCamelContext - Apache Camel
> > 2.23.0-SNAPSHOT (CamelContext: camel-1) started in 0.006 seconds
> >
> > Guillaume
> >
> > Le mar. 31 juil. 2018 à 09:44, Guillaume Nodet <gn...@apache.org> a
> > écrit :
> >
> > >
> > >
> > > Le lun. 30 juil. 2018 à 17:49, Nicola Ferraro <ni...@gmail.com> a
> > > écrit :
> > >
> > >> Hi Cameleers,
> > >> it seems from the comments that this "Kamel" subproject is something
> we
> > >> want to start and I think that also the main camel core will benefit
> > from
> > >> the new features it will bring.
> > >>
> > >> I would like to donate the current "Kamel" code to Apache Camel, in
> > order
> > >> to have a initial brick to start from.
> > >>
> > >> From your reactions, the name "Kamel" seems cool and I'd like to keep
> > it.
> > >> However if you talk to people about the "Kamel" project, they won't
> > >> understand if you're talking about "Kamel with the K" or "Camel with
> the
> > >> C".
> > >> For this reasons I propose to keep "Kamel", but also use "Camel K"
> > [ˈkæməl
> > >> keɪ] as friendly name when talking about it. This is in line with the
> > >> repository we should create for the subproject, that needs to start
> with
> > >> "camel-" (it will be "apache/camel-k" in that case).
> > >>
> > >> These days I've been experimenting different ideas with Luca
> Burgazzoli.
> > >> In
> > >> particular, there have been some concerns here on the power of a
> > >> declarative DSL (a new one or the existing XML one) and we've found a
> > >> strategy that will allow us to use also the Java DSL in Kamel
> > >> integrations.
> > >> Without adding too many details here, at integration build time we can
> > run
> > >> the user code in a build container and inspect the produced routes to
> > get
> > >> metadata for the optimizer. This is one of the first things we should
> do
> > >> next.
> > >>
> > >> This possibility of running code that produces integrations opened a
> lot
> > >> of
> > >> unexpected paths, that I've written down in the project roadmap (
> > >> https://github.com/nicolaferraro/integration-operator) and summarized
> > >> below.
> > >>
> > >> One nice feature (kudos to Luca) is that we can simplify life for
> Kamel
> > >> users up to the point that they'll just need to write their routes on
> a
> > >> Java file and run them with e.g.:
> > >> "kamel run Routes.java"
> > >>
> > >> Where "kamel" is a binary we release within the project. Kubernetes
> > custom
> > >> resources will be used under the hood, but the "kamel" binary is a
> > utility
> > >> that will provide a user experience comparable to that of current
> > >> serverless platforms (or even better).
> > >>
> > >> But there's not just that.
> > >>
> > >> Luca wrote this weekend a prototype for having a polyglot Camel (
> > >> https://github.com/lburgazzoli/camel-routes-loader).
> > >>
> > >> With that, you will be able to e.g. write integrations in groovy and
> use
> > >> the same Kamel engine:
> > >> "kamel run routes.groovy"
> > >>
> > >> But also JavaScript:
> > >>
> > >>
> >
> https://github.com/lburgazzoli/camel-routes-loader/blob/ca986541f7c422ee02c21727cdfe4293d64a364e/src/test/resources/ext/camel/routes.js#L2
> > >>
> > >> This is a prototype right now, but a similar approach based on GraalVM
> > has
> > >> a lot of potential, because users can not only use their preferred
> > >> language
> > >> to write routes and processors, but also bind them to functionalities
> > >> available in their preferred libraries.
> > >>
> > >> How much this will be feasible depends on the adoption of GraalVM, but
> > I'm
> > >> seeing many frameworks adding metadata to make GraalVM work with
> > >> reflection. We've also done some work in Camel, there are some Jiras
> for
> > >> it
> > >> and first tests made by Guillaume signal that it's something feasible,
> > at
> > >> least for the Camel core and a subset of components. GraalVM would be
> > >> important also to reduce memory footprint and improve startup time, as
> > >> already said.
> > >>
> > >
> > > I'll continue experimenting and I'll report back.
> > > I did some initial experiments leveraging some modifications I did to
> > > speed up the start up time [1] and on a single route [2].
> > > There may be some limitations down the road of course, but at least it
> > > shows that it's feasible. Fwiw, the experiments lead to a startup time
> of
> > > 14 ms compared to a few hundreds milliseconds (roughly 680 with [1]).
> > >
> > >
> > >>
> > >> If the GraalVM approach works (it is working for other frameworks),
> > >> instead
> > >> of rewriting a subset of Camel in Go (as the original proposal
> > mentions),
> > >> we can just sanitize and recompile our existing codebase: this way we
> > >> fully
> > >> leverage the strength of Apache Camel.
> > >>
> > >
> > > Right, if we can avoid rewriting Camel, that would be much better !
> > >
> > > [1] https://issues.apache.org/jira/browse/CAMEL-12688
> > > [2]
> > >
> >
> https://github.com/gnodet/openwhisk-runtime-camel/blob/master/camel-openwhisk-example/src/main/java/org/jboss/fuse/openwhisk/camel/example/SimpleCamelFunction.java
> > >
> > >
> > >>
> > >> Another important thing in the roadmap is actually how we'll integrate
> > >> with
> > >> current FaaS platforms. We will evaluate together if it's better to
> > have a
> > >> tighter integration with some of them or to leverage knative for some
> > >> aspects of Kamel...
> > >>
> > >> But, let's get started!
> > >>
> > >> Il 19 lug 2018 10:45, "Nicola Ferraro" <ni...@gmail.com> ha
> > scritto:
> > >>
> > >> It's clear to me that we need to add support for our existing XML DSL,
> > >> that
> > >> is powerful. But there are multiple reasons why I'd like to also
> "add" a
> > >> limited yaml/json notation to "Kamel".
> > >>
> > >> The first one (and simplest) is that json/yaml is the primary encoding
> > for
> > >> all resources exchanged with the Kubernetes API server. I'm not saying
> > >> that
> > >> for this reason the Integration resource cannot contain a XML
> > >> (/Java/Ballerina) section, but from a pure presentational point of
> view,
> > >> having the possibility to write simple use cases (even a "hello
> world")
> > in
> > >> pure json/yaml is better than always requiring a mix of json and XML.
> > >>
> > >> The second one is simplicity. Writing a optimizer (the module that
> > >> materializes the integration into running code, choosing a specific
> > >> runtime
> > >> and configuration) for a turing-complete language like Java is not
> easy:
> > >> even if you manage to create a good parser, it may be able to do
> > >> optimizations only if you write routes in a particular way, without
> > >> complex
> > >> constructs.
> > >> XML is ok from this point of view. The thing is that both json/yaml
> and
> > >> XML
> > >> are just two different ways to serialize object trees, that can be
> then
> > >> statically analyzed.
> > >> The point is not XML vs json/yaml, it's more about tailoring a new
> > >> minimalistic DSL to the emerging use cases vs proposing "only" our
> > classic
> > >> way of writing integrations. I think XML can be the "advanced" way. We
> > can
> > >> experiment optimizations easily with the new DSL, and enable them also
> > on
> > >> XML if it's worth.
> > >>
> > >> Scripting should be part of the spec, but I'd try to use programming
> > >> languages only for processing/transformation, not for the route
> > >> definition.
> > >>
> > >> Third one is performance. Apart from the fact that json parsers are in
> > >> general said to be faster than XML parsers... Given the "fast startup"
> > >> target that we want to reach, we may think e.g. to translate the new
> DSL
> > >> into Java or Go code, then compiling it. This allows doing parsing at
> > >> build
> > >> time in order to avoid it on startup. This kind of improvements are
> much
> > >> easier with a limited DSL but much more difficult with a existing
> > >> fully-fledged DSL..
> > >>
> > >> Nicola
> > >>
> > >>
> > >>
> > >> On Wed, Jul 18, 2018 at 11:51 PM Pontus Ullgren <ul...@gmail.com>
> > >> wrote:
> > >>
> > >> > I also like the idea but with some comments.
> > >> >
> > >> > As Hiram Chirino I'm not sure YAML/JSON is the best language for
> this.
> > >> > Perhaps a more fluent DSL like the Camel Java DSL or perhaps
> something
> > >> like
> > >> > Ballerina language would be better suited ?
> > >> > Also in my experience even simple integrations, that is simple real
> > >> world
> > >> > integration and not just hello world demos, requires
> > >> > you to add one or more Java class or scripting in addition to the
> core
> > >> > components. So for it to be useful there must be some way to add
> > custom
> > >> > code in some way for aggregation/enrichment strategies.
> > >> >
> > >> > If you go with a GO implementation I would also like some fallback
> for
> > >> > using Java since I find it unlikely that the existing base of
> > components
> > >> > will be ported any time soon.
> > >> >
> > >> > As for the name as other I like the "Kamel" name however a being a
> > >> newborn
> > >> > kotlin fan I know of one pre-existing project with that name
> > >> > that is close enough to allow for potential confusion.
> > >> > https://github.com/topicusoverheid/kamel
> > >> >
> > >> > Looking forward to see the result.
> > >> >
> > >> > // Pontus
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > On Mon, 16 Jul 2018 at 15:15 Hiram Chirino <hi...@hiramchirino.com>
> > >> wrote:
> > >> >
> > >> > > Love the idea.
> > >> > >
> > >> > > Personally, I'd keep using the existing Camel XML DSL if possible.
> > >> You
> > >> > can
> > >> > > still embed it in the CRD. The operator that deploys it could
> > >> inspect it
> > >> > > and figure whats the most optimized runtime that can support the
> > DSL.
> > >> > > Perhaps if it's only using the restricted set of camel components
> > >> > supported
> > >> > > by camel-go (https://github.com/lburgazzoli/camel-go) then is
> uses
> > >> that.
> > >> > > Otherwise it falls back to using camel spring boot.
> > >> > >
> > >> > > For that to work I think we need the different runtime
> > >> implementations to
> > >> > > provide a way to ask them: 'hey do you support running this camel
> > >> route?'
> > >> > > Not a trivial thing to respond to, it might require a build step
> in
> > >> there
> > >> > > for traditional Camel runtimes.
> > >> > >
> > >> > >
> > >> > > On Fri, Jul 13, 2018 at 4:32 AM Antonin Stefanutti <
> > >> > antonin@stefanutti.fr>
> > >> > > wrote:
> > >> > >
> > >> > > > Hi Nicola,
> > >> > > >
> > >> > > > I love the idea.
> > >> > > >
> > >> > > > I just wonder whether YAML/JSON is an expressive enough format
> in
> > >> the
> > >> > > long
> > >> > > > term. But as you’ve mentioned, starting simple would enable
> > >> > experimenting
> > >> > > > some very interesting / promising optimisations. So it seems
> worth
> > >> > taking
> > >> > > > that path, instead of trying to embed a complex DSL or the
> > existing
> > >> XML
> > >> > > DSL
> > >> > > > into the CRD.
> > >> > > >
> > >> > > > Definitely +1
> > >> > > >
> > >> > > > > On 13 Jul 2018, at 01:30, Nicola Ferraro <
> ni.ferraro@gmail.com>
> > >> > wrote:
> > >> > > > >
> > >> > > > > Hi Cameleers,
> > >> > > > > it's now passed some time since I started thinking about a new
> > >> > project
> > >> > > > that
> > >> > > > > we can begin here at Apache Camel, and I'd like to have your
> > >> opinion.
> > >> > > > >
> > >> > > > > We've already been targeting cloud-native applications with
> > Camel,
> > >> > > > > especially on top of Kubernetes, that is becoming "the
> standard"
> > >> > cloud
> > >> > > > > platform. But writing a Camel integration and running it on
> > >> > Kubernetes
> > >> > > > > requires some effort: choosing the base platform (spring-boot,
> > >> karaf,
> > >> > > > > simple main?), adding health checks (actuator?), packaging a
> > >> docker
> > >> > > image
> > >> > > > > and creating the Kubernetes resources (fabric8-maven-plugin,
> > >> helm?),
> > >> > > > > publishing the image on a docker registry, then finally
> > deploying
> > >> the
> > >> > > > > resources on a Kubernetes cluster.
> > >> > > > >
> > >> > > > > The resulting integration container is then far from being
> > optimal
> > >> > > from a
> > >> > > > > resource consumption point of view: it is likely that a Camel
> > >> > > Spring-Boot
> > >> > > > > application will require at least 200MB of RAM and also some
> CPU
> > >> > shares
> > >> > > > > because of polling threads used by many components.
> > >> > > > >
> > >> > > > > In case people use a CI/CD pipeline, it will take also a long
> > >> time to
> > >> > > get
> > >> > > > > from a code update to having a Kubernetes POD up and running.
> > >> > > > > Apart from compilation and image push/pull time, also startup
> > >> time is
> > >> > > > often
> > >> > > > > ~10 seconds for Camel + Spring-Boot in a container with
> standard
> > >> > limits
> > >> > > > on
> > >> > > > > resources, making it difficult to propose this combination for
> > >> > > > "serverless
> > >> > > > > integration" (this term is becoming increasingly more
> popular).
> > >> > > > >
> > >> > > > > So, my proposal is to start to investigate a "more
> cloud-native"
> > >> > > approach
> > >> > > > > to integration: *making Camel integrations first-class
> citizens
> > in
> > >> > > > > Kubernetes, and making them super fast and lightweight.*
> > >> > > > >
> > >> > > > > We can base the project on Kubernetes Custom Resource
> > Definitions
> > >> > (CRD)
> > >> > > > > <
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/
> > >> > > > >,
> > >> > > > > for example a Integration CRD and have a Kubernetes "operator"
> > >> > > > > <https://coreos.com/operators/> taking care of:
> > >> > > > > - Optimizing the integration that we want to run
> > >> > > > > - Packaging in a container
> > >> > > > > - Running it on Kubernetes
> > >> > > > > - Managing its entire lifecycle
> > >> > > > >
> > >> > > > > A Kubernetes-native integration may look like:
> > >> > > > >
> > >> > > > > -------------------
> > >> > > > > kind: "Integration"
> > >> > > > > apiVersion: "camel.apache.org/v1alpha1"
> > >> > > > > metadata:
> > >> > > > > name: "example"
> > >> > > > > spec:
> > >> > > > > replicas: 1
> > >> > > > > routes:
> > >> > > > > - id: timer
> > >> > > > > route:
> > >> > > > > - type: endpoint
> > >> > > > > uri: timer:tick
> > >> > > > > - type: endpoint
> > >> > > > > uri: log:info
> > >> > > > > -------------------
> > >> > > > >
> > >> > > > > For those who are not familiar with Kubernetes resources, this
> > >> kind
> > >> > of
> > >> > > > > YAML/JSON resource definitions are really common.
> > >> > > > > The example route is embedded in the Kubernetes resource
> > >> declaration
> > >> > > and
> > >> > > > > follows a basic "flow DSL". We may start from a basic one and
> > >> evolve
> > >> > it
> > >> > > > as
> > >> > > > > new requirements arrive from the community.
> > >> > > > >
> > >> > > > > I've made a very simple (but working) POC here:
> > >> > > > > https://github.com/nicolaferraro/integration-operator.
> > >> > > > >
> > >> > > > > This idea of a "Cloud-Native Camel" on Kubernetes (project
> > >> codename
> > >> > can
> > >> > > > be "
> > >> > > > > *Kamel*", if you like it :D), will be an enabler for a lot of
> > nice
> > >> > > > features.
> > >> > > > >
> > >> > > > > For example, we can propose "Kamel" as "ideal" platform for
> > >> > "serverless
> > >> > > > > integration" (I see many people reinventing the wheel out
> > there):
> > >> the
> > >> > > > > operator can reduce resource consumption of a single
> integration
> > >> by
> > >> > > > > optimizing the runtime and also pause/resume integrations when
> > >> they
> > >> > are
> > >> > > > not
> > >> > > > > used, that is the basic idea behind "serverless" (e.g. think
> to
> > >> > > > > HTTP-triggered integrations, but not only).
> > >> > > > > Focusing on serverless will bring more emphasis on push-based
> > >> > > > notifications
> > >> > > > > (webhooks, cloud events <https://cloudevents.io/>), that are
> > >> rarely
> > >> > > > used in
> > >> > > > > Camel components, that prefer a poll based approach being it
> > >> simpler
> > >> > to
> > >> > > > use
> > >> > > > > in classic deployments, but not so good in the cloud, where
> more
> > >> > > > resources
> > >> > > > > become higher direct costs for the users.
> > >> > > > >
> > >> > > > > The presence of the simplified DSL enables also experimenting
> on
> > >> > > > "*reduced*
> > >> > > > > subsets of Camel" implemented in languages other than Java,
> for
> > >> > example
> > >> > > > one
> > >> > > > > language that has a reactive approach on thread scheduling
> and a
> > >> > really
> > >> > > > low
> > >> > > > > memory footprint, like Go.
> > >> > > > >
> > >> > > > > But apart from this kind of experiments (that are valid IMO),
> > the
> > >> > > "Kamel"
> > >> > > > > optimizer will have free room to choose the right platform for
> > the
> > >> > > > > integration that the user wants to run, including, in the
> > future,
> > >> > doing
> > >> > > > AOT
> > >> > > > > compilation using Graal/VM (less memory, faster startup) if
> the
> > >> > > features
> > >> > > > > (components) used in the integration are supporting it (maybe
> we
> > >> can
> > >> > > add
> > >> > > > > AOT compilation in the roadmap for Camel 3).
> > >> > > > > A silly optimization: integrations starting from "timer:..."
> may
> > >> be
> > >> > > > > scheduled directly with Kubernetes CronJobs, so they will
> > consume
> > >> > > > resources
> > >> > > > > only when actually running.
> > >> > > > >
> > >> > > > > Being the final integrations lightweight and being the DSL
> > >> > > > > language-independent, we may see a increased adoption of Camel
> > >> also
> > >> > as
> > >> > > > > agile integration layer for not-only-java applications (both
> > >> "cloud"
> > >> > > and
> > >> > > > > "serverless" applications).
> > >> > > > >
> > >> > > > > I'm the first one that would like to work on a project ilke
> > this.
> > >> > I've
> > >> > > > > worked on many Kubernetes/Openshift based applications and
> > >> frameworks
> > >> > > in
> > >> > > > > the past years, also on operators and CRDs, and I think this
> way
> > >> of
> > >> > > > > redesigning integrations has a lot of potential.
> > >> > > > >
> > >> > > > > Integrations will not be necessarily limited to the simplified
> > >> DSL,
> > >> > but
> > >> > > > we
> > >> > > > > can add extension points for scripting and even custom
> libraries
> > >> > > > (although
> > >> > > > > limiting the freedom of the optimizer).
> > >> > > > >
> > >> > > > > The most important thing: it may become a great project, since
> > >> it's
> > >> > > > driven
> > >> > > > > by a great community.
> > >> > > > >
> > >> > > > > So, what do you think? Is it crazy enough?
> > >> > > > >
> > >> > > > > Nicola
> > >> > > >
> > >> > > >
> > >> > >
> > >> > > --
> > >> > > Hiram Chirino
> > >> > > Engineering | Red Hat, Inc.
> > >> > > hchirino@redhat.com | fusesource.com | redhat.com
> > >> > > skype: hiramchirino | twitter: @hiramchirino
> > >> > >
> > >> >
> > >>
> > >
> > >
> > > --
> > > ------------------------
> > > Guillaume Nodet
> > >
> > >
> >
> > --
> > ------------------------
> > Guillaume Nodet
> >
> --
> --
> Luca Burgazzoli
>
--
------------------------
Guillaume Nodet
Re: A "Kamel" crazy idea
Posted by Claus Ibsen <cl...@gmail.com>.
On Thu, Aug 2, 2018 at 12:17 AM, Luca Burgazzoli <lb...@gmail.com> wrote:
> That’s very nice start, thx Guillaume
>
Yes this is really awesome to see this prototype in action.
> On Wed, 1 Aug 2018 at 20:15, Guillaume Nodet <gn...@apache.org> wrote:
>
>> I've pushed a branch with graalvm experiments.
>> https://github.com/apache/camel/tree/graalvm
>> In order to build the project, you need to use the GraalVM jdk and use the
>> following ~/.m2/toolchains.xml adapted to your path:
>>
>> <?xml version="1.0" encoding="UTF8"?>
>>
>> <toolchains>
>>
>> <toolchain>
>>
>> <type>jdk</type>
>>
>> <provides>
>>
>> <version>1.0.0-rc4</version>
>>
>> <vendor>oracle</vendor>
>>
>> <id>graalvm-ee-1.0.0-rc4</id>
>>
>> </provides>
>>
>> <configuration>
>>
>>
>>
>> <jdkHome>/Users/gnodet/Downloads/graalvm-ee-1.0.0-rc4/Contents/Home/</jdkHome>
>>
>> </configuration>
>>
>> </toolchain>
>>
>> </toolchains>
>>
>> Build and run:
>> gnodet•camel*(*graalvm*)*» cd platforms/graalvm
>> gnodet•camel/platforms/graalvm*(*graalvm*)*» mvn install
>> ...
>> gnodet•camel/platforms/graalvm*(*graalvm*)*»
>> ./example/target/org.apache.camel.graalvm.main
>>
>> [main] INFO org.apache.camel.graalvm.FastCamelContext - Apache Camel
>> 2.23.0-SNAPSHOT (CamelContext: camel-1) is starting
>>
>> [main] INFO org.apache.camel.graalvm.FastCamelContext - StreamCaching is
>> not in use. If using streams then its recommended to enable stream caching.
>> See more details at http://camel.apache.org/stream-caching.html
>>
>> [main] INFO org.apache.camel.graalvm.FastCamelContext - Total 0 routes, of
>> which 0 are started
>>
>> [main] INFO org.apache.camel.graalvm.FastCamelContext - Apache Camel
>> 2.23.0-SNAPSHOT (CamelContext: camel-1) started in 0.006 seconds
>>
>> Guillaume
>>
>> Le mar. 31 juil. 2018 à 09:44, Guillaume Nodet <gn...@apache.org> a
>> écrit :
>>
>> >
>> >
>> > Le lun. 30 juil. 2018 à 17:49, Nicola Ferraro <ni...@gmail.com> a
>> > écrit :
>> >
>> >> Hi Cameleers,
>> >> it seems from the comments that this "Kamel" subproject is something we
>> >> want to start and I think that also the main camel core will benefit
>> from
>> >> the new features it will bring.
>> >>
>> >> I would like to donate the current "Kamel" code to Apache Camel, in
>> order
>> >> to have a initial brick to start from.
>> >>
>> >> From your reactions, the name "Kamel" seems cool and I'd like to keep
>> it.
>> >> However if you talk to people about the "Kamel" project, they won't
>> >> understand if you're talking about "Kamel with the K" or "Camel with the
>> >> C".
>> >> For this reasons I propose to keep "Kamel", but also use "Camel K"
>> [ˈkæməl
>> >> keɪ] as friendly name when talking about it. This is in line with the
>> >> repository we should create for the subproject, that needs to start with
>> >> "camel-" (it will be "apache/camel-k" in that case).
>> >>
>> >> These days I've been experimenting different ideas with Luca Burgazzoli.
>> >> In
>> >> particular, there have been some concerns here on the power of a
>> >> declarative DSL (a new one or the existing XML one) and we've found a
>> >> strategy that will allow us to use also the Java DSL in Kamel
>> >> integrations.
>> >> Without adding too many details here, at integration build time we can
>> run
>> >> the user code in a build container and inspect the produced routes to
>> get
>> >> metadata for the optimizer. This is one of the first things we should do
>> >> next.
>> >>
>> >> This possibility of running code that produces integrations opened a lot
>> >> of
>> >> unexpected paths, that I've written down in the project roadmap (
>> >> https://github.com/nicolaferraro/integration-operator) and summarized
>> >> below.
>> >>
>> >> One nice feature (kudos to Luca) is that we can simplify life for Kamel
>> >> users up to the point that they'll just need to write their routes on a
>> >> Java file and run them with e.g.:
>> >> "kamel run Routes.java"
>> >>
>> >> Where "kamel" is a binary we release within the project. Kubernetes
>> custom
>> >> resources will be used under the hood, but the "kamel" binary is a
>> utility
>> >> that will provide a user experience comparable to that of current
>> >> serverless platforms (or even better).
>> >>
>> >> But there's not just that.
>> >>
>> >> Luca wrote this weekend a prototype for having a polyglot Camel (
>> >> https://github.com/lburgazzoli/camel-routes-loader).
>> >>
>> >> With that, you will be able to e.g. write integrations in groovy and use
>> >> the same Kamel engine:
>> >> "kamel run routes.groovy"
>> >>
>> >> But also JavaScript:
>> >>
>> >>
>> https://github.com/lburgazzoli/camel-routes-loader/blob/ca986541f7c422ee02c21727cdfe4293d64a364e/src/test/resources/ext/camel/routes.js#L2
>> >>
>> >> This is a prototype right now, but a similar approach based on GraalVM
>> has
>> >> a lot of potential, because users can not only use their preferred
>> >> language
>> >> to write routes and processors, but also bind them to functionalities
>> >> available in their preferred libraries.
>> >>
>> >> How much this will be feasible depends on the adoption of GraalVM, but
>> I'm
>> >> seeing many frameworks adding metadata to make GraalVM work with
>> >> reflection. We've also done some work in Camel, there are some Jiras for
>> >> it
>> >> and first tests made by Guillaume signal that it's something feasible,
>> at
>> >> least for the Camel core and a subset of components. GraalVM would be
>> >> important also to reduce memory footprint and improve startup time, as
>> >> already said.
>> >>
>> >
>> > I'll continue experimenting and I'll report back.
>> > I did some initial experiments leveraging some modifications I did to
>> > speed up the start up time [1] and on a single route [2].
>> > There may be some limitations down the road of course, but at least it
>> > shows that it's feasible. Fwiw, the experiments lead to a startup time of
>> > 14 ms compared to a few hundreds milliseconds (roughly 680 with [1]).
>> >
>> >
>> >>
>> >> If the GraalVM approach works (it is working for other frameworks),
>> >> instead
>> >> of rewriting a subset of Camel in Go (as the original proposal
>> mentions),
>> >> we can just sanitize and recompile our existing codebase: this way we
>> >> fully
>> >> leverage the strength of Apache Camel.
>> >>
>> >
>> > Right, if we can avoid rewriting Camel, that would be much better !
>> >
>> > [1] https://issues.apache.org/jira/browse/CAMEL-12688
>> > [2]
>> >
>> https://github.com/gnodet/openwhisk-runtime-camel/blob/master/camel-openwhisk-example/src/main/java/org/jboss/fuse/openwhisk/camel/example/SimpleCamelFunction.java
>> >
>> >
>> >>
>> >> Another important thing in the roadmap is actually how we'll integrate
>> >> with
>> >> current FaaS platforms. We will evaluate together if it's better to
>> have a
>> >> tighter integration with some of them or to leverage knative for some
>> >> aspects of Kamel...
>> >>
>> >> But, let's get started!
>> >>
>> >> Il 19 lug 2018 10:45, "Nicola Ferraro" <ni...@gmail.com> ha
>> scritto:
>> >>
>> >> It's clear to me that we need to add support for our existing XML DSL,
>> >> that
>> >> is powerful. But there are multiple reasons why I'd like to also "add" a
>> >> limited yaml/json notation to "Kamel".
>> >>
>> >> The first one (and simplest) is that json/yaml is the primary encoding
>> for
>> >> all resources exchanged with the Kubernetes API server. I'm not saying
>> >> that
>> >> for this reason the Integration resource cannot contain a XML
>> >> (/Java/Ballerina) section, but from a pure presentational point of view,
>> >> having the possibility to write simple use cases (even a "hello world")
>> in
>> >> pure json/yaml is better than always requiring a mix of json and XML.
>> >>
>> >> The second one is simplicity. Writing a optimizer (the module that
>> >> materializes the integration into running code, choosing a specific
>> >> runtime
>> >> and configuration) for a turing-complete language like Java is not easy:
>> >> even if you manage to create a good parser, it may be able to do
>> >> optimizations only if you write routes in a particular way, without
>> >> complex
>> >> constructs.
>> >> XML is ok from this point of view. The thing is that both json/yaml and
>> >> XML
>> >> are just two different ways to serialize object trees, that can be then
>> >> statically analyzed.
>> >> The point is not XML vs json/yaml, it's more about tailoring a new
>> >> minimalistic DSL to the emerging use cases vs proposing "only" our
>> classic
>> >> way of writing integrations. I think XML can be the "advanced" way. We
>> can
>> >> experiment optimizations easily with the new DSL, and enable them also
>> on
>> >> XML if it's worth.
>> >>
>> >> Scripting should be part of the spec, but I'd try to use programming
>> >> languages only for processing/transformation, not for the route
>> >> definition.
>> >>
>> >> Third one is performance. Apart from the fact that json parsers are in
>> >> general said to be faster than XML parsers... Given the "fast startup"
>> >> target that we want to reach, we may think e.g. to translate the new DSL
>> >> into Java or Go code, then compiling it. This allows doing parsing at
>> >> build
>> >> time in order to avoid it on startup. This kind of improvements are much
>> >> easier with a limited DSL but much more difficult with a existing
>> >> fully-fledged DSL..
>> >>
>> >> Nicola
>> >>
>> >>
>> >>
>> >> On Wed, Jul 18, 2018 at 11:51 PM Pontus Ullgren <ul...@gmail.com>
>> >> wrote:
>> >>
>> >> > I also like the idea but with some comments.
>> >> >
>> >> > As Hiram Chirino I'm not sure YAML/JSON is the best language for this.
>> >> > Perhaps a more fluent DSL like the Camel Java DSL or perhaps something
>> >> like
>> >> > Ballerina language would be better suited ?
>> >> > Also in my experience even simple integrations, that is simple real
>> >> world
>> >> > integration and not just hello world demos, requires
>> >> > you to add one or more Java class or scripting in addition to the core
>> >> > components. So for it to be useful there must be some way to add
>> custom
>> >> > code in some way for aggregation/enrichment strategies.
>> >> >
>> >> > If you go with a GO implementation I would also like some fallback for
>> >> > using Java since I find it unlikely that the existing base of
>> components
>> >> > will be ported any time soon.
>> >> >
>> >> > As for the name as other I like the "Kamel" name however a being a
>> >> newborn
>> >> > kotlin fan I know of one pre-existing project with that name
>> >> > that is close enough to allow for potential confusion.
>> >> > https://github.com/topicusoverheid/kamel
>> >> >
>> >> > Looking forward to see the result.
>> >> >
>> >> > // Pontus
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > On Mon, 16 Jul 2018 at 15:15 Hiram Chirino <hi...@hiramchirino.com>
>> >> wrote:
>> >> >
>> >> > > Love the idea.
>> >> > >
>> >> > > Personally, I'd keep using the existing Camel XML DSL if possible.
>> >> You
>> >> > can
>> >> > > still embed it in the CRD. The operator that deploys it could
>> >> inspect it
>> >> > > and figure whats the most optimized runtime that can support the
>> DSL.
>> >> > > Perhaps if it's only using the restricted set of camel components
>> >> > supported
>> >> > > by camel-go (https://github.com/lburgazzoli/camel-go) then is uses
>> >> that.
>> >> > > Otherwise it falls back to using camel spring boot.
>> >> > >
>> >> > > For that to work I think we need the different runtime
>> >> implementations to
>> >> > > provide a way to ask them: 'hey do you support running this camel
>> >> route?'
>> >> > > Not a trivial thing to respond to, it might require a build step in
>> >> there
>> >> > > for traditional Camel runtimes.
>> >> > >
>> >> > >
>> >> > > On Fri, Jul 13, 2018 at 4:32 AM Antonin Stefanutti <
>> >> > antonin@stefanutti.fr>
>> >> > > wrote:
>> >> > >
>> >> > > > Hi Nicola,
>> >> > > >
>> >> > > > I love the idea.
>> >> > > >
>> >> > > > I just wonder whether YAML/JSON is an expressive enough format in
>> >> the
>> >> > > long
>> >> > > > term. But as you’ve mentioned, starting simple would enable
>> >> > experimenting
>> >> > > > some very interesting / promising optimisations. So it seems worth
>> >> > taking
>> >> > > > that path, instead of trying to embed a complex DSL or the
>> existing
>> >> XML
>> >> > > DSL
>> >> > > > into the CRD.
>> >> > > >
>> >> > > > Definitely +1
>> >> > > >
>> >> > > > > On 13 Jul 2018, at 01:30, Nicola Ferraro <ni...@gmail.com>
>> >> > wrote:
>> >> > > > >
>> >> > > > > Hi Cameleers,
>> >> > > > > it's now passed some time since I started thinking about a new
>> >> > project
>> >> > > > that
>> >> > > > > we can begin here at Apache Camel, and I'd like to have your
>> >> opinion.
>> >> > > > >
>> >> > > > > We've already been targeting cloud-native applications with
>> Camel,
>> >> > > > > especially on top of Kubernetes, that is becoming "the standard"
>> >> > cloud
>> >> > > > > platform. But writing a Camel integration and running it on
>> >> > Kubernetes
>> >> > > > > requires some effort: choosing the base platform (spring-boot,
>> >> karaf,
>> >> > > > > simple main?), adding health checks (actuator?), packaging a
>> >> docker
>> >> > > image
>> >> > > > > and creating the Kubernetes resources (fabric8-maven-plugin,
>> >> helm?),
>> >> > > > > publishing the image on a docker registry, then finally
>> deploying
>> >> the
>> >> > > > > resources on a Kubernetes cluster.
>> >> > > > >
>> >> > > > > The resulting integration container is then far from being
>> optimal
>> >> > > from a
>> >> > > > > resource consumption point of view: it is likely that a Camel
>> >> > > Spring-Boot
>> >> > > > > application will require at least 200MB of RAM and also some CPU
>> >> > shares
>> >> > > > > because of polling threads used by many components.
>> >> > > > >
>> >> > > > > In case people use a CI/CD pipeline, it will take also a long
>> >> time to
>> >> > > get
>> >> > > > > from a code update to having a Kubernetes POD up and running.
>> >> > > > > Apart from compilation and image push/pull time, also startup
>> >> time is
>> >> > > > often
>> >> > > > > ~10 seconds for Camel + Spring-Boot in a container with standard
>> >> > limits
>> >> > > > on
>> >> > > > > resources, making it difficult to propose this combination for
>> >> > > > "serverless
>> >> > > > > integration" (this term is becoming increasingly more popular).
>> >> > > > >
>> >> > > > > So, my proposal is to start to investigate a "more cloud-native"
>> >> > > approach
>> >> > > > > to integration: *making Camel integrations first-class citizens
>> in
>> >> > > > > Kubernetes, and making them super fast and lightweight.*
>> >> > > > >
>> >> > > > > We can base the project on Kubernetes Custom Resource
>> Definitions
>> >> > (CRD)
>> >> > > > > <
>> >> > > >
>> >> > >
>> >> >
>> >>
>> https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/
>> >> > > > >,
>> >> > > > > for example a Integration CRD and have a Kubernetes "operator"
>> >> > > > > <https://coreos.com/operators/> taking care of:
>> >> > > > > - Optimizing the integration that we want to run
>> >> > > > > - Packaging in a container
>> >> > > > > - Running it on Kubernetes
>> >> > > > > - Managing its entire lifecycle
>> >> > > > >
>> >> > > > > A Kubernetes-native integration may look like:
>> >> > > > >
>> >> > > > > -------------------
>> >> > > > > kind: "Integration"
>> >> > > > > apiVersion: "camel.apache.org/v1alpha1"
>> >> > > > > metadata:
>> >> > > > > name: "example"
>> >> > > > > spec:
>> >> > > > > replicas: 1
>> >> > > > > routes:
>> >> > > > > - id: timer
>> >> > > > > route:
>> >> > > > > - type: endpoint
>> >> > > > > uri: timer:tick
>> >> > > > > - type: endpoint
>> >> > > > > uri: log:info
>> >> > > > > -------------------
>> >> > > > >
>> >> > > > > For those who are not familiar with Kubernetes resources, this
>> >> kind
>> >> > of
>> >> > > > > YAML/JSON resource definitions are really common.
>> >> > > > > The example route is embedded in the Kubernetes resource
>> >> declaration
>> >> > > and
>> >> > > > > follows a basic "flow DSL". We may start from a basic one and
>> >> evolve
>> >> > it
>> >> > > > as
>> >> > > > > new requirements arrive from the community.
>> >> > > > >
>> >> > > > > I've made a very simple (but working) POC here:
>> >> > > > > https://github.com/nicolaferraro/integration-operator.
>> >> > > > >
>> >> > > > > This idea of a "Cloud-Native Camel" on Kubernetes (project
>> >> codename
>> >> > can
>> >> > > > be "
>> >> > > > > *Kamel*", if you like it :D), will be an enabler for a lot of
>> nice
>> >> > > > features.
>> >> > > > >
>> >> > > > > For example, we can propose "Kamel" as "ideal" platform for
>> >> > "serverless
>> >> > > > > integration" (I see many people reinventing the wheel out
>> there):
>> >> the
>> >> > > > > operator can reduce resource consumption of a single integration
>> >> by
>> >> > > > > optimizing the runtime and also pause/resume integrations when
>> >> they
>> >> > are
>> >> > > > not
>> >> > > > > used, that is the basic idea behind "serverless" (e.g. think to
>> >> > > > > HTTP-triggered integrations, but not only).
>> >> > > > > Focusing on serverless will bring more emphasis on push-based
>> >> > > > notifications
>> >> > > > > (webhooks, cloud events <https://cloudevents.io/>), that are
>> >> rarely
>> >> > > > used in
>> >> > > > > Camel components, that prefer a poll based approach being it
>> >> simpler
>> >> > to
>> >> > > > use
>> >> > > > > in classic deployments, but not so good in the cloud, where more
>> >> > > > resources
>> >> > > > > become higher direct costs for the users.
>> >> > > > >
>> >> > > > > The presence of the simplified DSL enables also experimenting on
>> >> > > > "*reduced*
>> >> > > > > subsets of Camel" implemented in languages other than Java, for
>> >> > example
>> >> > > > one
>> >> > > > > language that has a reactive approach on thread scheduling and a
>> >> > really
>> >> > > > low
>> >> > > > > memory footprint, like Go.
>> >> > > > >
>> >> > > > > But apart from this kind of experiments (that are valid IMO),
>> the
>> >> > > "Kamel"
>> >> > > > > optimizer will have free room to choose the right platform for
>> the
>> >> > > > > integration that the user wants to run, including, in the
>> future,
>> >> > doing
>> >> > > > AOT
>> >> > > > > compilation using Graal/VM (less memory, faster startup) if the
>> >> > > features
>> >> > > > > (components) used in the integration are supporting it (maybe we
>> >> can
>> >> > > add
>> >> > > > > AOT compilation in the roadmap for Camel 3).
>> >> > > > > A silly optimization: integrations starting from "timer:..." may
>> >> be
>> >> > > > > scheduled directly with Kubernetes CronJobs, so they will
>> consume
>> >> > > > resources
>> >> > > > > only when actually running.
>> >> > > > >
>> >> > > > > Being the final integrations lightweight and being the DSL
>> >> > > > > language-independent, we may see a increased adoption of Camel
>> >> also
>> >> > as
>> >> > > > > agile integration layer for not-only-java applications (both
>> >> "cloud"
>> >> > > and
>> >> > > > > "serverless" applications).
>> >> > > > >
>> >> > > > > I'm the first one that would like to work on a project ilke
>> this.
>> >> > I've
>> >> > > > > worked on many Kubernetes/Openshift based applications and
>> >> frameworks
>> >> > > in
>> >> > > > > the past years, also on operators and CRDs, and I think this way
>> >> of
>> >> > > > > redesigning integrations has a lot of potential.
>> >> > > > >
>> >> > > > > Integrations will not be necessarily limited to the simplified
>> >> DSL,
>> >> > but
>> >> > > > we
>> >> > > > > can add extension points for scripting and even custom libraries
>> >> > > > (although
>> >> > > > > limiting the freedom of the optimizer).
>> >> > > > >
>> >> > > > > The most important thing: it may become a great project, since
>> >> it's
>> >> > > > driven
>> >> > > > > by a great community.
>> >> > > > >
>> >> > > > > So, what do you think? Is it crazy enough?
>> >> > > > >
>> >> > > > > Nicola
>> >> > > >
>> >> > > >
>> >> > >
>> >> > > --
>> >> > > Hiram Chirino
>> >> > > Engineering | Red Hat, Inc.
>> >> > > hchirino@redhat.com | fusesource.com | redhat.com
>> >> > > skype: hiramchirino | twitter: @hiramchirino
>> >> > >
>> >> >
>> >>
>> >
>> >
>> > --
>> > ------------------------
>> > Guillaume Nodet
>> >
>> >
>>
>> --
>> ------------------------
>> Guillaume Nodet
>>
> --
> --
> Luca Burgazzoli
--
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2
Re: A "Kamel" crazy idea
Posted by Luca Burgazzoli <lb...@gmail.com>.
That’s very nice start, thx Guillaume
On Wed, 1 Aug 2018 at 20:15, Guillaume Nodet <gn...@apache.org> wrote:
> I've pushed a branch with graalvm experiments.
> https://github.com/apache/camel/tree/graalvm
> In order to build the project, you need to use the GraalVM jdk and use the
> following ~/.m2/toolchains.xml adapted to your path:
>
> <?xml version="1.0" encoding="UTF8"?>
>
> <toolchains>
>
> <toolchain>
>
> <type>jdk</type>
>
> <provides>
>
> <version>1.0.0-rc4</version>
>
> <vendor>oracle</vendor>
>
> <id>graalvm-ee-1.0.0-rc4</id>
>
> </provides>
>
> <configuration>
>
>
>
> <jdkHome>/Users/gnodet/Downloads/graalvm-ee-1.0.0-rc4/Contents/Home/</jdkHome>
>
> </configuration>
>
> </toolchain>
>
> </toolchains>
>
> Build and run:
> gnodet•camel*(*graalvm*)*» cd platforms/graalvm
> gnodet•camel/platforms/graalvm*(*graalvm*)*» mvn install
> ...
> gnodet•camel/platforms/graalvm*(*graalvm*)*»
> ./example/target/org.apache.camel.graalvm.main
>
> [main] INFO org.apache.camel.graalvm.FastCamelContext - Apache Camel
> 2.23.0-SNAPSHOT (CamelContext: camel-1) is starting
>
> [main] INFO org.apache.camel.graalvm.FastCamelContext - StreamCaching is
> not in use. If using streams then its recommended to enable stream caching.
> See more details at http://camel.apache.org/stream-caching.html
>
> [main] INFO org.apache.camel.graalvm.FastCamelContext - Total 0 routes, of
> which 0 are started
>
> [main] INFO org.apache.camel.graalvm.FastCamelContext - Apache Camel
> 2.23.0-SNAPSHOT (CamelContext: camel-1) started in 0.006 seconds
>
> Guillaume
>
> Le mar. 31 juil. 2018 à 09:44, Guillaume Nodet <gn...@apache.org> a
> écrit :
>
> >
> >
> > Le lun. 30 juil. 2018 à 17:49, Nicola Ferraro <ni...@gmail.com> a
> > écrit :
> >
> >> Hi Cameleers,
> >> it seems from the comments that this "Kamel" subproject is something we
> >> want to start and I think that also the main camel core will benefit
> from
> >> the new features it will bring.
> >>
> >> I would like to donate the current "Kamel" code to Apache Camel, in
> order
> >> to have a initial brick to start from.
> >>
> >> From your reactions, the name "Kamel" seems cool and I'd like to keep
> it.
> >> However if you talk to people about the "Kamel" project, they won't
> >> understand if you're talking about "Kamel with the K" or "Camel with the
> >> C".
> >> For this reasons I propose to keep "Kamel", but also use "Camel K"
> [ˈkæməl
> >> keɪ] as friendly name when talking about it. This is in line with the
> >> repository we should create for the subproject, that needs to start with
> >> "camel-" (it will be "apache/camel-k" in that case).
> >>
> >> These days I've been experimenting different ideas with Luca Burgazzoli.
> >> In
> >> particular, there have been some concerns here on the power of a
> >> declarative DSL (a new one or the existing XML one) and we've found a
> >> strategy that will allow us to use also the Java DSL in Kamel
> >> integrations.
> >> Without adding too many details here, at integration build time we can
> run
> >> the user code in a build container and inspect the produced routes to
> get
> >> metadata for the optimizer. This is one of the first things we should do
> >> next.
> >>
> >> This possibility of running code that produces integrations opened a lot
> >> of
> >> unexpected paths, that I've written down in the project roadmap (
> >> https://github.com/nicolaferraro/integration-operator) and summarized
> >> below.
> >>
> >> One nice feature (kudos to Luca) is that we can simplify life for Kamel
> >> users up to the point that they'll just need to write their routes on a
> >> Java file and run them with e.g.:
> >> "kamel run Routes.java"
> >>
> >> Where "kamel" is a binary we release within the project. Kubernetes
> custom
> >> resources will be used under the hood, but the "kamel" binary is a
> utility
> >> that will provide a user experience comparable to that of current
> >> serverless platforms (or even better).
> >>
> >> But there's not just that.
> >>
> >> Luca wrote this weekend a prototype for having a polyglot Camel (
> >> https://github.com/lburgazzoli/camel-routes-loader).
> >>
> >> With that, you will be able to e.g. write integrations in groovy and use
> >> the same Kamel engine:
> >> "kamel run routes.groovy"
> >>
> >> But also JavaScript:
> >>
> >>
> https://github.com/lburgazzoli/camel-routes-loader/blob/ca986541f7c422ee02c21727cdfe4293d64a364e/src/test/resources/ext/camel/routes.js#L2
> >>
> >> This is a prototype right now, but a similar approach based on GraalVM
> has
> >> a lot of potential, because users can not only use their preferred
> >> language
> >> to write routes and processors, but also bind them to functionalities
> >> available in their preferred libraries.
> >>
> >> How much this will be feasible depends on the adoption of GraalVM, but
> I'm
> >> seeing many frameworks adding metadata to make GraalVM work with
> >> reflection. We've also done some work in Camel, there are some Jiras for
> >> it
> >> and first tests made by Guillaume signal that it's something feasible,
> at
> >> least for the Camel core and a subset of components. GraalVM would be
> >> important also to reduce memory footprint and improve startup time, as
> >> already said.
> >>
> >
> > I'll continue experimenting and I'll report back.
> > I did some initial experiments leveraging some modifications I did to
> > speed up the start up time [1] and on a single route [2].
> > There may be some limitations down the road of course, but at least it
> > shows that it's feasible. Fwiw, the experiments lead to a startup time of
> > 14 ms compared to a few hundreds milliseconds (roughly 680 with [1]).
> >
> >
> >>
> >> If the GraalVM approach works (it is working for other frameworks),
> >> instead
> >> of rewriting a subset of Camel in Go (as the original proposal
> mentions),
> >> we can just sanitize and recompile our existing codebase: this way we
> >> fully
> >> leverage the strength of Apache Camel.
> >>
> >
> > Right, if we can avoid rewriting Camel, that would be much better !
> >
> > [1] https://issues.apache.org/jira/browse/CAMEL-12688
> > [2]
> >
> https://github.com/gnodet/openwhisk-runtime-camel/blob/master/camel-openwhisk-example/src/main/java/org/jboss/fuse/openwhisk/camel/example/SimpleCamelFunction.java
> >
> >
> >>
> >> Another important thing in the roadmap is actually how we'll integrate
> >> with
> >> current FaaS platforms. We will evaluate together if it's better to
> have a
> >> tighter integration with some of them or to leverage knative for some
> >> aspects of Kamel...
> >>
> >> But, let's get started!
> >>
> >> Il 19 lug 2018 10:45, "Nicola Ferraro" <ni...@gmail.com> ha
> scritto:
> >>
> >> It's clear to me that we need to add support for our existing XML DSL,
> >> that
> >> is powerful. But there are multiple reasons why I'd like to also "add" a
> >> limited yaml/json notation to "Kamel".
> >>
> >> The first one (and simplest) is that json/yaml is the primary encoding
> for
> >> all resources exchanged with the Kubernetes API server. I'm not saying
> >> that
> >> for this reason the Integration resource cannot contain a XML
> >> (/Java/Ballerina) section, but from a pure presentational point of view,
> >> having the possibility to write simple use cases (even a "hello world")
> in
> >> pure json/yaml is better than always requiring a mix of json and XML.
> >>
> >> The second one is simplicity. Writing a optimizer (the module that
> >> materializes the integration into running code, choosing a specific
> >> runtime
> >> and configuration) for a turing-complete language like Java is not easy:
> >> even if you manage to create a good parser, it may be able to do
> >> optimizations only if you write routes in a particular way, without
> >> complex
> >> constructs.
> >> XML is ok from this point of view. The thing is that both json/yaml and
> >> XML
> >> are just two different ways to serialize object trees, that can be then
> >> statically analyzed.
> >> The point is not XML vs json/yaml, it's more about tailoring a new
> >> minimalistic DSL to the emerging use cases vs proposing "only" our
> classic
> >> way of writing integrations. I think XML can be the "advanced" way. We
> can
> >> experiment optimizations easily with the new DSL, and enable them also
> on
> >> XML if it's worth.
> >>
> >> Scripting should be part of the spec, but I'd try to use programming
> >> languages only for processing/transformation, not for the route
> >> definition.
> >>
> >> Third one is performance. Apart from the fact that json parsers are in
> >> general said to be faster than XML parsers... Given the "fast startup"
> >> target that we want to reach, we may think e.g. to translate the new DSL
> >> into Java or Go code, then compiling it. This allows doing parsing at
> >> build
> >> time in order to avoid it on startup. This kind of improvements are much
> >> easier with a limited DSL but much more difficult with a existing
> >> fully-fledged DSL..
> >>
> >> Nicola
> >>
> >>
> >>
> >> On Wed, Jul 18, 2018 at 11:51 PM Pontus Ullgren <ul...@gmail.com>
> >> wrote:
> >>
> >> > I also like the idea but with some comments.
> >> >
> >> > As Hiram Chirino I'm not sure YAML/JSON is the best language for this.
> >> > Perhaps a more fluent DSL like the Camel Java DSL or perhaps something
> >> like
> >> > Ballerina language would be better suited ?
> >> > Also in my experience even simple integrations, that is simple real
> >> world
> >> > integration and not just hello world demos, requires
> >> > you to add one or more Java class or scripting in addition to the core
> >> > components. So for it to be useful there must be some way to add
> custom
> >> > code in some way for aggregation/enrichment strategies.
> >> >
> >> > If you go with a GO implementation I would also like some fallback for
> >> > using Java since I find it unlikely that the existing base of
> components
> >> > will be ported any time soon.
> >> >
> >> > As for the name as other I like the "Kamel" name however a being a
> >> newborn
> >> > kotlin fan I know of one pre-existing project with that name
> >> > that is close enough to allow for potential confusion.
> >> > https://github.com/topicusoverheid/kamel
> >> >
> >> > Looking forward to see the result.
> >> >
> >> > // Pontus
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > On Mon, 16 Jul 2018 at 15:15 Hiram Chirino <hi...@hiramchirino.com>
> >> wrote:
> >> >
> >> > > Love the idea.
> >> > >
> >> > > Personally, I'd keep using the existing Camel XML DSL if possible.
> >> You
> >> > can
> >> > > still embed it in the CRD. The operator that deploys it could
> >> inspect it
> >> > > and figure whats the most optimized runtime that can support the
> DSL.
> >> > > Perhaps if it's only using the restricted set of camel components
> >> > supported
> >> > > by camel-go (https://github.com/lburgazzoli/camel-go) then is uses
> >> that.
> >> > > Otherwise it falls back to using camel spring boot.
> >> > >
> >> > > For that to work I think we need the different runtime
> >> implementations to
> >> > > provide a way to ask them: 'hey do you support running this camel
> >> route?'
> >> > > Not a trivial thing to respond to, it might require a build step in
> >> there
> >> > > for traditional Camel runtimes.
> >> > >
> >> > >
> >> > > On Fri, Jul 13, 2018 at 4:32 AM Antonin Stefanutti <
> >> > antonin@stefanutti.fr>
> >> > > wrote:
> >> > >
> >> > > > Hi Nicola,
> >> > > >
> >> > > > I love the idea.
> >> > > >
> >> > > > I just wonder whether YAML/JSON is an expressive enough format in
> >> the
> >> > > long
> >> > > > term. But as you’ve mentioned, starting simple would enable
> >> > experimenting
> >> > > > some very interesting / promising optimisations. So it seems worth
> >> > taking
> >> > > > that path, instead of trying to embed a complex DSL or the
> existing
> >> XML
> >> > > DSL
> >> > > > into the CRD.
> >> > > >
> >> > > > Definitely +1
> >> > > >
> >> > > > > On 13 Jul 2018, at 01:30, Nicola Ferraro <ni...@gmail.com>
> >> > wrote:
> >> > > > >
> >> > > > > Hi Cameleers,
> >> > > > > it's now passed some time since I started thinking about a new
> >> > project
> >> > > > that
> >> > > > > we can begin here at Apache Camel, and I'd like to have your
> >> opinion.
> >> > > > >
> >> > > > > We've already been targeting cloud-native applications with
> Camel,
> >> > > > > especially on top of Kubernetes, that is becoming "the standard"
> >> > cloud
> >> > > > > platform. But writing a Camel integration and running it on
> >> > Kubernetes
> >> > > > > requires some effort: choosing the base platform (spring-boot,
> >> karaf,
> >> > > > > simple main?), adding health checks (actuator?), packaging a
> >> docker
> >> > > image
> >> > > > > and creating the Kubernetes resources (fabric8-maven-plugin,
> >> helm?),
> >> > > > > publishing the image on a docker registry, then finally
> deploying
> >> the
> >> > > > > resources on a Kubernetes cluster.
> >> > > > >
> >> > > > > The resulting integration container is then far from being
> optimal
> >> > > from a
> >> > > > > resource consumption point of view: it is likely that a Camel
> >> > > Spring-Boot
> >> > > > > application will require at least 200MB of RAM and also some CPU
> >> > shares
> >> > > > > because of polling threads used by many components.
> >> > > > >
> >> > > > > In case people use a CI/CD pipeline, it will take also a long
> >> time to
> >> > > get
> >> > > > > from a code update to having a Kubernetes POD up and running.
> >> > > > > Apart from compilation and image push/pull time, also startup
> >> time is
> >> > > > often
> >> > > > > ~10 seconds for Camel + Spring-Boot in a container with standard
> >> > limits
> >> > > > on
> >> > > > > resources, making it difficult to propose this combination for
> >> > > > "serverless
> >> > > > > integration" (this term is becoming increasingly more popular).
> >> > > > >
> >> > > > > So, my proposal is to start to investigate a "more cloud-native"
> >> > > approach
> >> > > > > to integration: *making Camel integrations first-class citizens
> in
> >> > > > > Kubernetes, and making them super fast and lightweight.*
> >> > > > >
> >> > > > > We can base the project on Kubernetes Custom Resource
> Definitions
> >> > (CRD)
> >> > > > > <
> >> > > >
> >> > >
> >> >
> >>
> https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/
> >> > > > >,
> >> > > > > for example a Integration CRD and have a Kubernetes "operator"
> >> > > > > <https://coreos.com/operators/> taking care of:
> >> > > > > - Optimizing the integration that we want to run
> >> > > > > - Packaging in a container
> >> > > > > - Running it on Kubernetes
> >> > > > > - Managing its entire lifecycle
> >> > > > >
> >> > > > > A Kubernetes-native integration may look like:
> >> > > > >
> >> > > > > -------------------
> >> > > > > kind: "Integration"
> >> > > > > apiVersion: "camel.apache.org/v1alpha1"
> >> > > > > metadata:
> >> > > > > name: "example"
> >> > > > > spec:
> >> > > > > replicas: 1
> >> > > > > routes:
> >> > > > > - id: timer
> >> > > > > route:
> >> > > > > - type: endpoint
> >> > > > > uri: timer:tick
> >> > > > > - type: endpoint
> >> > > > > uri: log:info
> >> > > > > -------------------
> >> > > > >
> >> > > > > For those who are not familiar with Kubernetes resources, this
> >> kind
> >> > of
> >> > > > > YAML/JSON resource definitions are really common.
> >> > > > > The example route is embedded in the Kubernetes resource
> >> declaration
> >> > > and
> >> > > > > follows a basic "flow DSL". We may start from a basic one and
> >> evolve
> >> > it
> >> > > > as
> >> > > > > new requirements arrive from the community.
> >> > > > >
> >> > > > > I've made a very simple (but working) POC here:
> >> > > > > https://github.com/nicolaferraro/integration-operator.
> >> > > > >
> >> > > > > This idea of a "Cloud-Native Camel" on Kubernetes (project
> >> codename
> >> > can
> >> > > > be "
> >> > > > > *Kamel*", if you like it :D), will be an enabler for a lot of
> nice
> >> > > > features.
> >> > > > >
> >> > > > > For example, we can propose "Kamel" as "ideal" platform for
> >> > "serverless
> >> > > > > integration" (I see many people reinventing the wheel out
> there):
> >> the
> >> > > > > operator can reduce resource consumption of a single integration
> >> by
> >> > > > > optimizing the runtime and also pause/resume integrations when
> >> they
> >> > are
> >> > > > not
> >> > > > > used, that is the basic idea behind "serverless" (e.g. think to
> >> > > > > HTTP-triggered integrations, but not only).
> >> > > > > Focusing on serverless will bring more emphasis on push-based
> >> > > > notifications
> >> > > > > (webhooks, cloud events <https://cloudevents.io/>), that are
> >> rarely
> >> > > > used in
> >> > > > > Camel components, that prefer a poll based approach being it
> >> simpler
> >> > to
> >> > > > use
> >> > > > > in classic deployments, but not so good in the cloud, where more
> >> > > > resources
> >> > > > > become higher direct costs for the users.
> >> > > > >
> >> > > > > The presence of the simplified DSL enables also experimenting on
> >> > > > "*reduced*
> >> > > > > subsets of Camel" implemented in languages other than Java, for
> >> > example
> >> > > > one
> >> > > > > language that has a reactive approach on thread scheduling and a
> >> > really
> >> > > > low
> >> > > > > memory footprint, like Go.
> >> > > > >
> >> > > > > But apart from this kind of experiments (that are valid IMO),
> the
> >> > > "Kamel"
> >> > > > > optimizer will have free room to choose the right platform for
> the
> >> > > > > integration that the user wants to run, including, in the
> future,
> >> > doing
> >> > > > AOT
> >> > > > > compilation using Graal/VM (less memory, faster startup) if the
> >> > > features
> >> > > > > (components) used in the integration are supporting it (maybe we
> >> can
> >> > > add
> >> > > > > AOT compilation in the roadmap for Camel 3).
> >> > > > > A silly optimization: integrations starting from "timer:..." may
> >> be
> >> > > > > scheduled directly with Kubernetes CronJobs, so they will
> consume
> >> > > > resources
> >> > > > > only when actually running.
> >> > > > >
> >> > > > > Being the final integrations lightweight and being the DSL
> >> > > > > language-independent, we may see a increased adoption of Camel
> >> also
> >> > as
> >> > > > > agile integration layer for not-only-java applications (both
> >> "cloud"
> >> > > and
> >> > > > > "serverless" applications).
> >> > > > >
> >> > > > > I'm the first one that would like to work on a project ilke
> this.
> >> > I've
> >> > > > > worked on many Kubernetes/Openshift based applications and
> >> frameworks
> >> > > in
> >> > > > > the past years, also on operators and CRDs, and I think this way
> >> of
> >> > > > > redesigning integrations has a lot of potential.
> >> > > > >
> >> > > > > Integrations will not be necessarily limited to the simplified
> >> DSL,
> >> > but
> >> > > > we
> >> > > > > can add extension points for scripting and even custom libraries
> >> > > > (although
> >> > > > > limiting the freedom of the optimizer).
> >> > > > >
> >> > > > > The most important thing: it may become a great project, since
> >> it's
> >> > > > driven
> >> > > > > by a great community.
> >> > > > >
> >> > > > > So, what do you think? Is it crazy enough?
> >> > > > >
> >> > > > > Nicola
> >> > > >
> >> > > >
> >> > >
> >> > > --
> >> > > Hiram Chirino
> >> > > Engineering | Red Hat, Inc.
> >> > > hchirino@redhat.com | fusesource.com | redhat.com
> >> > > skype: hiramchirino | twitter: @hiramchirino
> >> > >
> >> >
> >>
> >
> >
> > --
> > ------------------------
> > Guillaume Nodet
> >
> >
>
> --
> ------------------------
> Guillaume Nodet
>
--
--
Luca Burgazzoli
Re: A "Kamel" crazy idea
Posted by Guillaume Nodet <gn...@apache.org>.
I've pushed a branch with graalvm experiments.
https://github.com/apache/camel/tree/graalvm
In order to build the project, you need to use the GraalVM jdk and use the
following ~/.m2/toolchains.xml adapted to your path:
<?xml version="1.0" encoding="UTF8"?>
<toolchains>
<toolchain>
<type>jdk</type>
<provides>
<version>1.0.0-rc4</version>
<vendor>oracle</vendor>
<id>graalvm-ee-1.0.0-rc4</id>
</provides>
<configuration>
<jdkHome>/Users/gnodet/Downloads/graalvm-ee-1.0.0-rc4/Contents/Home/</jdkHome>
</configuration>
</toolchain>
</toolchains>
Build and run:
gnodet•camel*(*graalvm*)*» cd platforms/graalvm
gnodet•camel/platforms/graalvm*(*graalvm*)*» mvn install
...
gnodet•camel/platforms/graalvm*(*graalvm*)*»
./example/target/org.apache.camel.graalvm.main
[main] INFO org.apache.camel.graalvm.FastCamelContext - Apache Camel
2.23.0-SNAPSHOT (CamelContext: camel-1) is starting
[main] INFO org.apache.camel.graalvm.FastCamelContext - StreamCaching is
not in use. If using streams then its recommended to enable stream caching.
See more details at http://camel.apache.org/stream-caching.html
[main] INFO org.apache.camel.graalvm.FastCamelContext - Total 0 routes, of
which 0 are started
[main] INFO org.apache.camel.graalvm.FastCamelContext - Apache Camel
2.23.0-SNAPSHOT (CamelContext: camel-1) started in 0.006 seconds
Guillaume
Le mar. 31 juil. 2018 à 09:44, Guillaume Nodet <gn...@apache.org> a écrit :
>
>
> Le lun. 30 juil. 2018 à 17:49, Nicola Ferraro <ni...@gmail.com> a
> écrit :
>
>> Hi Cameleers,
>> it seems from the comments that this "Kamel" subproject is something we
>> want to start and I think that also the main camel core will benefit from
>> the new features it will bring.
>>
>> I would like to donate the current "Kamel" code to Apache Camel, in order
>> to have a initial brick to start from.
>>
>> From your reactions, the name "Kamel" seems cool and I'd like to keep it.
>> However if you talk to people about the "Kamel" project, they won't
>> understand if you're talking about "Kamel with the K" or "Camel with the
>> C".
>> For this reasons I propose to keep "Kamel", but also use "Camel K" [ˈkæməl
>> keɪ] as friendly name when talking about it. This is in line with the
>> repository we should create for the subproject, that needs to start with
>> "camel-" (it will be "apache/camel-k" in that case).
>>
>> These days I've been experimenting different ideas with Luca Burgazzoli.
>> In
>> particular, there have been some concerns here on the power of a
>> declarative DSL (a new one or the existing XML one) and we've found a
>> strategy that will allow us to use also the Java DSL in Kamel
>> integrations.
>> Without adding too many details here, at integration build time we can run
>> the user code in a build container and inspect the produced routes to get
>> metadata for the optimizer. This is one of the first things we should do
>> next.
>>
>> This possibility of running code that produces integrations opened a lot
>> of
>> unexpected paths, that I've written down in the project roadmap (
>> https://github.com/nicolaferraro/integration-operator) and summarized
>> below.
>>
>> One nice feature (kudos to Luca) is that we can simplify life for Kamel
>> users up to the point that they'll just need to write their routes on a
>> Java file and run them with e.g.:
>> "kamel run Routes.java"
>>
>> Where "kamel" is a binary we release within the project. Kubernetes custom
>> resources will be used under the hood, but the "kamel" binary is a utility
>> that will provide a user experience comparable to that of current
>> serverless platforms (or even better).
>>
>> But there's not just that.
>>
>> Luca wrote this weekend a prototype for having a polyglot Camel (
>> https://github.com/lburgazzoli/camel-routes-loader).
>>
>> With that, you will be able to e.g. write integrations in groovy and use
>> the same Kamel engine:
>> "kamel run routes.groovy"
>>
>> But also JavaScript:
>>
>> https://github.com/lburgazzoli/camel-routes-loader/blob/ca986541f7c422ee02c21727cdfe4293d64a364e/src/test/resources/ext/camel/routes.js#L2
>>
>> This is a prototype right now, but a similar approach based on GraalVM has
>> a lot of potential, because users can not only use their preferred
>> language
>> to write routes and processors, but also bind them to functionalities
>> available in their preferred libraries.
>>
>> How much this will be feasible depends on the adoption of GraalVM, but I'm
>> seeing many frameworks adding metadata to make GraalVM work with
>> reflection. We've also done some work in Camel, there are some Jiras for
>> it
>> and first tests made by Guillaume signal that it's something feasible, at
>> least for the Camel core and a subset of components. GraalVM would be
>> important also to reduce memory footprint and improve startup time, as
>> already said.
>>
>
> I'll continue experimenting and I'll report back.
> I did some initial experiments leveraging some modifications I did to
> speed up the start up time [1] and on a single route [2].
> There may be some limitations down the road of course, but at least it
> shows that it's feasible. Fwiw, the experiments lead to a startup time of
> 14 ms compared to a few hundreds milliseconds (roughly 680 with [1]).
>
>
>>
>> If the GraalVM approach works (it is working for other frameworks),
>> instead
>> of rewriting a subset of Camel in Go (as the original proposal mentions),
>> we can just sanitize and recompile our existing codebase: this way we
>> fully
>> leverage the strength of Apache Camel.
>>
>
> Right, if we can avoid rewriting Camel, that would be much better !
>
> [1] https://issues.apache.org/jira/browse/CAMEL-12688
> [2]
> https://github.com/gnodet/openwhisk-runtime-camel/blob/master/camel-openwhisk-example/src/main/java/org/jboss/fuse/openwhisk/camel/example/SimpleCamelFunction.java
>
>
>>
>> Another important thing in the roadmap is actually how we'll integrate
>> with
>> current FaaS platforms. We will evaluate together if it's better to have a
>> tighter integration with some of them or to leverage knative for some
>> aspects of Kamel...
>>
>> But, let's get started!
>>
>> Il 19 lug 2018 10:45, "Nicola Ferraro" <ni...@gmail.com> ha scritto:
>>
>> It's clear to me that we need to add support for our existing XML DSL,
>> that
>> is powerful. But there are multiple reasons why I'd like to also "add" a
>> limited yaml/json notation to "Kamel".
>>
>> The first one (and simplest) is that json/yaml is the primary encoding for
>> all resources exchanged with the Kubernetes API server. I'm not saying
>> that
>> for this reason the Integration resource cannot contain a XML
>> (/Java/Ballerina) section, but from a pure presentational point of view,
>> having the possibility to write simple use cases (even a "hello world") in
>> pure json/yaml is better than always requiring a mix of json and XML.
>>
>> The second one is simplicity. Writing a optimizer (the module that
>> materializes the integration into running code, choosing a specific
>> runtime
>> and configuration) for a turing-complete language like Java is not easy:
>> even if you manage to create a good parser, it may be able to do
>> optimizations only if you write routes in a particular way, without
>> complex
>> constructs.
>> XML is ok from this point of view. The thing is that both json/yaml and
>> XML
>> are just two different ways to serialize object trees, that can be then
>> statically analyzed.
>> The point is not XML vs json/yaml, it's more about tailoring a new
>> minimalistic DSL to the emerging use cases vs proposing "only" our classic
>> way of writing integrations. I think XML can be the "advanced" way. We can
>> experiment optimizations easily with the new DSL, and enable them also on
>> XML if it's worth.
>>
>> Scripting should be part of the spec, but I'd try to use programming
>> languages only for processing/transformation, not for the route
>> definition.
>>
>> Third one is performance. Apart from the fact that json parsers are in
>> general said to be faster than XML parsers... Given the "fast startup"
>> target that we want to reach, we may think e.g. to translate the new DSL
>> into Java or Go code, then compiling it. This allows doing parsing at
>> build
>> time in order to avoid it on startup. This kind of improvements are much
>> easier with a limited DSL but much more difficult with a existing
>> fully-fledged DSL..
>>
>> Nicola
>>
>>
>>
>> On Wed, Jul 18, 2018 at 11:51 PM Pontus Ullgren <ul...@gmail.com>
>> wrote:
>>
>> > I also like the idea but with some comments.
>> >
>> > As Hiram Chirino I'm not sure YAML/JSON is the best language for this.
>> > Perhaps a more fluent DSL like the Camel Java DSL or perhaps something
>> like
>> > Ballerina language would be better suited ?
>> > Also in my experience even simple integrations, that is simple real
>> world
>> > integration and not just hello world demos, requires
>> > you to add one or more Java class or scripting in addition to the core
>> > components. So for it to be useful there must be some way to add custom
>> > code in some way for aggregation/enrichment strategies.
>> >
>> > If you go with a GO implementation I would also like some fallback for
>> > using Java since I find it unlikely that the existing base of components
>> > will be ported any time soon.
>> >
>> > As for the name as other I like the "Kamel" name however a being a
>> newborn
>> > kotlin fan I know of one pre-existing project with that name
>> > that is close enough to allow for potential confusion.
>> > https://github.com/topicusoverheid/kamel
>> >
>> > Looking forward to see the result.
>> >
>> > // Pontus
>> >
>> >
>> >
>> >
>> >
>> >
>> > On Mon, 16 Jul 2018 at 15:15 Hiram Chirino <hi...@hiramchirino.com>
>> wrote:
>> >
>> > > Love the idea.
>> > >
>> > > Personally, I'd keep using the existing Camel XML DSL if possible.
>> You
>> > can
>> > > still embed it in the CRD. The operator that deploys it could
>> inspect it
>> > > and figure whats the most optimized runtime that can support the DSL.
>> > > Perhaps if it's only using the restricted set of camel components
>> > supported
>> > > by camel-go (https://github.com/lburgazzoli/camel-go) then is uses
>> that.
>> > > Otherwise it falls back to using camel spring boot.
>> > >
>> > > For that to work I think we need the different runtime
>> implementations to
>> > > provide a way to ask them: 'hey do you support running this camel
>> route?'
>> > > Not a trivial thing to respond to, it might require a build step in
>> there
>> > > for traditional Camel runtimes.
>> > >
>> > >
>> > > On Fri, Jul 13, 2018 at 4:32 AM Antonin Stefanutti <
>> > antonin@stefanutti.fr>
>> > > wrote:
>> > >
>> > > > Hi Nicola,
>> > > >
>> > > > I love the idea.
>> > > >
>> > > > I just wonder whether YAML/JSON is an expressive enough format in
>> the
>> > > long
>> > > > term. But as you’ve mentioned, starting simple would enable
>> > experimenting
>> > > > some very interesting / promising optimisations. So it seems worth
>> > taking
>> > > > that path, instead of trying to embed a complex DSL or the existing
>> XML
>> > > DSL
>> > > > into the CRD.
>> > > >
>> > > > Definitely +1
>> > > >
>> > > > > On 13 Jul 2018, at 01:30, Nicola Ferraro <ni...@gmail.com>
>> > wrote:
>> > > > >
>> > > > > Hi Cameleers,
>> > > > > it's now passed some time since I started thinking about a new
>> > project
>> > > > that
>> > > > > we can begin here at Apache Camel, and I'd like to have your
>> opinion.
>> > > > >
>> > > > > We've already been targeting cloud-native applications with Camel,
>> > > > > especially on top of Kubernetes, that is becoming "the standard"
>> > cloud
>> > > > > platform. But writing a Camel integration and running it on
>> > Kubernetes
>> > > > > requires some effort: choosing the base platform (spring-boot,
>> karaf,
>> > > > > simple main?), adding health checks (actuator?), packaging a
>> docker
>> > > image
>> > > > > and creating the Kubernetes resources (fabric8-maven-plugin,
>> helm?),
>> > > > > publishing the image on a docker registry, then finally deploying
>> the
>> > > > > resources on a Kubernetes cluster.
>> > > > >
>> > > > > The resulting integration container is then far from being optimal
>> > > from a
>> > > > > resource consumption point of view: it is likely that a Camel
>> > > Spring-Boot
>> > > > > application will require at least 200MB of RAM and also some CPU
>> > shares
>> > > > > because of polling threads used by many components.
>> > > > >
>> > > > > In case people use a CI/CD pipeline, it will take also a long
>> time to
>> > > get
>> > > > > from a code update to having a Kubernetes POD up and running.
>> > > > > Apart from compilation and image push/pull time, also startup
>> time is
>> > > > often
>> > > > > ~10 seconds for Camel + Spring-Boot in a container with standard
>> > limits
>> > > > on
>> > > > > resources, making it difficult to propose this combination for
>> > > > "serverless
>> > > > > integration" (this term is becoming increasingly more popular).
>> > > > >
>> > > > > So, my proposal is to start to investigate a "more cloud-native"
>> > > approach
>> > > > > to integration: *making Camel integrations first-class citizens in
>> > > > > Kubernetes, and making them super fast and lightweight.*
>> > > > >
>> > > > > We can base the project on Kubernetes Custom Resource Definitions
>> > (CRD)
>> > > > > <
>> > > >
>> > >
>> >
>> https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/
>> > > > >,
>> > > > > for example a Integration CRD and have a Kubernetes "operator"
>> > > > > <https://coreos.com/operators/> taking care of:
>> > > > > - Optimizing the integration that we want to run
>> > > > > - Packaging in a container
>> > > > > - Running it on Kubernetes
>> > > > > - Managing its entire lifecycle
>> > > > >
>> > > > > A Kubernetes-native integration may look like:
>> > > > >
>> > > > > -------------------
>> > > > > kind: "Integration"
>> > > > > apiVersion: "camel.apache.org/v1alpha1"
>> > > > > metadata:
>> > > > > name: "example"
>> > > > > spec:
>> > > > > replicas: 1
>> > > > > routes:
>> > > > > - id: timer
>> > > > > route:
>> > > > > - type: endpoint
>> > > > > uri: timer:tick
>> > > > > - type: endpoint
>> > > > > uri: log:info
>> > > > > -------------------
>> > > > >
>> > > > > For those who are not familiar with Kubernetes resources, this
>> kind
>> > of
>> > > > > YAML/JSON resource definitions are really common.
>> > > > > The example route is embedded in the Kubernetes resource
>> declaration
>> > > and
>> > > > > follows a basic "flow DSL". We may start from a basic one and
>> evolve
>> > it
>> > > > as
>> > > > > new requirements arrive from the community.
>> > > > >
>> > > > > I've made a very simple (but working) POC here:
>> > > > > https://github.com/nicolaferraro/integration-operator.
>> > > > >
>> > > > > This idea of a "Cloud-Native Camel" on Kubernetes (project
>> codename
>> > can
>> > > > be "
>> > > > > *Kamel*", if you like it :D), will be an enabler for a lot of nice
>> > > > features.
>> > > > >
>> > > > > For example, we can propose "Kamel" as "ideal" platform for
>> > "serverless
>> > > > > integration" (I see many people reinventing the wheel out there):
>> the
>> > > > > operator can reduce resource consumption of a single integration
>> by
>> > > > > optimizing the runtime and also pause/resume integrations when
>> they
>> > are
>> > > > not
>> > > > > used, that is the basic idea behind "serverless" (e.g. think to
>> > > > > HTTP-triggered integrations, but not only).
>> > > > > Focusing on serverless will bring more emphasis on push-based
>> > > > notifications
>> > > > > (webhooks, cloud events <https://cloudevents.io/>), that are
>> rarely
>> > > > used in
>> > > > > Camel components, that prefer a poll based approach being it
>> simpler
>> > to
>> > > > use
>> > > > > in classic deployments, but not so good in the cloud, where more
>> > > > resources
>> > > > > become higher direct costs for the users.
>> > > > >
>> > > > > The presence of the simplified DSL enables also experimenting on
>> > > > "*reduced*
>> > > > > subsets of Camel" implemented in languages other than Java, for
>> > example
>> > > > one
>> > > > > language that has a reactive approach on thread scheduling and a
>> > really
>> > > > low
>> > > > > memory footprint, like Go.
>> > > > >
>> > > > > But apart from this kind of experiments (that are valid IMO), the
>> > > "Kamel"
>> > > > > optimizer will have free room to choose the right platform for the
>> > > > > integration that the user wants to run, including, in the future,
>> > doing
>> > > > AOT
>> > > > > compilation using Graal/VM (less memory, faster startup) if the
>> > > features
>> > > > > (components) used in the integration are supporting it (maybe we
>> can
>> > > add
>> > > > > AOT compilation in the roadmap for Camel 3).
>> > > > > A silly optimization: integrations starting from "timer:..." may
>> be
>> > > > > scheduled directly with Kubernetes CronJobs, so they will consume
>> > > > resources
>> > > > > only when actually running.
>> > > > >
>> > > > > Being the final integrations lightweight and being the DSL
>> > > > > language-independent, we may see a increased adoption of Camel
>> also
>> > as
>> > > > > agile integration layer for not-only-java applications (both
>> "cloud"
>> > > and
>> > > > > "serverless" applications).
>> > > > >
>> > > > > I'm the first one that would like to work on a project ilke this.
>> > I've
>> > > > > worked on many Kubernetes/Openshift based applications and
>> frameworks
>> > > in
>> > > > > the past years, also on operators and CRDs, and I think this way
>> of
>> > > > > redesigning integrations has a lot of potential.
>> > > > >
>> > > > > Integrations will not be necessarily limited to the simplified
>> DSL,
>> > but
>> > > > we
>> > > > > can add extension points for scripting and even custom libraries
>> > > > (although
>> > > > > limiting the freedom of the optimizer).
>> > > > >
>> > > > > The most important thing: it may become a great project, since
>> it's
>> > > > driven
>> > > > > by a great community.
>> > > > >
>> > > > > So, what do you think? Is it crazy enough?
>> > > > >
>> > > > > Nicola
>> > > >
>> > > >
>> > >
>> > > --
>> > > Hiram Chirino
>> > > Engineering | Red Hat, Inc.
>> > > hchirino@redhat.com | fusesource.com | redhat.com
>> > > skype: hiramchirino | twitter: @hiramchirino
>> > >
>> >
>>
>
>
> --
> ------------------------
> Guillaume Nodet
>
>
--
------------------------
Guillaume Nodet
Re: A "Kamel" crazy idea
Posted by Guillaume Nodet <gn...@apache.org>.
Le lun. 30 juil. 2018 à 17:49, Nicola Ferraro <ni...@gmail.com> a
écrit :
> Hi Cameleers,
> it seems from the comments that this "Kamel" subproject is something we
> want to start and I think that also the main camel core will benefit from
> the new features it will bring.
>
> I would like to donate the current "Kamel" code to Apache Camel, in order
> to have a initial brick to start from.
>
> From your reactions, the name "Kamel" seems cool and I'd like to keep it.
> However if you talk to people about the "Kamel" project, they won't
> understand if you're talking about "Kamel with the K" or "Camel with the
> C".
> For this reasons I propose to keep "Kamel", but also use "Camel K" [ˈkæməl
> keɪ] as friendly name when talking about it. This is in line with the
> repository we should create for the subproject, that needs to start with
> "camel-" (it will be "apache/camel-k" in that case).
>
> These days I've been experimenting different ideas with Luca Burgazzoli. In
> particular, there have been some concerns here on the power of a
> declarative DSL (a new one or the existing XML one) and we've found a
> strategy that will allow us to use also the Java DSL in Kamel integrations.
> Without adding too many details here, at integration build time we can run
> the user code in a build container and inspect the produced routes to get
> metadata for the optimizer. This is one of the first things we should do
> next.
>
> This possibility of running code that produces integrations opened a lot of
> unexpected paths, that I've written down in the project roadmap (
> https://github.com/nicolaferraro/integration-operator) and summarized
> below.
>
> One nice feature (kudos to Luca) is that we can simplify life for Kamel
> users up to the point that they'll just need to write their routes on a
> Java file and run them with e.g.:
> "kamel run Routes.java"
>
> Where "kamel" is a binary we release within the project. Kubernetes custom
> resources will be used under the hood, but the "kamel" binary is a utility
> that will provide a user experience comparable to that of current
> serverless platforms (or even better).
>
> But there's not just that.
>
> Luca wrote this weekend a prototype for having a polyglot Camel (
> https://github.com/lburgazzoli/camel-routes-loader).
>
> With that, you will be able to e.g. write integrations in groovy and use
> the same Kamel engine:
> "kamel run routes.groovy"
>
> But also JavaScript:
>
> https://github.com/lburgazzoli/camel-routes-loader/blob/ca986541f7c422ee02c21727cdfe4293d64a364e/src/test/resources/ext/camel/routes.js#L2
>
> This is a prototype right now, but a similar approach based on GraalVM has
> a lot of potential, because users can not only use their preferred language
> to write routes and processors, but also bind them to functionalities
> available in their preferred libraries.
>
> How much this will be feasible depends on the adoption of GraalVM, but I'm
> seeing many frameworks adding metadata to make GraalVM work with
> reflection. We've also done some work in Camel, there are some Jiras for it
> and first tests made by Guillaume signal that it's something feasible, at
> least for the Camel core and a subset of components. GraalVM would be
> important also to reduce memory footprint and improve startup time, as
> already said.
>
I'll continue experimenting and I'll report back.
I did some initial experiments leveraging some modifications I did to
speed up the start up time [1] and on a single route [2].
There may be some limitations down the road of course, but at least it
shows that it's feasible. Fwiw, the experiments lead to a startup time of
14 ms compared to a few hundreds milliseconds (roughly 680 with [1]).
>
> If the GraalVM approach works (it is working for other frameworks), instead
> of rewriting a subset of Camel in Go (as the original proposal mentions),
> we can just sanitize and recompile our existing codebase: this way we fully
> leverage the strength of Apache Camel.
>
Right, if we can avoid rewriting Camel, that would be much better !
[1] https://issues.apache.org/jira/browse/CAMEL-12688
[2]
https://github.com/gnodet/openwhisk-runtime-camel/blob/master/camel-openwhisk-example/src/main/java/org/jboss/fuse/openwhisk/camel/example/SimpleCamelFunction.java
>
> Another important thing in the roadmap is actually how we'll integrate with
> current FaaS platforms. We will evaluate together if it's better to have a
> tighter integration with some of them or to leverage knative for some
> aspects of Kamel...
>
> But, let's get started!
>
> Il 19 lug 2018 10:45, "Nicola Ferraro" <ni...@gmail.com> ha scritto:
>
> It's clear to me that we need to add support for our existing XML DSL, that
> is powerful. But there are multiple reasons why I'd like to also "add" a
> limited yaml/json notation to "Kamel".
>
> The first one (and simplest) is that json/yaml is the primary encoding for
> all resources exchanged with the Kubernetes API server. I'm not saying that
> for this reason the Integration resource cannot contain a XML
> (/Java/Ballerina) section, but from a pure presentational point of view,
> having the possibility to write simple use cases (even a "hello world") in
> pure json/yaml is better than always requiring a mix of json and XML.
>
> The second one is simplicity. Writing a optimizer (the module that
> materializes the integration into running code, choosing a specific runtime
> and configuration) for a turing-complete language like Java is not easy:
> even if you manage to create a good parser, it may be able to do
> optimizations only if you write routes in a particular way, without complex
> constructs.
> XML is ok from this point of view. The thing is that both json/yaml and XML
> are just two different ways to serialize object trees, that can be then
> statically analyzed.
> The point is not XML vs json/yaml, it's more about tailoring a new
> minimalistic DSL to the emerging use cases vs proposing "only" our classic
> way of writing integrations. I think XML can be the "advanced" way. We can
> experiment optimizations easily with the new DSL, and enable them also on
> XML if it's worth.
>
> Scripting should be part of the spec, but I'd try to use programming
> languages only for processing/transformation, not for the route definition.
>
> Third one is performance. Apart from the fact that json parsers are in
> general said to be faster than XML parsers... Given the "fast startup"
> target that we want to reach, we may think e.g. to translate the new DSL
> into Java or Go code, then compiling it. This allows doing parsing at build
> time in order to avoid it on startup. This kind of improvements are much
> easier with a limited DSL but much more difficult with a existing
> fully-fledged DSL..
>
> Nicola
>
>
>
> On Wed, Jul 18, 2018 at 11:51 PM Pontus Ullgren <ul...@gmail.com> wrote:
>
> > I also like the idea but with some comments.
> >
> > As Hiram Chirino I'm not sure YAML/JSON is the best language for this.
> > Perhaps a more fluent DSL like the Camel Java DSL or perhaps something
> like
> > Ballerina language would be better suited ?
> > Also in my experience even simple integrations, that is simple real world
> > integration and not just hello world demos, requires
> > you to add one or more Java class or scripting in addition to the core
> > components. So for it to be useful there must be some way to add custom
> > code in some way for aggregation/enrichment strategies.
> >
> > If you go with a GO implementation I would also like some fallback for
> > using Java since I find it unlikely that the existing base of components
> > will be ported any time soon.
> >
> > As for the name as other I like the "Kamel" name however a being a
> newborn
> > kotlin fan I know of one pre-existing project with that name
> > that is close enough to allow for potential confusion.
> > https://github.com/topicusoverheid/kamel
> >
> > Looking forward to see the result.
> >
> > // Pontus
> >
> >
> >
> >
> >
> >
> > On Mon, 16 Jul 2018 at 15:15 Hiram Chirino <hi...@hiramchirino.com>
> wrote:
> >
> > > Love the idea.
> > >
> > > Personally, I'd keep using the existing Camel XML DSL if possible. You
> > can
> > > still embed it in the CRD. The operator that deploys it could inspect
> it
> > > and figure whats the most optimized runtime that can support the DSL.
> > > Perhaps if it's only using the restricted set of camel components
> > supported
> > > by camel-go (https://github.com/lburgazzoli/camel-go) then is uses
> that.
> > > Otherwise it falls back to using camel spring boot.
> > >
> > > For that to work I think we need the different runtime implementations
> to
> > > provide a way to ask them: 'hey do you support running this camel
> route?'
> > > Not a trivial thing to respond to, it might require a build step in
> there
> > > for traditional Camel runtimes.
> > >
> > >
> > > On Fri, Jul 13, 2018 at 4:32 AM Antonin Stefanutti <
> > antonin@stefanutti.fr>
> > > wrote:
> > >
> > > > Hi Nicola,
> > > >
> > > > I love the idea.
> > > >
> > > > I just wonder whether YAML/JSON is an expressive enough format in the
> > > long
> > > > term. But as you’ve mentioned, starting simple would enable
> > experimenting
> > > > some very interesting / promising optimisations. So it seems worth
> > taking
> > > > that path, instead of trying to embed a complex DSL or the existing
> XML
> > > DSL
> > > > into the CRD.
> > > >
> > > > Definitely +1
> > > >
> > > > > On 13 Jul 2018, at 01:30, Nicola Ferraro <ni...@gmail.com>
> > wrote:
> > > > >
> > > > > Hi Cameleers,
> > > > > it's now passed some time since I started thinking about a new
> > project
> > > > that
> > > > > we can begin here at Apache Camel, and I'd like to have your
> opinion.
> > > > >
> > > > > We've already been targeting cloud-native applications with Camel,
> > > > > especially on top of Kubernetes, that is becoming "the standard"
> > cloud
> > > > > platform. But writing a Camel integration and running it on
> > Kubernetes
> > > > > requires some effort: choosing the base platform (spring-boot,
> karaf,
> > > > > simple main?), adding health checks (actuator?), packaging a docker
> > > image
> > > > > and creating the Kubernetes resources (fabric8-maven-plugin,
> helm?),
> > > > > publishing the image on a docker registry, then finally deploying
> the
> > > > > resources on a Kubernetes cluster.
> > > > >
> > > > > The resulting integration container is then far from being optimal
> > > from a
> > > > > resource consumption point of view: it is likely that a Camel
> > > Spring-Boot
> > > > > application will require at least 200MB of RAM and also some CPU
> > shares
> > > > > because of polling threads used by many components.
> > > > >
> > > > > In case people use a CI/CD pipeline, it will take also a long time
> to
> > > get
> > > > > from a code update to having a Kubernetes POD up and running.
> > > > > Apart from compilation and image push/pull time, also startup time
> is
> > > > often
> > > > > ~10 seconds for Camel + Spring-Boot in a container with standard
> > limits
> > > > on
> > > > > resources, making it difficult to propose this combination for
> > > > "serverless
> > > > > integration" (this term is becoming increasingly more popular).
> > > > >
> > > > > So, my proposal is to start to investigate a "more cloud-native"
> > > approach
> > > > > to integration: *making Camel integrations first-class citizens in
> > > > > Kubernetes, and making them super fast and lightweight.*
> > > > >
> > > > > We can base the project on Kubernetes Custom Resource Definitions
> > (CRD)
> > > > > <
> > > >
> > >
> >
> https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/
> > > > >,
> > > > > for example a Integration CRD and have a Kubernetes "operator"
> > > > > <https://coreos.com/operators/> taking care of:
> > > > > - Optimizing the integration that we want to run
> > > > > - Packaging in a container
> > > > > - Running it on Kubernetes
> > > > > - Managing its entire lifecycle
> > > > >
> > > > > A Kubernetes-native integration may look like:
> > > > >
> > > > > -------------------
> > > > > kind: "Integration"
> > > > > apiVersion: "camel.apache.org/v1alpha1"
> > > > > metadata:
> > > > > name: "example"
> > > > > spec:
> > > > > replicas: 1
> > > > > routes:
> > > > > - id: timer
> > > > > route:
> > > > > - type: endpoint
> > > > > uri: timer:tick
> > > > > - type: endpoint
> > > > > uri: log:info
> > > > > -------------------
> > > > >
> > > > > For those who are not familiar with Kubernetes resources, this kind
> > of
> > > > > YAML/JSON resource definitions are really common.
> > > > > The example route is embedded in the Kubernetes resource
> declaration
> > > and
> > > > > follows a basic "flow DSL". We may start from a basic one and
> evolve
> > it
> > > > as
> > > > > new requirements arrive from the community.
> > > > >
> > > > > I've made a very simple (but working) POC here:
> > > > > https://github.com/nicolaferraro/integration-operator.
> > > > >
> > > > > This idea of a "Cloud-Native Camel" on Kubernetes (project codename
> > can
> > > > be "
> > > > > *Kamel*", if you like it :D), will be an enabler for a lot of nice
> > > > features.
> > > > >
> > > > > For example, we can propose "Kamel" as "ideal" platform for
> > "serverless
> > > > > integration" (I see many people reinventing the wheel out there):
> the
> > > > > operator can reduce resource consumption of a single integration by
> > > > > optimizing the runtime and also pause/resume integrations when they
> > are
> > > > not
> > > > > used, that is the basic idea behind "serverless" (e.g. think to
> > > > > HTTP-triggered integrations, but not only).
> > > > > Focusing on serverless will bring more emphasis on push-based
> > > > notifications
> > > > > (webhooks, cloud events <https://cloudevents.io/>), that are
> rarely
> > > > used in
> > > > > Camel components, that prefer a poll based approach being it
> simpler
> > to
> > > > use
> > > > > in classic deployments, but not so good in the cloud, where more
> > > > resources
> > > > > become higher direct costs for the users.
> > > > >
> > > > > The presence of the simplified DSL enables also experimenting on
> > > > "*reduced*
> > > > > subsets of Camel" implemented in languages other than Java, for
> > example
> > > > one
> > > > > language that has a reactive approach on thread scheduling and a
> > really
> > > > low
> > > > > memory footprint, like Go.
> > > > >
> > > > > But apart from this kind of experiments (that are valid IMO), the
> > > "Kamel"
> > > > > optimizer will have free room to choose the right platform for the
> > > > > integration that the user wants to run, including, in the future,
> > doing
> > > > AOT
> > > > > compilation using Graal/VM (less memory, faster startup) if the
> > > features
> > > > > (components) used in the integration are supporting it (maybe we
> can
> > > add
> > > > > AOT compilation in the roadmap for Camel 3).
> > > > > A silly optimization: integrations starting from "timer:..." may be
> > > > > scheduled directly with Kubernetes CronJobs, so they will consume
> > > > resources
> > > > > only when actually running.
> > > > >
> > > > > Being the final integrations lightweight and being the DSL
> > > > > language-independent, we may see a increased adoption of Camel also
> > as
> > > > > agile integration layer for not-only-java applications (both
> "cloud"
> > > and
> > > > > "serverless" applications).
> > > > >
> > > > > I'm the first one that would like to work on a project ilke this.
> > I've
> > > > > worked on many Kubernetes/Openshift based applications and
> frameworks
> > > in
> > > > > the past years, also on operators and CRDs, and I think this way of
> > > > > redesigning integrations has a lot of potential.
> > > > >
> > > > > Integrations will not be necessarily limited to the simplified DSL,
> > but
> > > > we
> > > > > can add extension points for scripting and even custom libraries
> > > > (although
> > > > > limiting the freedom of the optimizer).
> > > > >
> > > > > The most important thing: it may become a great project, since it's
> > > > driven
> > > > > by a great community.
> > > > >
> > > > > So, what do you think? Is it crazy enough?
> > > > >
> > > > > Nicola
> > > >
> > > >
> > >
> > > --
> > > Hiram Chirino
> > > Engineering | Red Hat, Inc.
> > > hchirino@redhat.com | fusesource.com | redhat.com
> > > skype: hiramchirino | twitter: @hiramchirino
> > >
> >
>
--
------------------------
Guillaume Nodet
Re: A "Kamel" crazy idea
Posted by Andrea Tarocchi <an...@gmail.com>.
Hi Nicola,
very nice idea +1!
On Mon, Jul 30, 2018 at 5:49 PM Nicola Ferraro <ni...@gmail.com> wrote:
> Hi Cameleers,
> it seems from the comments that this "Kamel" subproject is something we
> want to start and I think that also the main camel core will benefit from
> the new features it will bring.
>
> I would like to donate the current "Kamel" code to Apache Camel, in order
> to have a initial brick to start from.
>
> From your reactions, the name "Kamel" seems cool and I'd like to keep it.
> However if you talk to people about the "Kamel" project, they won't
> understand if you're talking about "Kamel with the K" or "Camel with the
> C".
> For this reasons I propose to keep "Kamel", but also use "Camel K" [ˈkæməl
> keɪ] as friendly name when talking about it. This is in line with the
> repository we should create for the subproject, that needs to start with
> "camel-" (it will be "apache/camel-k" in that case).
>
> These days I've been experimenting different ideas with Luca Burgazzoli. In
> particular, there have been some concerns here on the power of a
> declarative DSL (a new one or the existing XML one) and we've found a
> strategy that will allow us to use also the Java DSL in Kamel integrations.
> Without adding too many details here, at integration build time we can run
> the user code in a build container and inspect the produced routes to get
> metadata for the optimizer. This is one of the first things we should do
> next.
>
> This possibility of running code that produces integrations opened a lot of
> unexpected paths, that I've written down in the project roadmap (
> https://github.com/nicolaferraro/integration-operator) and summarized
> below.
>
> One nice feature (kudos to Luca) is that we can simplify life for Kamel
> users up to the point that they'll just need to write their routes on a
> Java file and run them with e.g.:
> "kamel run Routes.java"
>
> Where "kamel" is a binary we release within the project. Kubernetes custom
> resources will be used under the hood, but the "kamel" binary is a utility
> that will provide a user experience comparable to that of current
> serverless platforms (or even better).
>
> But there's not just that.
>
> Luca wrote this weekend a prototype for having a polyglot Camel (
> https://github.com/lburgazzoli/camel-routes-loader).
>
> With that, you will be able to e.g. write integrations in groovy and use
> the same Kamel engine:
> "kamel run routes.groovy"
>
> But also JavaScript:
>
> https://github.com/lburgazzoli/camel-routes-loader/blob/ca986541f7c422ee02c21727cdfe4293d64a364e/src/test/resources/ext/camel/routes.js#L2
>
> This is a prototype right now, but a similar approach based on GraalVM has
> a lot of potential, because users can not only use their preferred language
> to write routes and processors, but also bind them to functionalities
> available in their preferred libraries.
>
> How much this will be feasible depends on the adoption of GraalVM, but I'm
> seeing many frameworks adding metadata to make GraalVM work with
> reflection. We've also done some work in Camel, there are some Jiras for it
> and first tests made by Guillaume signal that it's something feasible, at
> least for the Camel core and a subset of components. GraalVM would be
> important also to reduce memory footprint and improve startup time, as
> already said.
>
> If the GraalVM approach works (it is working for other frameworks), instead
> of rewriting a subset of Camel in Go (as the original proposal mentions),
> we can just sanitize and recompile our existing codebase: this way we fully
> leverage the strength of Apache Camel.
>
> Another important thing in the roadmap is actually how we'll integrate with
> current FaaS platforms. We will evaluate together if it's better to have a
> tighter integration with some of them or to leverage knative for some
> aspects of Kamel...
>
> But, let's get started!
>
> Il 19 lug 2018 10:45, "Nicola Ferraro" <ni...@gmail.com> ha scritto:
>
> It's clear to me that we need to add support for our existing XML DSL, that
> is powerful. But there are multiple reasons why I'd like to also "add" a
> limited yaml/json notation to "Kamel".
>
> The first one (and simplest) is that json/yaml is the primary encoding for
> all resources exchanged with the Kubernetes API server. I'm not saying that
> for this reason the Integration resource cannot contain a XML
> (/Java/Ballerina) section, but from a pure presentational point of view,
> having the possibility to write simple use cases (even a "hello world") in
> pure json/yaml is better than always requiring a mix of json and XML.
>
> The second one is simplicity. Writing a optimizer (the module that
> materializes the integration into running code, choosing a specific runtime
> and configuration) for a turing-complete language like Java is not easy:
> even if you manage to create a good parser, it may be able to do
> optimizations only if you write routes in a particular way, without complex
> constructs.
> XML is ok from this point of view. The thing is that both json/yaml and XML
> are just two different ways to serialize object trees, that can be then
> statically analyzed.
> The point is not XML vs json/yaml, it's more about tailoring a new
> minimalistic DSL to the emerging use cases vs proposing "only" our classic
> way of writing integrations. I think XML can be the "advanced" way. We can
> experiment optimizations easily with the new DSL, and enable them also on
> XML if it's worth.
>
> Scripting should be part of the spec, but I'd try to use programming
> languages only for processing/transformation, not for the route definition.
>
> Third one is performance. Apart from the fact that json parsers are in
> general said to be faster than XML parsers... Given the "fast startup"
> target that we want to reach, we may think e.g. to translate the new DSL
> into Java or Go code, then compiling it. This allows doing parsing at build
> time in order to avoid it on startup. This kind of improvements are much
> easier with a limited DSL but much more difficult with a existing
> fully-fledged DSL..
>
> Nicola
>
>
>
> On Wed, Jul 18, 2018 at 11:51 PM Pontus Ullgren <ul...@gmail.com> wrote:
>
> > I also like the idea but with some comments.
> >
> > As Hiram Chirino I'm not sure YAML/JSON is the best language for this.
> > Perhaps a more fluent DSL like the Camel Java DSL or perhaps something
> like
> > Ballerina language would be better suited ?
> > Also in my experience even simple integrations, that is simple real world
> > integration and not just hello world demos, requires
> > you to add one or more Java class or scripting in addition to the core
> > components. So for it to be useful there must be some way to add custom
> > code in some way for aggregation/enrichment strategies.
> >
> > If you go with a GO implementation I would also like some fallback for
> > using Java since I find it unlikely that the existing base of components
> > will be ported any time soon.
> >
> > As for the name as other I like the "Kamel" name however a being a
> newborn
> > kotlin fan I know of one pre-existing project with that name
> > that is close enough to allow for potential confusion.
> > https://github.com/topicusoverheid/kamel
> >
> > Looking forward to see the result.
> >
> > // Pontus
> >
> >
> >
> >
> >
> >
> > On Mon, 16 Jul 2018 at 15:15 Hiram Chirino <hi...@hiramchirino.com>
> wrote:
> >
> > > Love the idea.
> > >
> > > Personally, I'd keep using the existing Camel XML DSL if possible. You
> > can
> > > still embed it in the CRD. The operator that deploys it could inspect
> it
> > > and figure whats the most optimized runtime that can support the DSL.
> > > Perhaps if it's only using the restricted set of camel components
> > supported
> > > by camel-go (https://github.com/lburgazzoli/camel-go) then is uses
> that.
> > > Otherwise it falls back to using camel spring boot.
> > >
> > > For that to work I think we need the different runtime implementations
> to
> > > provide a way to ask them: 'hey do you support running this camel
> route?'
> > > Not a trivial thing to respond to, it might require a build step in
> there
> > > for traditional Camel runtimes.
> > >
> > >
> > > On Fri, Jul 13, 2018 at 4:32 AM Antonin Stefanutti <
> > antonin@stefanutti.fr>
> > > wrote:
> > >
> > > > Hi Nicola,
> > > >
> > > > I love the idea.
> > > >
> > > > I just wonder whether YAML/JSON is an expressive enough format in the
> > > long
> > > > term. But as you’ve mentioned, starting simple would enable
> > experimenting
> > > > some very interesting / promising optimisations. So it seems worth
> > taking
> > > > that path, instead of trying to embed a complex DSL or the existing
> XML
> > > DSL
> > > > into the CRD.
> > > >
> > > > Definitely +1
> > > >
> > > > > On 13 Jul 2018, at 01:30, Nicola Ferraro <ni...@gmail.com>
> > wrote:
> > > > >
> > > > > Hi Cameleers,
> > > > > it's now passed some time since I started thinking about a new
> > project
> > > > that
> > > > > we can begin here at Apache Camel, and I'd like to have your
> opinion.
> > > > >
> > > > > We've already been targeting cloud-native applications with Camel,
> > > > > especially on top of Kubernetes, that is becoming "the standard"
> > cloud
> > > > > platform. But writing a Camel integration and running it on
> > Kubernetes
> > > > > requires some effort: choosing the base platform (spring-boot,
> karaf,
> > > > > simple main?), adding health checks (actuator?), packaging a docker
> > > image
> > > > > and creating the Kubernetes resources (fabric8-maven-plugin,
> helm?),
> > > > > publishing the image on a docker registry, then finally deploying
> the
> > > > > resources on a Kubernetes cluster.
> > > > >
> > > > > The resulting integration container is then far from being optimal
> > > from a
> > > > > resource consumption point of view: it is likely that a Camel
> > > Spring-Boot
> > > > > application will require at least 200MB of RAM and also some CPU
> > shares
> > > > > because of polling threads used by many components.
> > > > >
> > > > > In case people use a CI/CD pipeline, it will take also a long time
> to
> > > get
> > > > > from a code update to having a Kubernetes POD up and running.
> > > > > Apart from compilation and image push/pull time, also startup time
> is
> > > > often
> > > > > ~10 seconds for Camel + Spring-Boot in a container with standard
> > limits
> > > > on
> > > > > resources, making it difficult to propose this combination for
> > > > "serverless
> > > > > integration" (this term is becoming increasingly more popular).
> > > > >
> > > > > So, my proposal is to start to investigate a "more cloud-native"
> > > approach
> > > > > to integration: *making Camel integrations first-class citizens in
> > > > > Kubernetes, and making them super fast and lightweight.*
> > > > >
> > > > > We can base the project on Kubernetes Custom Resource Definitions
> > (CRD)
> > > > > <
> > > >
> > >
> >
> https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/
> > > > >,
> > > > > for example a Integration CRD and have a Kubernetes "operator"
> > > > > <https://coreos.com/operators/> taking care of:
> > > > > - Optimizing the integration that we want to run
> > > > > - Packaging in a container
> > > > > - Running it on Kubernetes
> > > > > - Managing its entire lifecycle
> > > > >
> > > > > A Kubernetes-native integration may look like:
> > > > >
> > > > > -------------------
> > > > > kind: "Integration"
> > > > > apiVersion: "camel.apache.org/v1alpha1"
> > > > > metadata:
> > > > > name: "example"
> > > > > spec:
> > > > > replicas: 1
> > > > > routes:
> > > > > - id: timer
> > > > > route:
> > > > > - type: endpoint
> > > > > uri: timer:tick
> > > > > - type: endpoint
> > > > > uri: log:info
> > > > > -------------------
> > > > >
> > > > > For those who are not familiar with Kubernetes resources, this kind
> > of
> > > > > YAML/JSON resource definitions are really common.
> > > > > The example route is embedded in the Kubernetes resource
> declaration
> > > and
> > > > > follows a basic "flow DSL". We may start from a basic one and
> evolve
> > it
> > > > as
> > > > > new requirements arrive from the community.
> > > > >
> > > > > I've made a very simple (but working) POC here:
> > > > > https://github.com/nicolaferraro/integration-operator.
> > > > >
> > > > > This idea of a "Cloud-Native Camel" on Kubernetes (project codename
> > can
> > > > be "
> > > > > *Kamel*", if you like it :D), will be an enabler for a lot of nice
> > > > features.
> > > > >
> > > > > For example, we can propose "Kamel" as "ideal" platform for
> > "serverless
> > > > > integration" (I see many people reinventing the wheel out there):
> the
> > > > > operator can reduce resource consumption of a single integration by
> > > > > optimizing the runtime and also pause/resume integrations when they
> > are
> > > > not
> > > > > used, that is the basic idea behind "serverless" (e.g. think to
> > > > > HTTP-triggered integrations, but not only).
> > > > > Focusing on serverless will bring more emphasis on push-based
> > > > notifications
> > > > > (webhooks, cloud events <https://cloudevents.io/>), that are
> rarely
> > > > used in
> > > > > Camel components, that prefer a poll based approach being it
> simpler
> > to
> > > > use
> > > > > in classic deployments, but not so good in the cloud, where more
> > > > resources
> > > > > become higher direct costs for the users.
> > > > >
> > > > > The presence of the simplified DSL enables also experimenting on
> > > > "*reduced*
> > > > > subsets of Camel" implemented in languages other than Java, for
> > example
> > > > one
> > > > > language that has a reactive approach on thread scheduling and a
> > really
> > > > low
> > > > > memory footprint, like Go.
> > > > >
> > > > > But apart from this kind of experiments (that are valid IMO), the
> > > "Kamel"
> > > > > optimizer will have free room to choose the right platform for the
> > > > > integration that the user wants to run, including, in the future,
> > doing
> > > > AOT
> > > > > compilation using Graal/VM (less memory, faster startup) if the
> > > features
> > > > > (components) used in the integration are supporting it (maybe we
> can
> > > add
> > > > > AOT compilation in the roadmap for Camel 3).
> > > > > A silly optimization: integrations starting from "timer:..." may be
> > > > > scheduled directly with Kubernetes CronJobs, so they will consume
> > > > resources
> > > > > only when actually running.
> > > > >
> > > > > Being the final integrations lightweight and being the DSL
> > > > > language-independent, we may see a increased adoption of Camel also
> > as
> > > > > agile integration layer for not-only-java applications (both
> "cloud"
> > > and
> > > > > "serverless" applications).
> > > > >
> > > > > I'm the first one that would like to work on a project ilke this.
> > I've
> > > > > worked on many Kubernetes/Openshift based applications and
> frameworks
> > > in
> > > > > the past years, also on operators and CRDs, and I think this way of
> > > > > redesigning integrations has a lot of potential.
> > > > >
> > > > > Integrations will not be necessarily limited to the simplified DSL,
> > but
> > > > we
> > > > > can add extension points for scripting and even custom libraries
> > > > (although
> > > > > limiting the freedom of the optimizer).
> > > > >
> > > > > The most important thing: it may become a great project, since it's
> > > > driven
> > > > > by a great community.
> > > > >
> > > > > So, what do you think? Is it crazy enough?
> > > > >
> > > > > Nicola
> > > >
> > > >
> > >
> > > --
> > > Hiram Chirino
> > > Engineering | Red Hat, Inc.
> > > hchirino@redhat.com | fusesource.com | redhat.com
> > > skype: hiramchirino | twitter: @hiramchirino
> > >
> >
>
--
"In a world without walls and fences who needs Windows and Gates?"
<https://about.me/andrea.tarocchi?promo=email_sig&utm_source=product&utm_medium=email_sig&utm_campaign=edit_panel&utm_content=thumb>
Andrea Tarocchi
about.me/andrea.tarocchi
<https://about.me/andrea.tarocchi?promo=email_sig&utm_source=product&utm_medium=email_sig&utm_campaign=edit_panel&utm_content=thumb>
Re: A "Kamel" crazy idea
Posted by Nicola Ferraro <ni...@gmail.com>.
Hi Cameleers,
it seems from the comments that this "Kamel" subproject is something we
want to start and I think that also the main camel core will benefit from
the new features it will bring.
I would like to donate the current "Kamel" code to Apache Camel, in order
to have a initial brick to start from.
From your reactions, the name "Kamel" seems cool and I'd like to keep it.
However if you talk to people about the "Kamel" project, they won't
understand if you're talking about "Kamel with the K" or "Camel with the C".
For this reasons I propose to keep "Kamel", but also use "Camel K" [ˈkæməl
keɪ] as friendly name when talking about it. This is in line with the
repository we should create for the subproject, that needs to start with
"camel-" (it will be "apache/camel-k" in that case).
These days I've been experimenting different ideas with Luca Burgazzoli. In
particular, there have been some concerns here on the power of a
declarative DSL (a new one or the existing XML one) and we've found a
strategy that will allow us to use also the Java DSL in Kamel integrations.
Without adding too many details here, at integration build time we can run
the user code in a build container and inspect the produced routes to get
metadata for the optimizer. This is one of the first things we should do
next.
This possibility of running code that produces integrations opened a lot of
unexpected paths, that I've written down in the project roadmap (
https://github.com/nicolaferraro/integration-operator) and summarized below.
One nice feature (kudos to Luca) is that we can simplify life for Kamel
users up to the point that they'll just need to write their routes on a
Java file and run them with e.g.:
"kamel run Routes.java"
Where "kamel" is a binary we release within the project. Kubernetes custom
resources will be used under the hood, but the "kamel" binary is a utility
that will provide a user experience comparable to that of current
serverless platforms (or even better).
But there's not just that.
Luca wrote this weekend a prototype for having a polyglot Camel (
https://github.com/lburgazzoli/camel-routes-loader).
With that, you will be able to e.g. write integrations in groovy and use
the same Kamel engine:
"kamel run routes.groovy"
But also JavaScript:
https://github.com/lburgazzoli/camel-routes-loader/blob/ca986541f7c422ee02c21727cdfe4293d64a364e/src/test/resources/ext/camel/routes.js#L2
This is a prototype right now, but a similar approach based on GraalVM has
a lot of potential, because users can not only use their preferred language
to write routes and processors, but also bind them to functionalities
available in their preferred libraries.
How much this will be feasible depends on the adoption of GraalVM, but I'm
seeing many frameworks adding metadata to make GraalVM work with
reflection. We've also done some work in Camel, there are some Jiras for it
and first tests made by Guillaume signal that it's something feasible, at
least for the Camel core and a subset of components. GraalVM would be
important also to reduce memory footprint and improve startup time, as
already said.
If the GraalVM approach works (it is working for other frameworks), instead
of rewriting a subset of Camel in Go (as the original proposal mentions),
we can just sanitize and recompile our existing codebase: this way we fully
leverage the strength of Apache Camel.
Another important thing in the roadmap is actually how we'll integrate with
current FaaS platforms. We will evaluate together if it's better to have a
tighter integration with some of them or to leverage knative for some
aspects of Kamel...
But, let's get started!
Il 19 lug 2018 10:45, "Nicola Ferraro" <ni...@gmail.com> ha scritto:
It's clear to me that we need to add support for our existing XML DSL, that
is powerful. But there are multiple reasons why I'd like to also "add" a
limited yaml/json notation to "Kamel".
The first one (and simplest) is that json/yaml is the primary encoding for
all resources exchanged with the Kubernetes API server. I'm not saying that
for this reason the Integration resource cannot contain a XML
(/Java/Ballerina) section, but from a pure presentational point of view,
having the possibility to write simple use cases (even a "hello world") in
pure json/yaml is better than always requiring a mix of json and XML.
The second one is simplicity. Writing a optimizer (the module that
materializes the integration into running code, choosing a specific runtime
and configuration) for a turing-complete language like Java is not easy:
even if you manage to create a good parser, it may be able to do
optimizations only if you write routes in a particular way, without complex
constructs.
XML is ok from this point of view. The thing is that both json/yaml and XML
are just two different ways to serialize object trees, that can be then
statically analyzed.
The point is not XML vs json/yaml, it's more about tailoring a new
minimalistic DSL to the emerging use cases vs proposing "only" our classic
way of writing integrations. I think XML can be the "advanced" way. We can
experiment optimizations easily with the new DSL, and enable them also on
XML if it's worth.
Scripting should be part of the spec, but I'd try to use programming
languages only for processing/transformation, not for the route definition.
Third one is performance. Apart from the fact that json parsers are in
general said to be faster than XML parsers... Given the "fast startup"
target that we want to reach, we may think e.g. to translate the new DSL
into Java or Go code, then compiling it. This allows doing parsing at build
time in order to avoid it on startup. This kind of improvements are much
easier with a limited DSL but much more difficult with a existing
fully-fledged DSL..
Nicola
On Wed, Jul 18, 2018 at 11:51 PM Pontus Ullgren <ul...@gmail.com> wrote:
> I also like the idea but with some comments.
>
> As Hiram Chirino I'm not sure YAML/JSON is the best language for this.
> Perhaps a more fluent DSL like the Camel Java DSL or perhaps something like
> Ballerina language would be better suited ?
> Also in my experience even simple integrations, that is simple real world
> integration and not just hello world demos, requires
> you to add one or more Java class or scripting in addition to the core
> components. So for it to be useful there must be some way to add custom
> code in some way for aggregation/enrichment strategies.
>
> If you go with a GO implementation I would also like some fallback for
> using Java since I find it unlikely that the existing base of components
> will be ported any time soon.
>
> As for the name as other I like the "Kamel" name however a being a newborn
> kotlin fan I know of one pre-existing project with that name
> that is close enough to allow for potential confusion.
> https://github.com/topicusoverheid/kamel
>
> Looking forward to see the result.
>
> // Pontus
>
>
>
>
>
>
> On Mon, 16 Jul 2018 at 15:15 Hiram Chirino <hi...@hiramchirino.com> wrote:
>
> > Love the idea.
> >
> > Personally, I'd keep using the existing Camel XML DSL if possible. You
> can
> > still embed it in the CRD. The operator that deploys it could inspect it
> > and figure whats the most optimized runtime that can support the DSL.
> > Perhaps if it's only using the restricted set of camel components
> supported
> > by camel-go (https://github.com/lburgazzoli/camel-go) then is uses that.
> > Otherwise it falls back to using camel spring boot.
> >
> > For that to work I think we need the different runtime implementations to
> > provide a way to ask them: 'hey do you support running this camel route?'
> > Not a trivial thing to respond to, it might require a build step in there
> > for traditional Camel runtimes.
> >
> >
> > On Fri, Jul 13, 2018 at 4:32 AM Antonin Stefanutti <
> antonin@stefanutti.fr>
> > wrote:
> >
> > > Hi Nicola,
> > >
> > > I love the idea.
> > >
> > > I just wonder whether YAML/JSON is an expressive enough format in the
> > long
> > > term. But as you’ve mentioned, starting simple would enable
> experimenting
> > > some very interesting / promising optimisations. So it seems worth
> taking
> > > that path, instead of trying to embed a complex DSL or the existing XML
> > DSL
> > > into the CRD.
> > >
> > > Definitely +1
> > >
> > > > On 13 Jul 2018, at 01:30, Nicola Ferraro <ni...@gmail.com>
> wrote:
> > > >
> > > > Hi Cameleers,
> > > > it's now passed some time since I started thinking about a new
> project
> > > that
> > > > we can begin here at Apache Camel, and I'd like to have your opinion.
> > > >
> > > > We've already been targeting cloud-native applications with Camel,
> > > > especially on top of Kubernetes, that is becoming "the standard"
> cloud
> > > > platform. But writing a Camel integration and running it on
> Kubernetes
> > > > requires some effort: choosing the base platform (spring-boot, karaf,
> > > > simple main?), adding health checks (actuator?), packaging a docker
> > image
> > > > and creating the Kubernetes resources (fabric8-maven-plugin, helm?),
> > > > publishing the image on a docker registry, then finally deploying the
> > > > resources on a Kubernetes cluster.
> > > >
> > > > The resulting integration container is then far from being optimal
> > from a
> > > > resource consumption point of view: it is likely that a Camel
> > Spring-Boot
> > > > application will require at least 200MB of RAM and also some CPU
> shares
> > > > because of polling threads used by many components.
> > > >
> > > > In case people use a CI/CD pipeline, it will take also a long time to
> > get
> > > > from a code update to having a Kubernetes POD up and running.
> > > > Apart from compilation and image push/pull time, also startup time is
> > > often
> > > > ~10 seconds for Camel + Spring-Boot in a container with standard
> limits
> > > on
> > > > resources, making it difficult to propose this combination for
> > > "serverless
> > > > integration" (this term is becoming increasingly more popular).
> > > >
> > > > So, my proposal is to start to investigate a "more cloud-native"
> > approach
> > > > to integration: *making Camel integrations first-class citizens in
> > > > Kubernetes, and making them super fast and lightweight.*
> > > >
> > > > We can base the project on Kubernetes Custom Resource Definitions
> (CRD)
> > > > <
> > >
> >
> https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/
> > > >,
> > > > for example a Integration CRD and have a Kubernetes "operator"
> > > > <https://coreos.com/operators/> taking care of:
> > > > - Optimizing the integration that we want to run
> > > > - Packaging in a container
> > > > - Running it on Kubernetes
> > > > - Managing its entire lifecycle
> > > >
> > > > A Kubernetes-native integration may look like:
> > > >
> > > > -------------------
> > > > kind: "Integration"
> > > > apiVersion: "camel.apache.org/v1alpha1"
> > > > metadata:
> > > > name: "example"
> > > > spec:
> > > > replicas: 1
> > > > routes:
> > > > - id: timer
> > > > route:
> > > > - type: endpoint
> > > > uri: timer:tick
> > > > - type: endpoint
> > > > uri: log:info
> > > > -------------------
> > > >
> > > > For those who are not familiar with Kubernetes resources, this kind
> of
> > > > YAML/JSON resource definitions are really common.
> > > > The example route is embedded in the Kubernetes resource declaration
> > and
> > > > follows a basic "flow DSL". We may start from a basic one and evolve
> it
> > > as
> > > > new requirements arrive from the community.
> > > >
> > > > I've made a very simple (but working) POC here:
> > > > https://github.com/nicolaferraro/integration-operator.
> > > >
> > > > This idea of a "Cloud-Native Camel" on Kubernetes (project codename
> can
> > > be "
> > > > *Kamel*", if you like it :D), will be an enabler for a lot of nice
> > > features.
> > > >
> > > > For example, we can propose "Kamel" as "ideal" platform for
> "serverless
> > > > integration" (I see many people reinventing the wheel out there): the
> > > > operator can reduce resource consumption of a single integration by
> > > > optimizing the runtime and also pause/resume integrations when they
> are
> > > not
> > > > used, that is the basic idea behind "serverless" (e.g. think to
> > > > HTTP-triggered integrations, but not only).
> > > > Focusing on serverless will bring more emphasis on push-based
> > > notifications
> > > > (webhooks, cloud events <https://cloudevents.io/>), that are rarely
> > > used in
> > > > Camel components, that prefer a poll based approach being it simpler
> to
> > > use
> > > > in classic deployments, but not so good in the cloud, where more
> > > resources
> > > > become higher direct costs for the users.
> > > >
> > > > The presence of the simplified DSL enables also experimenting on
> > > "*reduced*
> > > > subsets of Camel" implemented in languages other than Java, for
> example
> > > one
> > > > language that has a reactive approach on thread scheduling and a
> really
> > > low
> > > > memory footprint, like Go.
> > > >
> > > > But apart from this kind of experiments (that are valid IMO), the
> > "Kamel"
> > > > optimizer will have free room to choose the right platform for the
> > > > integration that the user wants to run, including, in the future,
> doing
> > > AOT
> > > > compilation using Graal/VM (less memory, faster startup) if the
> > features
> > > > (components) used in the integration are supporting it (maybe we can
> > add
> > > > AOT compilation in the roadmap for Camel 3).
> > > > A silly optimization: integrations starting from "timer:..." may be
> > > > scheduled directly with Kubernetes CronJobs, so they will consume
> > > resources
> > > > only when actually running.
> > > >
> > > > Being the final integrations lightweight and being the DSL
> > > > language-independent, we may see a increased adoption of Camel also
> as
> > > > agile integration layer for not-only-java applications (both "cloud"
> > and
> > > > "serverless" applications).
> > > >
> > > > I'm the first one that would like to work on a project ilke this.
> I've
> > > > worked on many Kubernetes/Openshift based applications and frameworks
> > in
> > > > the past years, also on operators and CRDs, and I think this way of
> > > > redesigning integrations has a lot of potential.
> > > >
> > > > Integrations will not be necessarily limited to the simplified DSL,
> but
> > > we
> > > > can add extension points for scripting and even custom libraries
> > > (although
> > > > limiting the freedom of the optimizer).
> > > >
> > > > The most important thing: it may become a great project, since it's
> > > driven
> > > > by a great community.
> > > >
> > > > So, what do you think? Is it crazy enough?
> > > >
> > > > Nicola
> > >
> > >
> >
> > --
> > Hiram Chirino
> > Engineering | Red Hat, Inc.
> > hchirino@redhat.com | fusesource.com | redhat.com
> > skype: hiramchirino | twitter: @hiramchirino
> >
>
Re: A "Kamel" crazy idea
Posted by Nicola Ferraro <ni...@gmail.com>.
It's clear to me that we need to add support for our existing XML DSL, that
is powerful. But there are multiple reasons why I'd like to also "add" a
limited yaml/json notation to "Kamel".
The first one (and simplest) is that json/yaml is the primary encoding for
all resources exchanged with the Kubernetes API server. I'm not saying that
for this reason the Integration resource cannot contain a XML
(/Java/Ballerina) section, but from a pure presentational point of view,
having the possibility to write simple use cases (even a "hello world") in
pure json/yaml is better than always requiring a mix of json and XML.
The second one is simplicity. Writing a optimizer (the module that
materializes the integration into running code, choosing a specific runtime
and configuration) for a turing-complete language like Java is not easy:
even if you manage to create a good parser, it may be able to do
optimizations only if you write routes in a particular way, without complex
constructs.
XML is ok from this point of view. The thing is that both json/yaml and XML
are just two different ways to serialize object trees, that can be then
statically analyzed.
The point is not XML vs json/yaml, it's more about tailoring a new
minimalistic DSL to the emerging use cases vs proposing "only" our classic
way of writing integrations. I think XML can be the "advanced" way. We can
experiment optimizations easily with the new DSL, and enable them also on
XML if it's worth.
Scripting should be part of the spec, but I'd try to use programming
languages only for processing/transformation, not for the route definition.
Third one is performance. Apart from the fact that json parsers are in
general said to be faster than XML parsers... Given the "fast startup"
target that we want to reach, we may think e.g. to translate the new DSL
into Java or Go code, then compiling it. This allows doing parsing at build
time in order to avoid it on startup. This kind of improvements are much
easier with a limited DSL but much more difficult with a existing
fully-fledged DSL..
Nicola
On Wed, Jul 18, 2018 at 11:51 PM Pontus Ullgren <ul...@gmail.com> wrote:
> I also like the idea but with some comments.
>
> As Hiram Chirino I'm not sure YAML/JSON is the best language for this.
> Perhaps a more fluent DSL like the Camel Java DSL or perhaps something like
> Ballerina language would be better suited ?
> Also in my experience even simple integrations, that is simple real world
> integration and not just hello world demos, requires
> you to add one or more Java class or scripting in addition to the core
> components. So for it to be useful there must be some way to add custom
> code in some way for aggregation/enrichment strategies.
>
> If you go with a GO implementation I would also like some fallback for
> using Java since I find it unlikely that the existing base of components
> will be ported any time soon.
>
> As for the name as other I like the "Kamel" name however a being a newborn
> kotlin fan I know of one pre-existing project with that name
> that is close enough to allow for potential confusion.
> https://github.com/topicusoverheid/kamel
>
> Looking forward to see the result.
>
> // Pontus
>
>
>
>
>
>
> On Mon, 16 Jul 2018 at 15:15 Hiram Chirino <hi...@hiramchirino.com> wrote:
>
> > Love the idea.
> >
> > Personally, I'd keep using the existing Camel XML DSL if possible. You
> can
> > still embed it in the CRD. The operator that deploys it could inspect it
> > and figure whats the most optimized runtime that can support the DSL.
> > Perhaps if it's only using the restricted set of camel components
> supported
> > by camel-go (https://github.com/lburgazzoli/camel-go) then is uses that.
> > Otherwise it falls back to using camel spring boot.
> >
> > For that to work I think we need the different runtime implementations to
> > provide a way to ask them: 'hey do you support running this camel route?'
> > Not a trivial thing to respond to, it might require a build step in there
> > for traditional Camel runtimes.
> >
> >
> > On Fri, Jul 13, 2018 at 4:32 AM Antonin Stefanutti <
> antonin@stefanutti.fr>
> > wrote:
> >
> > > Hi Nicola,
> > >
> > > I love the idea.
> > >
> > > I just wonder whether YAML/JSON is an expressive enough format in the
> > long
> > > term. But as you’ve mentioned, starting simple would enable
> experimenting
> > > some very interesting / promising optimisations. So it seems worth
> taking
> > > that path, instead of trying to embed a complex DSL or the existing XML
> > DSL
> > > into the CRD.
> > >
> > > Definitely +1
> > >
> > > > On 13 Jul 2018, at 01:30, Nicola Ferraro <ni...@gmail.com>
> wrote:
> > > >
> > > > Hi Cameleers,
> > > > it's now passed some time since I started thinking about a new
> project
> > > that
> > > > we can begin here at Apache Camel, and I'd like to have your opinion.
> > > >
> > > > We've already been targeting cloud-native applications with Camel,
> > > > especially on top of Kubernetes, that is becoming "the standard"
> cloud
> > > > platform. But writing a Camel integration and running it on
> Kubernetes
> > > > requires some effort: choosing the base platform (spring-boot, karaf,
> > > > simple main?), adding health checks (actuator?), packaging a docker
> > image
> > > > and creating the Kubernetes resources (fabric8-maven-plugin, helm?),
> > > > publishing the image on a docker registry, then finally deploying the
> > > > resources on a Kubernetes cluster.
> > > >
> > > > The resulting integration container is then far from being optimal
> > from a
> > > > resource consumption point of view: it is likely that a Camel
> > Spring-Boot
> > > > application will require at least 200MB of RAM and also some CPU
> shares
> > > > because of polling threads used by many components.
> > > >
> > > > In case people use a CI/CD pipeline, it will take also a long time to
> > get
> > > > from a code update to having a Kubernetes POD up and running.
> > > > Apart from compilation and image push/pull time, also startup time is
> > > often
> > > > ~10 seconds for Camel + Spring-Boot in a container with standard
> limits
> > > on
> > > > resources, making it difficult to propose this combination for
> > > "serverless
> > > > integration" (this term is becoming increasingly more popular).
> > > >
> > > > So, my proposal is to start to investigate a "more cloud-native"
> > approach
> > > > to integration: *making Camel integrations first-class citizens in
> > > > Kubernetes, and making them super fast and lightweight.*
> > > >
> > > > We can base the project on Kubernetes Custom Resource Definitions
> (CRD)
> > > > <
> > >
> >
> https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/
> > > >,
> > > > for example a Integration CRD and have a Kubernetes "operator"
> > > > <https://coreos.com/operators/> taking care of:
> > > > - Optimizing the integration that we want to run
> > > > - Packaging in a container
> > > > - Running it on Kubernetes
> > > > - Managing its entire lifecycle
> > > >
> > > > A Kubernetes-native integration may look like:
> > > >
> > > > -------------------
> > > > kind: "Integration"
> > > > apiVersion: "camel.apache.org/v1alpha1"
> > > > metadata:
> > > > name: "example"
> > > > spec:
> > > > replicas: 1
> > > > routes:
> > > > - id: timer
> > > > route:
> > > > - type: endpoint
> > > > uri: timer:tick
> > > > - type: endpoint
> > > > uri: log:info
> > > > -------------------
> > > >
> > > > For those who are not familiar with Kubernetes resources, this kind
> of
> > > > YAML/JSON resource definitions are really common.
> > > > The example route is embedded in the Kubernetes resource declaration
> > and
> > > > follows a basic "flow DSL". We may start from a basic one and evolve
> it
> > > as
> > > > new requirements arrive from the community.
> > > >
> > > > I've made a very simple (but working) POC here:
> > > > https://github.com/nicolaferraro/integration-operator.
> > > >
> > > > This idea of a "Cloud-Native Camel" on Kubernetes (project codename
> can
> > > be "
> > > > *Kamel*", if you like it :D), will be an enabler for a lot of nice
> > > features.
> > > >
> > > > For example, we can propose "Kamel" as "ideal" platform for
> "serverless
> > > > integration" (I see many people reinventing the wheel out there): the
> > > > operator can reduce resource consumption of a single integration by
> > > > optimizing the runtime and also pause/resume integrations when they
> are
> > > not
> > > > used, that is the basic idea behind "serverless" (e.g. think to
> > > > HTTP-triggered integrations, but not only).
> > > > Focusing on serverless will bring more emphasis on push-based
> > > notifications
> > > > (webhooks, cloud events <https://cloudevents.io/>), that are rarely
> > > used in
> > > > Camel components, that prefer a poll based approach being it simpler
> to
> > > use
> > > > in classic deployments, but not so good in the cloud, where more
> > > resources
> > > > become higher direct costs for the users.
> > > >
> > > > The presence of the simplified DSL enables also experimenting on
> > > "*reduced*
> > > > subsets of Camel" implemented in languages other than Java, for
> example
> > > one
> > > > language that has a reactive approach on thread scheduling and a
> really
> > > low
> > > > memory footprint, like Go.
> > > >
> > > > But apart from this kind of experiments (that are valid IMO), the
> > "Kamel"
> > > > optimizer will have free room to choose the right platform for the
> > > > integration that the user wants to run, including, in the future,
> doing
> > > AOT
> > > > compilation using Graal/VM (less memory, faster startup) if the
> > features
> > > > (components) used in the integration are supporting it (maybe we can
> > add
> > > > AOT compilation in the roadmap for Camel 3).
> > > > A silly optimization: integrations starting from "timer:..." may be
> > > > scheduled directly with Kubernetes CronJobs, so they will consume
> > > resources
> > > > only when actually running.
> > > >
> > > > Being the final integrations lightweight and being the DSL
> > > > language-independent, we may see a increased adoption of Camel also
> as
> > > > agile integration layer for not-only-java applications (both "cloud"
> > and
> > > > "serverless" applications).
> > > >
> > > > I'm the first one that would like to work on a project ilke this.
> I've
> > > > worked on many Kubernetes/Openshift based applications and frameworks
> > in
> > > > the past years, also on operators and CRDs, and I think this way of
> > > > redesigning integrations has a lot of potential.
> > > >
> > > > Integrations will not be necessarily limited to the simplified DSL,
> but
> > > we
> > > > can add extension points for scripting and even custom libraries
> > > (although
> > > > limiting the freedom of the optimizer).
> > > >
> > > > The most important thing: it may become a great project, since it's
> > > driven
> > > > by a great community.
> > > >
> > > > So, what do you think? Is it crazy enough?
> > > >
> > > > Nicola
> > >
> > >
> >
> > --
> > Hiram Chirino
> > Engineering | Red Hat, Inc.
> > hchirino@redhat.com | fusesource.com | redhat.com
> > skype: hiramchirino | twitter: @hiramchirino
> >
>
Re: A "Kamel" crazy idea
Posted by Pontus Ullgren <ul...@gmail.com>.
I also like the idea but with some comments.
As Hiram Chirino I'm not sure YAML/JSON is the best language for this.
Perhaps a more fluent DSL like the Camel Java DSL or perhaps something like
Ballerina language would be better suited ?
Also in my experience even simple integrations, that is simple real world
integration and not just hello world demos, requires
you to add one or more Java class or scripting in addition to the core
components. So for it to be useful there must be some way to add custom
code in some way for aggregation/enrichment strategies.
If you go with a GO implementation I would also like some fallback for
using Java since I find it unlikely that the existing base of components
will be ported any time soon.
As for the name as other I like the "Kamel" name however a being a newborn
kotlin fan I know of one pre-existing project with that name
that is close enough to allow for potential confusion.
https://github.com/topicusoverheid/kamel
Looking forward to see the result.
// Pontus
On Mon, 16 Jul 2018 at 15:15 Hiram Chirino <hi...@hiramchirino.com> wrote:
> Love the idea.
>
> Personally, I'd keep using the existing Camel XML DSL if possible. You can
> still embed it in the CRD. The operator that deploys it could inspect it
> and figure whats the most optimized runtime that can support the DSL.
> Perhaps if it's only using the restricted set of camel components supported
> by camel-go (https://github.com/lburgazzoli/camel-go) then is uses that.
> Otherwise it falls back to using camel spring boot.
>
> For that to work I think we need the different runtime implementations to
> provide a way to ask them: 'hey do you support running this camel route?'
> Not a trivial thing to respond to, it might require a build step in there
> for traditional Camel runtimes.
>
>
> On Fri, Jul 13, 2018 at 4:32 AM Antonin Stefanutti <an...@stefanutti.fr>
> wrote:
>
> > Hi Nicola,
> >
> > I love the idea.
> >
> > I just wonder whether YAML/JSON is an expressive enough format in the
> long
> > term. But as you’ve mentioned, starting simple would enable experimenting
> > some very interesting / promising optimisations. So it seems worth taking
> > that path, instead of trying to embed a complex DSL or the existing XML
> DSL
> > into the CRD.
> >
> > Definitely +1
> >
> > > On 13 Jul 2018, at 01:30, Nicola Ferraro <ni...@gmail.com> wrote:
> > >
> > > Hi Cameleers,
> > > it's now passed some time since I started thinking about a new project
> > that
> > > we can begin here at Apache Camel, and I'd like to have your opinion.
> > >
> > > We've already been targeting cloud-native applications with Camel,
> > > especially on top of Kubernetes, that is becoming "the standard" cloud
> > > platform. But writing a Camel integration and running it on Kubernetes
> > > requires some effort: choosing the base platform (spring-boot, karaf,
> > > simple main?), adding health checks (actuator?), packaging a docker
> image
> > > and creating the Kubernetes resources (fabric8-maven-plugin, helm?),
> > > publishing the image on a docker registry, then finally deploying the
> > > resources on a Kubernetes cluster.
> > >
> > > The resulting integration container is then far from being optimal
> from a
> > > resource consumption point of view: it is likely that a Camel
> Spring-Boot
> > > application will require at least 200MB of RAM and also some CPU shares
> > > because of polling threads used by many components.
> > >
> > > In case people use a CI/CD pipeline, it will take also a long time to
> get
> > > from a code update to having a Kubernetes POD up and running.
> > > Apart from compilation and image push/pull time, also startup time is
> > often
> > > ~10 seconds for Camel + Spring-Boot in a container with standard limits
> > on
> > > resources, making it difficult to propose this combination for
> > "serverless
> > > integration" (this term is becoming increasingly more popular).
> > >
> > > So, my proposal is to start to investigate a "more cloud-native"
> approach
> > > to integration: *making Camel integrations first-class citizens in
> > > Kubernetes, and making them super fast and lightweight.*
> > >
> > > We can base the project on Kubernetes Custom Resource Definitions (CRD)
> > > <
> >
> https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/
> > >,
> > > for example a Integration CRD and have a Kubernetes "operator"
> > > <https://coreos.com/operators/> taking care of:
> > > - Optimizing the integration that we want to run
> > > - Packaging in a container
> > > - Running it on Kubernetes
> > > - Managing its entire lifecycle
> > >
> > > A Kubernetes-native integration may look like:
> > >
> > > -------------------
> > > kind: "Integration"
> > > apiVersion: "camel.apache.org/v1alpha1"
> > > metadata:
> > > name: "example"
> > > spec:
> > > replicas: 1
> > > routes:
> > > - id: timer
> > > route:
> > > - type: endpoint
> > > uri: timer:tick
> > > - type: endpoint
> > > uri: log:info
> > > -------------------
> > >
> > > For those who are not familiar with Kubernetes resources, this kind of
> > > YAML/JSON resource definitions are really common.
> > > The example route is embedded in the Kubernetes resource declaration
> and
> > > follows a basic "flow DSL". We may start from a basic one and evolve it
> > as
> > > new requirements arrive from the community.
> > >
> > > I've made a very simple (but working) POC here:
> > > https://github.com/nicolaferraro/integration-operator.
> > >
> > > This idea of a "Cloud-Native Camel" on Kubernetes (project codename can
> > be "
> > > *Kamel*", if you like it :D), will be an enabler for a lot of nice
> > features.
> > >
> > > For example, we can propose "Kamel" as "ideal" platform for "serverless
> > > integration" (I see many people reinventing the wheel out there): the
> > > operator can reduce resource consumption of a single integration by
> > > optimizing the runtime and also pause/resume integrations when they are
> > not
> > > used, that is the basic idea behind "serverless" (e.g. think to
> > > HTTP-triggered integrations, but not only).
> > > Focusing on serverless will bring more emphasis on push-based
> > notifications
> > > (webhooks, cloud events <https://cloudevents.io/>), that are rarely
> > used in
> > > Camel components, that prefer a poll based approach being it simpler to
> > use
> > > in classic deployments, but not so good in the cloud, where more
> > resources
> > > become higher direct costs for the users.
> > >
> > > The presence of the simplified DSL enables also experimenting on
> > "*reduced*
> > > subsets of Camel" implemented in languages other than Java, for example
> > one
> > > language that has a reactive approach on thread scheduling and a really
> > low
> > > memory footprint, like Go.
> > >
> > > But apart from this kind of experiments (that are valid IMO), the
> "Kamel"
> > > optimizer will have free room to choose the right platform for the
> > > integration that the user wants to run, including, in the future, doing
> > AOT
> > > compilation using Graal/VM (less memory, faster startup) if the
> features
> > > (components) used in the integration are supporting it (maybe we can
> add
> > > AOT compilation in the roadmap for Camel 3).
> > > A silly optimization: integrations starting from "timer:..." may be
> > > scheduled directly with Kubernetes CronJobs, so they will consume
> > resources
> > > only when actually running.
> > >
> > > Being the final integrations lightweight and being the DSL
> > > language-independent, we may see a increased adoption of Camel also as
> > > agile integration layer for not-only-java applications (both "cloud"
> and
> > > "serverless" applications).
> > >
> > > I'm the first one that would like to work on a project ilke this. I've
> > > worked on many Kubernetes/Openshift based applications and frameworks
> in
> > > the past years, also on operators and CRDs, and I think this way of
> > > redesigning integrations has a lot of potential.
> > >
> > > Integrations will not be necessarily limited to the simplified DSL, but
> > we
> > > can add extension points for scripting and even custom libraries
> > (although
> > > limiting the freedom of the optimizer).
> > >
> > > The most important thing: it may become a great project, since it's
> > driven
> > > by a great community.
> > >
> > > So, what do you think? Is it crazy enough?
> > >
> > > Nicola
> >
> >
>
> --
> Hiram Chirino
> Engineering | Red Hat, Inc.
> hchirino@redhat.com | fusesource.com | redhat.com
> skype: hiramchirino | twitter: @hiramchirino
>
Re: A "Kamel" crazy idea
Posted by Hiram Chirino <hi...@hiramchirino.com>.
Love the idea.
Personally, I'd keep using the existing Camel XML DSL if possible. You can
still embed it in the CRD. The operator that deploys it could inspect it
and figure whats the most optimized runtime that can support the DSL.
Perhaps if it's only using the restricted set of camel components supported
by camel-go (https://github.com/lburgazzoli/camel-go) then is uses that.
Otherwise it falls back to using camel spring boot.
For that to work I think we need the different runtime implementations to
provide a way to ask them: 'hey do you support running this camel route?'
Not a trivial thing to respond to, it might require a build step in there
for traditional Camel runtimes.
On Fri, Jul 13, 2018 at 4:32 AM Antonin Stefanutti <an...@stefanutti.fr>
wrote:
> Hi Nicola,
>
> I love the idea.
>
> I just wonder whether YAML/JSON is an expressive enough format in the long
> term. But as you’ve mentioned, starting simple would enable experimenting
> some very interesting / promising optimisations. So it seems worth taking
> that path, instead of trying to embed a complex DSL or the existing XML DSL
> into the CRD.
>
> Definitely +1
>
> > On 13 Jul 2018, at 01:30, Nicola Ferraro <ni...@gmail.com> wrote:
> >
> > Hi Cameleers,
> > it's now passed some time since I started thinking about a new project
> that
> > we can begin here at Apache Camel, and I'd like to have your opinion.
> >
> > We've already been targeting cloud-native applications with Camel,
> > especially on top of Kubernetes, that is becoming "the standard" cloud
> > platform. But writing a Camel integration and running it on Kubernetes
> > requires some effort: choosing the base platform (spring-boot, karaf,
> > simple main?), adding health checks (actuator?), packaging a docker image
> > and creating the Kubernetes resources (fabric8-maven-plugin, helm?),
> > publishing the image on a docker registry, then finally deploying the
> > resources on a Kubernetes cluster.
> >
> > The resulting integration container is then far from being optimal from a
> > resource consumption point of view: it is likely that a Camel Spring-Boot
> > application will require at least 200MB of RAM and also some CPU shares
> > because of polling threads used by many components.
> >
> > In case people use a CI/CD pipeline, it will take also a long time to get
> > from a code update to having a Kubernetes POD up and running.
> > Apart from compilation and image push/pull time, also startup time is
> often
> > ~10 seconds for Camel + Spring-Boot in a container with standard limits
> on
> > resources, making it difficult to propose this combination for
> "serverless
> > integration" (this term is becoming increasingly more popular).
> >
> > So, my proposal is to start to investigate a "more cloud-native" approach
> > to integration: *making Camel integrations first-class citizens in
> > Kubernetes, and making them super fast and lightweight.*
> >
> > We can base the project on Kubernetes Custom Resource Definitions (CRD)
> > <
> https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/
> >,
> > for example a Integration CRD and have a Kubernetes "operator"
> > <https://coreos.com/operators/> taking care of:
> > - Optimizing the integration that we want to run
> > - Packaging in a container
> > - Running it on Kubernetes
> > - Managing its entire lifecycle
> >
> > A Kubernetes-native integration may look like:
> >
> > -------------------
> > kind: "Integration"
> > apiVersion: "camel.apache.org/v1alpha1"
> > metadata:
> > name: "example"
> > spec:
> > replicas: 1
> > routes:
> > - id: timer
> > route:
> > - type: endpoint
> > uri: timer:tick
> > - type: endpoint
> > uri: log:info
> > -------------------
> >
> > For those who are not familiar with Kubernetes resources, this kind of
> > YAML/JSON resource definitions are really common.
> > The example route is embedded in the Kubernetes resource declaration and
> > follows a basic "flow DSL". We may start from a basic one and evolve it
> as
> > new requirements arrive from the community.
> >
> > I've made a very simple (but working) POC here:
> > https://github.com/nicolaferraro/integration-operator.
> >
> > This idea of a "Cloud-Native Camel" on Kubernetes (project codename can
> be "
> > *Kamel*", if you like it :D), will be an enabler for a lot of nice
> features.
> >
> > For example, we can propose "Kamel" as "ideal" platform for "serverless
> > integration" (I see many people reinventing the wheel out there): the
> > operator can reduce resource consumption of a single integration by
> > optimizing the runtime and also pause/resume integrations when they are
> not
> > used, that is the basic idea behind "serverless" (e.g. think to
> > HTTP-triggered integrations, but not only).
> > Focusing on serverless will bring more emphasis on push-based
> notifications
> > (webhooks, cloud events <https://cloudevents.io/>), that are rarely
> used in
> > Camel components, that prefer a poll based approach being it simpler to
> use
> > in classic deployments, but not so good in the cloud, where more
> resources
> > become higher direct costs for the users.
> >
> > The presence of the simplified DSL enables also experimenting on
> "*reduced*
> > subsets of Camel" implemented in languages other than Java, for example
> one
> > language that has a reactive approach on thread scheduling and a really
> low
> > memory footprint, like Go.
> >
> > But apart from this kind of experiments (that are valid IMO), the "Kamel"
> > optimizer will have free room to choose the right platform for the
> > integration that the user wants to run, including, in the future, doing
> AOT
> > compilation using Graal/VM (less memory, faster startup) if the features
> > (components) used in the integration are supporting it (maybe we can add
> > AOT compilation in the roadmap for Camel 3).
> > A silly optimization: integrations starting from "timer:..." may be
> > scheduled directly with Kubernetes CronJobs, so they will consume
> resources
> > only when actually running.
> >
> > Being the final integrations lightweight and being the DSL
> > language-independent, we may see a increased adoption of Camel also as
> > agile integration layer for not-only-java applications (both "cloud" and
> > "serverless" applications).
> >
> > I'm the first one that would like to work on a project ilke this. I've
> > worked on many Kubernetes/Openshift based applications and frameworks in
> > the past years, also on operators and CRDs, and I think this way of
> > redesigning integrations has a lot of potential.
> >
> > Integrations will not be necessarily limited to the simplified DSL, but
> we
> > can add extension points for scripting and even custom libraries
> (although
> > limiting the freedom of the optimizer).
> >
> > The most important thing: it may become a great project, since it's
> driven
> > by a great community.
> >
> > So, what do you think? Is it crazy enough?
> >
> > Nicola
>
>
--
Hiram Chirino
Engineering | Red Hat, Inc.
hchirino@redhat.com | fusesource.com | redhat.com
skype: hiramchirino | twitter: @hiramchirino
Re: A "Kamel" crazy idea
Posted by Ricardo Zanini <ri...@gmail.com>.
Hi!
Awesome idea. Seeing this being viable in a platform like Apache OpenWhisk
would be great too. I keep hearing customers asking about lightweight
containers to run Camel context. :)
+++1
Zanini
On Fri, Jul 13, 2018 at 5:31 AM, Antonin Stefanutti <an...@stefanutti.fr>
wrote:
> Hi Nicola,
>
> I love the idea.
>
> I just wonder whether YAML/JSON is an expressive enough format in the long
> term. But as you’ve mentioned, starting simple would enable experimenting
> some very interesting / promising optimisations. So it seems worth taking
> that path, instead of trying to embed a complex DSL or the existing XML DSL
> into the CRD.
>
> Definitely +1
>
> > On 13 Jul 2018, at 01:30, Nicola Ferraro <ni...@gmail.com> wrote:
> >
> > Hi Cameleers,
> > it's now passed some time since I started thinking about a new project
> that
> > we can begin here at Apache Camel, and I'd like to have your opinion.
> >
> > We've already been targeting cloud-native applications with Camel,
> > especially on top of Kubernetes, that is becoming "the standard" cloud
> > platform. But writing a Camel integration and running it on Kubernetes
> > requires some effort: choosing the base platform (spring-boot, karaf,
> > simple main?), adding health checks (actuator?), packaging a docker image
> > and creating the Kubernetes resources (fabric8-maven-plugin, helm?),
> > publishing the image on a docker registry, then finally deploying the
> > resources on a Kubernetes cluster.
> >
> > The resulting integration container is then far from being optimal from a
> > resource consumption point of view: it is likely that a Camel Spring-Boot
> > application will require at least 200MB of RAM and also some CPU shares
> > because of polling threads used by many components.
> >
> > In case people use a CI/CD pipeline, it will take also a long time to get
> > from a code update to having a Kubernetes POD up and running.
> > Apart from compilation and image push/pull time, also startup time is
> often
> > ~10 seconds for Camel + Spring-Boot in a container with standard limits
> on
> > resources, making it difficult to propose this combination for
> "serverless
> > integration" (this term is becoming increasingly more popular).
> >
> > So, my proposal is to start to investigate a "more cloud-native" approach
> > to integration: *making Camel integrations first-class citizens in
> > Kubernetes, and making them super fast and lightweight.*
> >
> > We can base the project on Kubernetes Custom Resource Definitions (CRD)
> > <https://kubernetes.io/docs/concepts/extend-kubernetes/
> api-extension/custom-resources/>,
> > for example a Integration CRD and have a Kubernetes "operator"
> > <https://coreos.com/operators/> taking care of:
> > - Optimizing the integration that we want to run
> > - Packaging in a container
> > - Running it on Kubernetes
> > - Managing its entire lifecycle
> >
> > A Kubernetes-native integration may look like:
> >
> > -------------------
> > kind: "Integration"
> > apiVersion: "camel.apache.org/v1alpha1"
> > metadata:
> > name: "example"
> > spec:
> > replicas: 1
> > routes:
> > - id: timer
> > route:
> > - type: endpoint
> > uri: timer:tick
> > - type: endpoint
> > uri: log:info
> > -------------------
> >
> > For those who are not familiar with Kubernetes resources, this kind of
> > YAML/JSON resource definitions are really common.
> > The example route is embedded in the Kubernetes resource declaration and
> > follows a basic "flow DSL". We may start from a basic one and evolve it
> as
> > new requirements arrive from the community.
> >
> > I've made a very simple (but working) POC here:
> > https://github.com/nicolaferraro/integration-operator.
> >
> > This idea of a "Cloud-Native Camel" on Kubernetes (project codename can
> be "
> > *Kamel*", if you like it :D), will be an enabler for a lot of nice
> features.
> >
> > For example, we can propose "Kamel" as "ideal" platform for "serverless
> > integration" (I see many people reinventing the wheel out there): the
> > operator can reduce resource consumption of a single integration by
> > optimizing the runtime and also pause/resume integrations when they are
> not
> > used, that is the basic idea behind "serverless" (e.g. think to
> > HTTP-triggered integrations, but not only).
> > Focusing on serverless will bring more emphasis on push-based
> notifications
> > (webhooks, cloud events <https://cloudevents.io/>), that are rarely
> used in
> > Camel components, that prefer a poll based approach being it simpler to
> use
> > in classic deployments, but not so good in the cloud, where more
> resources
> > become higher direct costs for the users.
> >
> > The presence of the simplified DSL enables also experimenting on
> "*reduced*
> > subsets of Camel" implemented in languages other than Java, for example
> one
> > language that has a reactive approach on thread scheduling and a really
> low
> > memory footprint, like Go.
> >
> > But apart from this kind of experiments (that are valid IMO), the "Kamel"
> > optimizer will have free room to choose the right platform for the
> > integration that the user wants to run, including, in the future, doing
> AOT
> > compilation using Graal/VM (less memory, faster startup) if the features
> > (components) used in the integration are supporting it (maybe we can add
> > AOT compilation in the roadmap for Camel 3).
> > A silly optimization: integrations starting from "timer:..." may be
> > scheduled directly with Kubernetes CronJobs, so they will consume
> resources
> > only when actually running.
> >
> > Being the final integrations lightweight and being the DSL
> > language-independent, we may see a increased adoption of Camel also as
> > agile integration layer for not-only-java applications (both "cloud" and
> > "serverless" applications).
> >
> > I'm the first one that would like to work on a project ilke this. I've
> > worked on many Kubernetes/Openshift based applications and frameworks in
> > the past years, also on operators and CRDs, and I think this way of
> > redesigning integrations has a lot of potential.
> >
> > Integrations will not be necessarily limited to the simplified DSL, but
> we
> > can add extension points for scripting and even custom libraries
> (although
> > limiting the freedom of the optimizer).
> >
> > The most important thing: it may become a great project, since it's
> driven
> > by a great community.
> >
> > So, what do you think? Is it crazy enough?
> >
> > Nicola
>
>
Re: A "Kamel" crazy idea
Posted by Antonin Stefanutti <an...@stefanutti.fr>.
Hi Nicola,
I love the idea.
I just wonder whether YAML/JSON is an expressive enough format in the long term. But as you’ve mentioned, starting simple would enable experimenting some very interesting / promising optimisations. So it seems worth taking that path, instead of trying to embed a complex DSL or the existing XML DSL into the CRD.
Definitely +1
> On 13 Jul 2018, at 01:30, Nicola Ferraro <ni...@gmail.com> wrote:
>
> Hi Cameleers,
> it's now passed some time since I started thinking about a new project that
> we can begin here at Apache Camel, and I'd like to have your opinion.
>
> We've already been targeting cloud-native applications with Camel,
> especially on top of Kubernetes, that is becoming "the standard" cloud
> platform. But writing a Camel integration and running it on Kubernetes
> requires some effort: choosing the base platform (spring-boot, karaf,
> simple main?), adding health checks (actuator?), packaging a docker image
> and creating the Kubernetes resources (fabric8-maven-plugin, helm?),
> publishing the image on a docker registry, then finally deploying the
> resources on a Kubernetes cluster.
>
> The resulting integration container is then far from being optimal from a
> resource consumption point of view: it is likely that a Camel Spring-Boot
> application will require at least 200MB of RAM and also some CPU shares
> because of polling threads used by many components.
>
> In case people use a CI/CD pipeline, it will take also a long time to get
> from a code update to having a Kubernetes POD up and running.
> Apart from compilation and image push/pull time, also startup time is often
> ~10 seconds for Camel + Spring-Boot in a container with standard limits on
> resources, making it difficult to propose this combination for "serverless
> integration" (this term is becoming increasingly more popular).
>
> So, my proposal is to start to investigate a "more cloud-native" approach
> to integration: *making Camel integrations first-class citizens in
> Kubernetes, and making them super fast and lightweight.*
>
> We can base the project on Kubernetes Custom Resource Definitions (CRD)
> <https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/>,
> for example a Integration CRD and have a Kubernetes "operator"
> <https://coreos.com/operators/> taking care of:
> - Optimizing the integration that we want to run
> - Packaging in a container
> - Running it on Kubernetes
> - Managing its entire lifecycle
>
> A Kubernetes-native integration may look like:
>
> -------------------
> kind: "Integration"
> apiVersion: "camel.apache.org/v1alpha1"
> metadata:
> name: "example"
> spec:
> replicas: 1
> routes:
> - id: timer
> route:
> - type: endpoint
> uri: timer:tick
> - type: endpoint
> uri: log:info
> -------------------
>
> For those who are not familiar with Kubernetes resources, this kind of
> YAML/JSON resource definitions are really common.
> The example route is embedded in the Kubernetes resource declaration and
> follows a basic "flow DSL". We may start from a basic one and evolve it as
> new requirements arrive from the community.
>
> I've made a very simple (but working) POC here:
> https://github.com/nicolaferraro/integration-operator.
>
> This idea of a "Cloud-Native Camel" on Kubernetes (project codename can be "
> *Kamel*", if you like it :D), will be an enabler for a lot of nice features.
>
> For example, we can propose "Kamel" as "ideal" platform for "serverless
> integration" (I see many people reinventing the wheel out there): the
> operator can reduce resource consumption of a single integration by
> optimizing the runtime and also pause/resume integrations when they are not
> used, that is the basic idea behind "serverless" (e.g. think to
> HTTP-triggered integrations, but not only).
> Focusing on serverless will bring more emphasis on push-based notifications
> (webhooks, cloud events <https://cloudevents.io/>), that are rarely used in
> Camel components, that prefer a poll based approach being it simpler to use
> in classic deployments, but not so good in the cloud, where more resources
> become higher direct costs for the users.
>
> The presence of the simplified DSL enables also experimenting on "*reduced*
> subsets of Camel" implemented in languages other than Java, for example one
> language that has a reactive approach on thread scheduling and a really low
> memory footprint, like Go.
>
> But apart from this kind of experiments (that are valid IMO), the "Kamel"
> optimizer will have free room to choose the right platform for the
> integration that the user wants to run, including, in the future, doing AOT
> compilation using Graal/VM (less memory, faster startup) if the features
> (components) used in the integration are supporting it (maybe we can add
> AOT compilation in the roadmap for Camel 3).
> A silly optimization: integrations starting from "timer:..." may be
> scheduled directly with Kubernetes CronJobs, so they will consume resources
> only when actually running.
>
> Being the final integrations lightweight and being the DSL
> language-independent, we may see a increased adoption of Camel also as
> agile integration layer for not-only-java applications (both "cloud" and
> "serverless" applications).
>
> I'm the first one that would like to work on a project ilke this. I've
> worked on many Kubernetes/Openshift based applications and frameworks in
> the past years, also on operators and CRDs, and I think this way of
> redesigning integrations has a lot of potential.
>
> Integrations will not be necessarily limited to the simplified DSL, but we
> can add extension points for scripting and even custom libraries (although
> limiting the freedom of the optimizer).
>
> The most important thing: it may become a great project, since it's driven
> by a great community.
>
> So, what do you think? Is it crazy enough?
>
> Nicola
Re: A "Kamel" crazy idea
Posted by Johan Edstrom <se...@gmail.com>.
I find it super +1 - If not only for your awesome narrative.
> On Jul 12, 2018, at 5:30 PM, Nicola Ferraro <ni...@gmail.com> wrote:
>
> Hi Cameleers,
> it's now passed some time since I started thinking about a new project that
> we can begin here at Apache Camel, and I'd like to have your opinion.
>
> We've already been targeting cloud-native applications with Camel,
> especially on top of Kubernetes, that is becoming "the standard" cloud
> platform. But writing a Camel integration and running it on Kubernetes
> requires some effort: choosing the base platform (spring-boot, karaf,
> simple main?), adding health checks (actuator?), packaging a docker image
> and creating the Kubernetes resources (fabric8-maven-plugin, helm?),
> publishing the image on a docker registry, then finally deploying the
> resources on a Kubernetes cluster.
>
> The resulting integration container is then far from being optimal from a
> resource consumption point of view: it is likely that a Camel Spring-Boot
> application will require at least 200MB of RAM and also some CPU shares
> because of polling threads used by many components.
>
> In case people use a CI/CD pipeline, it will take also a long time to get
> from a code update to having a Kubernetes POD up and running.
> Apart from compilation and image push/pull time, also startup time is often
> ~10 seconds for Camel + Spring-Boot in a container with standard limits on
> resources, making it difficult to propose this combination for "serverless
> integration" (this term is becoming increasingly more popular).
>
> So, my proposal is to start to investigate a "more cloud-native" approach
> to integration: *making Camel integrations first-class citizens in
> Kubernetes, and making them super fast and lightweight.*
>
> We can base the project on Kubernetes Custom Resource Definitions (CRD)
> <https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/>,
> for example a Integration CRD and have a Kubernetes "operator"
> <https://coreos.com/operators/> taking care of:
> - Optimizing the integration that we want to run
> - Packaging in a container
> - Running it on Kubernetes
> - Managing its entire lifecycle
>
> A Kubernetes-native integration may look like:
>
> -------------------
> kind: "Integration"
> apiVersion: "camel.apache.org/v1alpha1"
> metadata:
> name: "example"
> spec:
> replicas: 1
> routes:
> - id: timer
> route:
> - type: endpoint
> uri: timer:tick
> - type: endpoint
> uri: log:info
> -------------------
>
> For those who are not familiar with Kubernetes resources, this kind of
> YAML/JSON resource definitions are really common.
> The example route is embedded in the Kubernetes resource declaration and
> follows a basic "flow DSL". We may start from a basic one and evolve it as
> new requirements arrive from the community.
>
> I've made a very simple (but working) POC here:
> https://github.com/nicolaferraro/integration-operator.
>
> This idea of a "Cloud-Native Camel" on Kubernetes (project codename can be "
> *Kamel*", if you like it :D), will be an enabler for a lot of nice features.
>
> For example, we can propose "Kamel" as "ideal" platform for "serverless
> integration" (I see many people reinventing the wheel out there): the
> operator can reduce resource consumption of a single integration by
> optimizing the runtime and also pause/resume integrations when they are not
> used, that is the basic idea behind "serverless" (e.g. think to
> HTTP-triggered integrations, but not only).
> Focusing on serverless will bring more emphasis on push-based notifications
> (webhooks, cloud events <https://cloudevents.io/>), that are rarely used in
> Camel components, that prefer a poll based approach being it simpler to use
> in classic deployments, but not so good in the cloud, where more resources
> become higher direct costs for the users.
>
> The presence of the simplified DSL enables also experimenting on "*reduced*
> subsets of Camel" implemented in languages other than Java, for example one
> language that has a reactive approach on thread scheduling and a really low
> memory footprint, like Go.
>
> But apart from this kind of experiments (that are valid IMO), the "Kamel"
> optimizer will have free room to choose the right platform for the
> integration that the user wants to run, including, in the future, doing AOT
> compilation using Graal/VM (less memory, faster startup) if the features
> (components) used in the integration are supporting it (maybe we can add
> AOT compilation in the roadmap for Camel 3).
> A silly optimization: integrations starting from "timer:..." may be
> scheduled directly with Kubernetes CronJobs, so they will consume resources
> only when actually running.
>
> Being the final integrations lightweight and being the DSL
> language-independent, we may see a increased adoption of Camel also as
> agile integration layer for not-only-java applications (both "cloud" and
> "serverless" applications).
>
> I'm the first one that would like to work on a project ilke this. I've
> worked on many Kubernetes/Openshift based applications and frameworks in
> the past years, also on operators and CRDs, and I think this way of
> redesigning integrations has a lot of potential.
>
> Integrations will not be necessarily limited to the simplified DSL, but we
> can add extension points for scripting and even custom libraries (although
> limiting the freedom of the optimizer).
>
> The most important thing: it may become a great project, since it's driven
> by a great community.
>
> So, what do you think? Is it crazy enough?
>
> Nicola
Re: A "Kamel" crazy idea
Posted by Onder SEZGIN <on...@gmail.com>.
Hi,
This can make Kamel language, dsl agnostic and will give more room to
integrate faster.
Definitely +1.
On Fri, Jul 13, 2018 at 3:54 AM Willem Jiang <wi...@gmail.com> wrote:
> Yeah, it's a really good idea to combine the K8S with Camel.
> In my mind if we want to host a camel application on the cloud, it could
> be first step that we can run the camel engine on demand in K8S.
>
> So I really like the project idea *making Camel integrations first-class
> citizens in Kubernetes, and making them super fast and lightweight.*
>
> I think the Apache Openwhisk is doing the same thing here[1].
>
> [1]https://github.com/apache/incubator-openwhisk-deploy-kube
>
>
>
>
> Willem Jiang
>
> Twitter: willemjiang
> Weibo: 姜宁willem
>
> On Fri, Jul 13, 2018 at 8:31 AM, Johan Edstrom <se...@gmail.com> wrote:
>
> > Jeff Genender, I and James Carman long ago were tossing
> > around the idea of Ibex, it would have been a Scala based AKKA
> > eco system for putting in routes. You’d just say run in this namespace,
> > conform
> > to these Actor roles and we compose the tree for you.
> >
> > What you propose is a bit easier and more tangible to attract space
> > for Camel as an engine to well, keep open source in “I can’t even write a
> > main class”
> >
> > :)
> >
> > > On Jul 12, 2018, at 5:30 PM, Nicola Ferraro <ni...@gmail.com>
> > wrote:
> > >
> > > Hi Cameleers,
> > > it's now passed some time since I started thinking about a new project
> > that
> > > we can begin here at Apache Camel, and I'd like to have your opinion.
> > >
> > > We've already been targeting cloud-native applications with Camel,
> > > especially on top of Kubernetes, that is becoming "the standard" cloud
> > > platform. But writing a Camel integration and running it on Kubernetes
> > > requires some effort: choosing the base platform (spring-boot, karaf,
> > > simple main?), adding health checks (actuator?), packaging a docker
> image
> > > and creating the Kubernetes resources (fabric8-maven-plugin, helm?),
> > > publishing the image on a docker registry, then finally deploying the
> > > resources on a Kubernetes cluster.
> > >
> > > The resulting integration container is then far from being optimal
> from a
> > > resource consumption point of view: it is likely that a Camel
> Spring-Boot
> > > application will require at least 200MB of RAM and also some CPU shares
> > > because of polling threads used by many components.
> > >
> > > In case people use a CI/CD pipeline, it will take also a long time to
> get
> > > from a code update to having a Kubernetes POD up and running.
> > > Apart from compilation and image push/pull time, also startup time is
> > often
> > > ~10 seconds for Camel + Spring-Boot in a container with standard limits
> > on
> > > resources, making it difficult to propose this combination for
> > "serverless
> > > integration" (this term is becoming increasingly more popular).
> > >
> > > So, my proposal is to start to investigate a "more cloud-native"
> approach
> > > to integration: *making Camel integrations first-class citizens in
> > > Kubernetes, and making them super fast and lightweight.*
> > >
> > > We can base the project on Kubernetes Custom Resource Definitions (CRD)
> > > <https://kubernetes.io/docs/concepts/extend-kubernetes/
> > api-extension/custom-resources/>,
> > > for example a Integration CRD and have a Kubernetes "operator"
> > > <https://coreos.com/operators/> taking care of:
> > > - Optimizing the integration that we want to run
> > > - Packaging in a container
> > > - Running it on Kubernetes
> > > - Managing its entire lifecycle
> > >
> > > A Kubernetes-native integration may look like:
> > >
> > > -------------------
> > > kind: "Integration"
> > > apiVersion: "camel.apache.org/v1alpha1"
> > > metadata:
> > > name: "example"
> > > spec:
> > > replicas: 1
> > > routes:
> > > - id: timer
> > > route:
> > > - type: endpoint
> > > uri: timer:tick
> > > - type: endpoint
> > > uri: log:info
> > > -------------------
> > >
> > > For those who are not familiar with Kubernetes resources, this kind of
> > > YAML/JSON resource definitions are really common.
> > > The example route is embedded in the Kubernetes resource declaration
> and
> > > follows a basic "flow DSL". We may start from a basic one and evolve it
> > as
> > > new requirements arrive from the community.
> > >
> > > I've made a very simple (but working) POC here:
> > > https://github.com/nicolaferraro/integration-operator.
> > >
> > > This idea of a "Cloud-Native Camel" on Kubernetes (project codename can
> > be "
> > > *Kamel*", if you like it :D), will be an enabler for a lot of nice
> > features.
> > >
> > > For example, we can propose "Kamel" as "ideal" platform for "serverless
> > > integration" (I see many people reinventing the wheel out there): the
> > > operator can reduce resource consumption of a single integration by
> > > optimizing the runtime and also pause/resume integrations when they are
> > not
> > > used, that is the basic idea behind "serverless" (e.g. think to
> > > HTTP-triggered integrations, but not only).
> > > Focusing on serverless will bring more emphasis on push-based
> > notifications
> > > (webhooks, cloud events <https://cloudevents.io/>), that are rarely
> > used in
> > > Camel components, that prefer a poll based approach being it simpler to
> > use
> > > in classic deployments, but not so good in the cloud, where more
> > resources
> > > become higher direct costs for the users.
> > >
> > > The presence of the simplified DSL enables also experimenting on
> > "*reduced*
> > > subsets of Camel" implemented in languages other than Java, for example
> > one
> > > language that has a reactive approach on thread scheduling and a really
> > low
> > > memory footprint, like Go.
> > >
> > > But apart from this kind of experiments (that are valid IMO), the
> "Kamel"
> > > optimizer will have free room to choose the right platform for the
> > > integration that the user wants to run, including, in the future, doing
> > AOT
> > > compilation using Graal/VM (less memory, faster startup) if the
> features
> > > (components) used in the integration are supporting it (maybe we can
> add
> > > AOT compilation in the roadmap for Camel 3).
> > > A silly optimization: integrations starting from "timer:..." may be
> > > scheduled directly with Kubernetes CronJobs, so they will consume
> > resources
> > > only when actually running.
> > >
> > > Being the final integrations lightweight and being the DSL
> > > language-independent, we may see a increased adoption of Camel also as
> > > agile integration layer for not-only-java applications (both "cloud"
> and
> > > "serverless" applications).
> > >
> > > I'm the first one that would like to work on a project ilke this. I've
> > > worked on many Kubernetes/Openshift based applications and frameworks
> in
> > > the past years, also on operators and CRDs, and I think this way of
> > > redesigning integrations has a lot of potential.
> > >
> > > Integrations will not be necessarily limited to the simplified DSL, but
> > we
> > > can add extension points for scripting and even custom libraries
> > (although
> > > limiting the freedom of the optimizer).
> > >
> > > The most important thing: it may become a great project, since it's
> > driven
> > > by a great community.
> > >
> > > So, what do you think? Is it crazy enough?
> > >
> > > Nicola
> >
> >
>
Re: A "Kamel" crazy idea
Posted by Willem Jiang <wi...@gmail.com>.
Yeah, it's a really good idea to combine the K8S with Camel.
In my mind if we want to host a camel application on the cloud, it could
be first step that we can run the camel engine on demand in K8S.
So I really like the project idea *making Camel integrations first-class
citizens in Kubernetes, and making them super fast and lightweight.*
I think the Apache Openwhisk is doing the same thing here[1].
[1]https://github.com/apache/incubator-openwhisk-deploy-kube
Willem Jiang
Twitter: willemjiang
Weibo: 姜宁willem
On Fri, Jul 13, 2018 at 8:31 AM, Johan Edstrom <se...@gmail.com> wrote:
> Jeff Genender, I and James Carman long ago were tossing
> around the idea of Ibex, it would have been a Scala based AKKA
> eco system for putting in routes. You’d just say run in this namespace,
> conform
> to these Actor roles and we compose the tree for you.
>
> What you propose is a bit easier and more tangible to attract space
> for Camel as an engine to well, keep open source in “I can’t even write a
> main class”
>
> :)
>
> > On Jul 12, 2018, at 5:30 PM, Nicola Ferraro <ni...@gmail.com>
> wrote:
> >
> > Hi Cameleers,
> > it's now passed some time since I started thinking about a new project
> that
> > we can begin here at Apache Camel, and I'd like to have your opinion.
> >
> > We've already been targeting cloud-native applications with Camel,
> > especially on top of Kubernetes, that is becoming "the standard" cloud
> > platform. But writing a Camel integration and running it on Kubernetes
> > requires some effort: choosing the base platform (spring-boot, karaf,
> > simple main?), adding health checks (actuator?), packaging a docker image
> > and creating the Kubernetes resources (fabric8-maven-plugin, helm?),
> > publishing the image on a docker registry, then finally deploying the
> > resources on a Kubernetes cluster.
> >
> > The resulting integration container is then far from being optimal from a
> > resource consumption point of view: it is likely that a Camel Spring-Boot
> > application will require at least 200MB of RAM and also some CPU shares
> > because of polling threads used by many components.
> >
> > In case people use a CI/CD pipeline, it will take also a long time to get
> > from a code update to having a Kubernetes POD up and running.
> > Apart from compilation and image push/pull time, also startup time is
> often
> > ~10 seconds for Camel + Spring-Boot in a container with standard limits
> on
> > resources, making it difficult to propose this combination for
> "serverless
> > integration" (this term is becoming increasingly more popular).
> >
> > So, my proposal is to start to investigate a "more cloud-native" approach
> > to integration: *making Camel integrations first-class citizens in
> > Kubernetes, and making them super fast and lightweight.*
> >
> > We can base the project on Kubernetes Custom Resource Definitions (CRD)
> > <https://kubernetes.io/docs/concepts/extend-kubernetes/
> api-extension/custom-resources/>,
> > for example a Integration CRD and have a Kubernetes "operator"
> > <https://coreos.com/operators/> taking care of:
> > - Optimizing the integration that we want to run
> > - Packaging in a container
> > - Running it on Kubernetes
> > - Managing its entire lifecycle
> >
> > A Kubernetes-native integration may look like:
> >
> > -------------------
> > kind: "Integration"
> > apiVersion: "camel.apache.org/v1alpha1"
> > metadata:
> > name: "example"
> > spec:
> > replicas: 1
> > routes:
> > - id: timer
> > route:
> > - type: endpoint
> > uri: timer:tick
> > - type: endpoint
> > uri: log:info
> > -------------------
> >
> > For those who are not familiar with Kubernetes resources, this kind of
> > YAML/JSON resource definitions are really common.
> > The example route is embedded in the Kubernetes resource declaration and
> > follows a basic "flow DSL". We may start from a basic one and evolve it
> as
> > new requirements arrive from the community.
> >
> > I've made a very simple (but working) POC here:
> > https://github.com/nicolaferraro/integration-operator.
> >
> > This idea of a "Cloud-Native Camel" on Kubernetes (project codename can
> be "
> > *Kamel*", if you like it :D), will be an enabler for a lot of nice
> features.
> >
> > For example, we can propose "Kamel" as "ideal" platform for "serverless
> > integration" (I see many people reinventing the wheel out there): the
> > operator can reduce resource consumption of a single integration by
> > optimizing the runtime and also pause/resume integrations when they are
> not
> > used, that is the basic idea behind "serverless" (e.g. think to
> > HTTP-triggered integrations, but not only).
> > Focusing on serverless will bring more emphasis on push-based
> notifications
> > (webhooks, cloud events <https://cloudevents.io/>), that are rarely
> used in
> > Camel components, that prefer a poll based approach being it simpler to
> use
> > in classic deployments, but not so good in the cloud, where more
> resources
> > become higher direct costs for the users.
> >
> > The presence of the simplified DSL enables also experimenting on
> "*reduced*
> > subsets of Camel" implemented in languages other than Java, for example
> one
> > language that has a reactive approach on thread scheduling and a really
> low
> > memory footprint, like Go.
> >
> > But apart from this kind of experiments (that are valid IMO), the "Kamel"
> > optimizer will have free room to choose the right platform for the
> > integration that the user wants to run, including, in the future, doing
> AOT
> > compilation using Graal/VM (less memory, faster startup) if the features
> > (components) used in the integration are supporting it (maybe we can add
> > AOT compilation in the roadmap for Camel 3).
> > A silly optimization: integrations starting from "timer:..." may be
> > scheduled directly with Kubernetes CronJobs, so they will consume
> resources
> > only when actually running.
> >
> > Being the final integrations lightweight and being the DSL
> > language-independent, we may see a increased adoption of Camel also as
> > agile integration layer for not-only-java applications (both "cloud" and
> > "serverless" applications).
> >
> > I'm the first one that would like to work on a project ilke this. I've
> > worked on many Kubernetes/Openshift based applications and frameworks in
> > the past years, also on operators and CRDs, and I think this way of
> > redesigning integrations has a lot of potential.
> >
> > Integrations will not be necessarily limited to the simplified DSL, but
> we
> > can add extension points for scripting and even custom libraries
> (although
> > limiting the freedom of the optimizer).
> >
> > The most important thing: it may become a great project, since it's
> driven
> > by a great community.
> >
> > So, what do you think? Is it crazy enough?
> >
> > Nicola
>
>
Re: A "Kamel" crazy idea
Posted by Johan Edstrom <se...@gmail.com>.
Jeff Genender, I and James Carman long ago were tossing
around the idea of Ibex, it would have been a Scala based AKKA
eco system for putting in routes. You’d just say run in this namespace, conform
to these Actor roles and we compose the tree for you.
What you propose is a bit easier and more tangible to attract space
for Camel as an engine to well, keep open source in “I can’t even write a main class”
:)
> On Jul 12, 2018, at 5:30 PM, Nicola Ferraro <ni...@gmail.com> wrote:
>
> Hi Cameleers,
> it's now passed some time since I started thinking about a new project that
> we can begin here at Apache Camel, and I'd like to have your opinion.
>
> We've already been targeting cloud-native applications with Camel,
> especially on top of Kubernetes, that is becoming "the standard" cloud
> platform. But writing a Camel integration and running it on Kubernetes
> requires some effort: choosing the base platform (spring-boot, karaf,
> simple main?), adding health checks (actuator?), packaging a docker image
> and creating the Kubernetes resources (fabric8-maven-plugin, helm?),
> publishing the image on a docker registry, then finally deploying the
> resources on a Kubernetes cluster.
>
> The resulting integration container is then far from being optimal from a
> resource consumption point of view: it is likely that a Camel Spring-Boot
> application will require at least 200MB of RAM and also some CPU shares
> because of polling threads used by many components.
>
> In case people use a CI/CD pipeline, it will take also a long time to get
> from a code update to having a Kubernetes POD up and running.
> Apart from compilation and image push/pull time, also startup time is often
> ~10 seconds for Camel + Spring-Boot in a container with standard limits on
> resources, making it difficult to propose this combination for "serverless
> integration" (this term is becoming increasingly more popular).
>
> So, my proposal is to start to investigate a "more cloud-native" approach
> to integration: *making Camel integrations first-class citizens in
> Kubernetes, and making them super fast and lightweight.*
>
> We can base the project on Kubernetes Custom Resource Definitions (CRD)
> <https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/>,
> for example a Integration CRD and have a Kubernetes "operator"
> <https://coreos.com/operators/> taking care of:
> - Optimizing the integration that we want to run
> - Packaging in a container
> - Running it on Kubernetes
> - Managing its entire lifecycle
>
> A Kubernetes-native integration may look like:
>
> -------------------
> kind: "Integration"
> apiVersion: "camel.apache.org/v1alpha1"
> metadata:
> name: "example"
> spec:
> replicas: 1
> routes:
> - id: timer
> route:
> - type: endpoint
> uri: timer:tick
> - type: endpoint
> uri: log:info
> -------------------
>
> For those who are not familiar with Kubernetes resources, this kind of
> YAML/JSON resource definitions are really common.
> The example route is embedded in the Kubernetes resource declaration and
> follows a basic "flow DSL". We may start from a basic one and evolve it as
> new requirements arrive from the community.
>
> I've made a very simple (but working) POC here:
> https://github.com/nicolaferraro/integration-operator.
>
> This idea of a "Cloud-Native Camel" on Kubernetes (project codename can be "
> *Kamel*", if you like it :D), will be an enabler for a lot of nice features.
>
> For example, we can propose "Kamel" as "ideal" platform for "serverless
> integration" (I see many people reinventing the wheel out there): the
> operator can reduce resource consumption of a single integration by
> optimizing the runtime and also pause/resume integrations when they are not
> used, that is the basic idea behind "serverless" (e.g. think to
> HTTP-triggered integrations, but not only).
> Focusing on serverless will bring more emphasis on push-based notifications
> (webhooks, cloud events <https://cloudevents.io/>), that are rarely used in
> Camel components, that prefer a poll based approach being it simpler to use
> in classic deployments, but not so good in the cloud, where more resources
> become higher direct costs for the users.
>
> The presence of the simplified DSL enables also experimenting on "*reduced*
> subsets of Camel" implemented in languages other than Java, for example one
> language that has a reactive approach on thread scheduling and a really low
> memory footprint, like Go.
>
> But apart from this kind of experiments (that are valid IMO), the "Kamel"
> optimizer will have free room to choose the right platform for the
> integration that the user wants to run, including, in the future, doing AOT
> compilation using Graal/VM (less memory, faster startup) if the features
> (components) used in the integration are supporting it (maybe we can add
> AOT compilation in the roadmap for Camel 3).
> A silly optimization: integrations starting from "timer:..." may be
> scheduled directly with Kubernetes CronJobs, so they will consume resources
> only when actually running.
>
> Being the final integrations lightweight and being the DSL
> language-independent, we may see a increased adoption of Camel also as
> agile integration layer for not-only-java applications (both "cloud" and
> "serverless" applications).
>
> I'm the first one that would like to work on a project ilke this. I've
> worked on many Kubernetes/Openshift based applications and frameworks in
> the past years, also on operators and CRDs, and I think this way of
> redesigning integrations has a lot of potential.
>
> Integrations will not be necessarily limited to the simplified DSL, but we
> can add extension points for scripting and even custom libraries (although
> limiting the freedom of the optimizer).
>
> The most important thing: it may become a great project, since it's driven
> by a great community.
>
> So, what do you think? Is it crazy enough?
>
> Nicola