Re: A "Kamel" crazy idea

[Guillaume Nodet <gn...@apache.org>]

I've pushed a branch with graalvm experiments.
   https://github.com/apache/camel/tree/graalvm
In order to build the project, you need to use the GraalVM jdk and use the
following ~/.m2/toolchains.xml adapted to your path:

<?xml version="1.0" encoding="UTF8"?>

<toolchains>

  <toolchain>

    <type>jdk</type>

    <provides>

      <version>1.0.0-rc4</version>

      <vendor>oracle</vendor>

      <id>graalvm-ee-1.0.0-rc4</id>

    </provides>

    <configuration>


<jdkHome>/Users/gnodet/Downloads/graalvm-ee-1.0.0-rc4/Contents/Home/</jdkHome>

    </configuration>

  </toolchain>

</toolchains>

Build and run:
 gnodet•camel*(*graalvm*)*» cd platforms/graalvm
 gnodet•camel/platforms/graalvm*(*graalvm*)*» mvn install
...
 gnodet•camel/platforms/graalvm*(*graalvm*)*»
./example/target/org.apache.camel.graalvm.main

[main] INFO org.apache.camel.graalvm.FastCamelContext - Apache Camel
2.23.0-SNAPSHOT (CamelContext: camel-1) is starting

[main] INFO org.apache.camel.graalvm.FastCamelContext - StreamCaching is
not in use. If using streams then its recommended to enable stream caching.
See more details at http://camel.apache.org/stream-caching.html

[main] INFO org.apache.camel.graalvm.FastCamelContext - Total 0 routes, of
which 0 are started

[main] INFO org.apache.camel.graalvm.FastCamelContext - Apache Camel
2.23.0-SNAPSHOT (CamelContext: camel-1) started in 0.006 seconds

 Guillaume

Le mar. 31 juil. 2018 à 09:44, Guillaume Nodet <gn...@apache.org> a écrit :

>
>
> Le lun. 30 juil. 2018 à 17:49, Nicola Ferraro <ni...@gmail.com> a
> écrit :
>
>> Hi Cameleers,
>> it seems from the comments that this "Kamel" subproject is something we
>> want to start and I think that also the main camel core will benefit from
>> the new features it will bring.
>>
>> I would like to donate the current "Kamel" code to Apache Camel, in order
>> to have a initial brick to start from.
>>
>> From your reactions, the name "Kamel" seems cool and I'd like to keep it.
>> However if you talk to people about the "Kamel" project, they won't
>> understand if you're talking about "Kamel with the K" or "Camel with the
>> C".
>> For this reasons I propose to keep "Kamel", but also use "Camel K" [ˈkæməl
>> keɪ] as friendly name when talking about it. This is in line with the
>> repository we should create for the subproject, that needs to start with
>> "camel-" (it will be "apache/camel-k" in that case).
>>
>> These days I've been experimenting different ideas with Luca Burgazzoli.
>> In
>> particular, there have been some concerns here on the power of a
>> declarative DSL (a new one or the existing XML one) and we've found a
>> strategy that will allow us to use also the Java DSL in Kamel
>> integrations.
>> Without adding too many details here, at integration build time we can run
>> the user code in a build container and inspect the produced routes to get
>> metadata for the optimizer. This is one of the first things we should do
>> next.
>>
>> This possibility of running code that produces integrations opened a lot
>> of
>> unexpected paths, that I've written down in the project roadmap (
>> https://github.com/nicolaferraro/integration-operator) and summarized
>> below.
>>
>> One nice feature (kudos to Luca) is that we can simplify life for Kamel
>> users up to the point that they'll just need to write their routes on a
>> Java file and run them with e.g.:
>> "kamel run Routes.java"
>>
>> Where "kamel" is a binary we release within the project. Kubernetes custom
>> resources will be used under the hood, but the "kamel" binary is a utility
>> that will provide a user experience comparable to that of current
>> serverless platforms (or even better).
>>
>> But there's not just that.
>>
>> Luca wrote this weekend a prototype for having a polyglot Camel (
>> https://github.com/lburgazzoli/camel-routes-loader).
>>
>> With that, you will be able to e.g. write integrations in groovy and use
>> the same Kamel engine:
>> "kamel run routes.groovy"
>>
>> But also JavaScript:
>>
>> https://github.com/lburgazzoli/camel-routes-loader/blob/ca986541f7c422ee02c21727cdfe4293d64a364e/src/test/resources/ext/camel/routes.js#L2
>>
>> This is a prototype right now, but a similar approach based on GraalVM has
>> a lot of potential, because users can not only use their preferred
>> language
>> to write routes and processors, but also bind them to functionalities
>> available in their preferred libraries.
>>
>> How much this will be feasible depends on the adoption of GraalVM, but I'm
>> seeing many frameworks adding metadata to make GraalVM work with
>> reflection. We've also done some work in Camel, there are some Jiras for
>> it
>> and first tests made by Guillaume signal that it's something feasible, at
>> least for the Camel core and a subset of components. GraalVM would be
>> important also to reduce memory footprint and improve startup time, as
>> already said.
>>
>
> I'll continue experimenting and I'll report back.
> I did some initial experiments leveraging some modifications I did  to
> speed up the start up time [1] and on a single route [2].
> There may be some limitations down the road of course, but at least it
> shows that it's feasible. Fwiw, the experiments lead to a startup time of
> 14 ms compared to a few hundreds milliseconds (roughly 680 with [1]).
>
>
>>
>> If the GraalVM approach works (it is working for other frameworks),
>> instead
>> of rewriting a subset of Camel in Go (as the original proposal mentions),
>> we can just sanitize and recompile our existing codebase: this way we
>> fully
>> leverage the strength of Apache Camel.
>>
>
> Right, if we can avoid rewriting Camel, that would be much better !
>
> [1] https://issues.apache.org/jira/browse/CAMEL-12688
> [2]
> https://github.com/gnodet/openwhisk-runtime-camel/blob/master/camel-openwhisk-example/src/main/java/org/jboss/fuse/openwhisk/camel/example/SimpleCamelFunction.java
>
>
>>
>> Another important thing in the roadmap is actually how we'll integrate
>> with
>> current FaaS platforms. We will evaluate together if it's better to have a
>> tighter integration with some of them or to leverage knative for some
>> aspects of Kamel...
>>
>> But, let's get started!
>>
>> Il 19 lug 2018 10:45, "Nicola Ferraro" <ni...@gmail.com> ha scritto:
>>
>> It's clear to me that we need to add support for our existing XML DSL,
>> that
>> is powerful. But there are multiple reasons why I'd like to also "add" a
>> limited yaml/json notation to "Kamel".
>>
>> The first one (and simplest) is that json/yaml is the primary encoding for
>> all resources exchanged with the Kubernetes API server. I'm not saying
>> that
>> for this reason the Integration resource cannot contain a XML
>> (/Java/Ballerina) section, but from a pure presentational point of view,
>> having the possibility to write simple use cases (even a "hello world") in
>> pure json/yaml is better than always requiring a mix of json and XML.
>>
>> The second one is simplicity. Writing a optimizer (the module that
>> materializes the integration into running code, choosing a specific
>> runtime
>> and configuration) for a turing-complete language like Java is not easy:
>> even if you manage to create a good parser, it may be able to do
>> optimizations only if you write routes in a particular way, without
>> complex
>> constructs.
>> XML is ok from this point of view. The thing is that both json/yaml and
>> XML
>> are just two different ways to serialize object trees, that can be then
>> statically analyzed.
>> The point is not XML vs json/yaml, it's more about tailoring a new
>> minimalistic DSL to the emerging use cases vs proposing "only" our classic
>> way of writing integrations. I think XML can be the "advanced" way. We can
>> experiment optimizations easily with the new DSL, and enable them also on
>> XML if it's worth.
>>
>> Scripting should be part of the spec, but I'd try to use programming
>> languages only for processing/transformation, not for the route
>> definition.
>>
>> Third one is performance. Apart from the fact that json parsers are in
>> general said to be faster than XML parsers... Given the "fast startup"
>> target that we want to reach, we may think e.g. to translate the new DSL
>> into Java or Go code, then compiling it. This allows doing parsing at
>> build
>> time in order to avoid it on startup. This kind of improvements are much
>> easier with a limited DSL but much more difficult with a existing
>> fully-fledged DSL..
>>
>> Nicola
>>
>>
>>
>> On Wed, Jul 18, 2018 at 11:51 PM Pontus Ullgren <ul...@gmail.com>
>> wrote:
>>
>> > I also like the idea but with some comments.
>> >
>> > As Hiram Chirino I'm not sure YAML/JSON is the best language for this.
>> > Perhaps a more fluent DSL like the Camel Java DSL or perhaps something
>> like
>> > Ballerina language would be better suited ?
>> > Also in my experience even simple integrations, that is simple real
>> world
>> > integration and not just hello world demos, requires
>> > you to add one or more Java class or scripting in addition to the core
>> > components. So for it to be useful there must be some way to add custom
>> > code in some way for aggregation/enrichment strategies.
>> >
>> > If you go with a GO implementation I would also like some fallback for
>> > using Java since I find it unlikely that the existing base of components
>> > will be ported any time soon.
>> >
>> > As for the name as other I like the "Kamel"  name however a being a
>> newborn
>> > kotlin fan I know of one pre-existing project with that name
>> > that is close enough to allow for potential confusion.
>> > https://github.com/topicusoverheid/kamel
>> >
>> > Looking forward to see the result.
>> >
>> > // Pontus
>> >
>> >
>> >
>> >
>> >
>> >
>> > On Mon, 16 Jul 2018 at 15:15 Hiram Chirino <hi...@hiramchirino.com>
>> wrote:
>> >
>> > > Love the idea.
>> > >
>> > > Personally, I'd keep using the existing Camel XML DSL if possible.
>> You
>> > can
>> > > still embed it in the CRD.  The operator that deploys it could
>> inspect it
>> > > and figure whats the most optimized runtime that can support the DSL.
>> > > Perhaps if it's only using the restricted set of camel components
>> > supported
>> > > by camel-go (https://github.com/lburgazzoli/camel-go) then is uses
>> that.
>> > > Otherwise it falls back to using camel spring boot.
>> > >
>> > > For that to work I think we need the different runtime
>> implementations to
>> > > provide a way to ask them: 'hey do you support running this camel
>> route?'
>> > > Not a trivial thing to respond to, it might require a build step in
>> there
>> > > for traditional Camel runtimes.
>> > >
>> > >
>> > > On Fri, Jul 13, 2018 at 4:32 AM Antonin Stefanutti <
>> > antonin@stefanutti.fr>
>> > > wrote:
>> > >
>> > > > Hi Nicola,
>> > > >
>> > > > I love the idea.
>> > > >
>> > > > I just wonder whether YAML/JSON is an expressive enough format in
>> the
>> > > long
>> > > > term. But as you’ve mentioned, starting simple would enable
>> > experimenting
>> > > > some very interesting / promising optimisations. So it seems worth
>> > taking
>> > > > that path, instead of trying to embed a complex DSL or the existing
>> XML
>> > > DSL
>> > > > into the CRD.
>> > > >
>> > > > Definitely +1
>> > > >
>> > > > > On 13 Jul 2018, at 01:30, Nicola Ferraro <ni...@gmail.com>
>> > wrote:
>> > > > >
>> > > > > Hi Cameleers,
>> > > > > it's now passed some time since I started thinking about a new
>> > project
>> > > > that
>> > > > > we can begin here at Apache Camel, and I'd like to have your
>> opinion.
>> > > > >
>> > > > > We've already been targeting cloud-native applications with Camel,
>> > > > > especially on top of Kubernetes, that is becoming "the standard"
>> > cloud
>> > > > > platform. But writing a Camel integration and running it on
>> > Kubernetes
>> > > > > requires some effort: choosing the base platform (spring-boot,
>> karaf,
>> > > > > simple main?), adding health checks (actuator?), packaging a
>> docker
>> > > image
>> > > > > and creating the Kubernetes resources (fabric8-maven-plugin,
>> helm?),
>> > > > > publishing the image on a docker registry, then finally deploying
>> the
>> > > > > resources on a Kubernetes cluster.
>> > > > >
>> > > > > The resulting integration container is then far from being optimal
>> > > from a
>> > > > > resource consumption point of view: it is likely that a Camel
>> > > Spring-Boot
>> > > > > application will require at least 200MB of RAM and also some CPU
>> > shares
>> > > > > because of polling threads used by many components.
>> > > > >
>> > > > > In case people use a CI/CD pipeline, it will take also a long
>> time to
>> > > get
>> > > > > from a code update to having a Kubernetes POD up and running.
>> > > > > Apart from compilation and image push/pull time, also startup
>> time is
>> > > > often
>> > > > > ~10 seconds for Camel + Spring-Boot in a container with standard
>> > limits
>> > > > on
>> > > > > resources, making it difficult to propose this combination for
>> > > > "serverless
>> > > > > integration" (this term is becoming increasingly more popular).
>> > > > >
>> > > > > So, my proposal is to start to investigate a "more cloud-native"
>> > > approach
>> > > > > to integration: *making Camel integrations first-class citizens in
>> > > > > Kubernetes, and making them super fast and lightweight.*
>> > > > >
>> > > > > We can base the project on Kubernetes Custom Resource Definitions
>> > (CRD)
>> > > > > <
>> > > >
>> > >
>> >
>> https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/
>> > > > >,
>> > > > > for example a Integration CRD and have a Kubernetes "operator"
>> > > > > <https://coreos.com/operators/> taking care of:
>> > > > > - Optimizing the integration that we want to run
>> > > > > - Packaging in a container
>> > > > > - Running it on Kubernetes
>> > > > > - Managing its entire lifecycle
>> > > > >
>> > > > > A Kubernetes-native integration may look like:
>> > > > >
>> > > > > -------------------
>> > > > > kind: "Integration"
>> > > > > apiVersion: "camel.apache.org/v1alpha1"
>> > > > > metadata:
>> > > > > name: "example"
>> > > > > spec:
>> > > > > replicas: 1
>> > > > > routes:
>> > > > >  - id: timer
>> > > > >    route:
>> > > > >     - type: endpoint
>> > > > >       uri: timer:tick
>> > > > >     - type: endpoint
>> > > > >       uri: log:info
>> > > > > -------------------
>> > > > >
>> > > > > For those who are not familiar with Kubernetes resources, this
>> kind
>> > of
>> > > > > YAML/JSON resource definitions are really common.
>> > > > > The example route is embedded in the Kubernetes resource
>> declaration
>> > > and
>> > > > > follows a basic "flow DSL". We may start from a basic one and
>> evolve
>> > it
>> > > > as
>> > > > > new requirements arrive from the community.
>> > > > >
>> > > > > I've made a very simple (but working) POC here:
>> > > > > https://github.com/nicolaferraro/integration-operator.
>> > > > >
>> > > > > This idea of a "Cloud-Native Camel" on Kubernetes (project
>> codename
>> > can
>> > > > be "
>> > > > > *Kamel*", if you like it :D), will be an enabler for a lot of nice
>> > > > features.
>> > > > >
>> > > > > For example, we can propose "Kamel" as "ideal" platform for
>> > "serverless
>> > > > > integration" (I see many people reinventing the wheel out there):
>> the
>> > > > > operator can reduce resource consumption of a single integration
>> by
>> > > > > optimizing the runtime and also pause/resume integrations when
>> they
>> > are
>> > > > not
>> > > > > used, that is the basic idea behind "serverless" (e.g. think to
>> > > > > HTTP-triggered integrations, but not only).
>> > > > > Focusing on serverless will bring more emphasis on push-based
>> > > > notifications
>> > > > > (webhooks, cloud events <https://cloudevents.io/>), that are
>> rarely
>> > > > used in
>> > > > > Camel components, that prefer a poll based approach being it
>> simpler
>> > to
>> > > > use
>> > > > > in classic deployments, but not so good in the cloud, where more
>> > > > resources
>> > > > > become higher direct costs for the users.
>> > > > >
>> > > > > The presence of the simplified DSL enables also experimenting on
>> > > > "*reduced*
>> > > > > subsets of Camel" implemented in languages other than Java, for
>> > example
>> > > > one
>> > > > > language that has a reactive approach on thread scheduling and a
>> > really
>> > > > low
>> > > > > memory footprint, like Go.
>> > > > >
>> > > > > But apart from this kind of experiments (that are valid IMO), the
>> > > "Kamel"
>> > > > > optimizer will have free room to choose the right platform for the
>> > > > > integration that the user wants to run, including, in the future,
>> > doing
>> > > > AOT
>> > > > > compilation using Graal/VM (less memory, faster startup) if the
>> > > features
>> > > > > (components) used in the integration are supporting it (maybe we
>> can
>> > > add
>> > > > > AOT compilation in the roadmap for Camel 3).
>> > > > > A silly optimization: integrations starting from "timer:..." may
>> be
>> > > > > scheduled directly with Kubernetes CronJobs, so they will consume
>> > > > resources
>> > > > > only when actually running.
>> > > > >
>> > > > > Being the final integrations lightweight and being the DSL
>> > > > > language-independent, we may see a increased adoption of Camel
>> also
>> > as
>> > > > > agile integration layer for not-only-java applications (both
>> "cloud"
>> > > and
>> > > > > "serverless" applications).
>> > > > >
>> > > > > I'm the first one that would like to work on a project ilke this.
>> > I've
>> > > > > worked on many Kubernetes/Openshift based applications and
>> frameworks
>> > > in
>> > > > > the past years, also on operators and CRDs, and I think this way
>> of
>> > > > > redesigning integrations has a lot of potential.
>> > > > >
>> > > > > Integrations will not be necessarily limited to the simplified
>> DSL,
>> > but
>> > > > we
>> > > > > can add extension points for scripting and even custom libraries
>> > > > (although
>> > > > > limiting the freedom of the optimizer).
>> > > > >
>> > > > > The most important thing: it may become a great project, since
>> it's
>> > > > driven
>> > > > > by a great community.
>> > > > >
>> > > > > So, what do you think? Is it crazy enough?
>> > > > >
>> > > > > Nicola
>> > > >
>> > > >
>> > >
>> > > --
>> > > Hiram Chirino
>> > > Engineering | Red Hat, Inc.
>> > > hchirino@redhat.com | fusesource.com | redhat.com
>> > > skype: hiramchirino | twitter: @hiramchirino
>> > >
>> >
>>
>
>
> --
> ------------------------
> Guillaume Nodet
>
>

-- 
------------------------
Guillaume Nodet

Re: A "Kamel" crazy idea

[Luca Burgazzoli <lb...@gmail.com>]

---
Luca Burgazzoli


On Thu, Aug 2, 2018 at 8:09 AM, Guillaume Nodet <gn...@apache.org> wrote:
> In order to actually start the route, I forgot to add the required
> parameter:
>
> gnodet•camel/platforms/graalvm*(*graalvm*)*»
> ./example/target/org.apache.camel.graalvm.main -r
> org.apache.camel.graalvm.example.SimpleCamelRouteBuilder
>
> [main] INFO org.apache.camel.graalvm.FastCamelContext - Apache Camel
> 2.23.0-SNAPSHOT (CamelContext: camel-1) is starting
>
> [main] INFO org.apache.camel.graalvm.FastCamelContext - StreamCaching is
> not in use. If using streams then its recommended to enable stream caching.
> See more details at http://camel.apache.org/stream-caching.html
>
> [main] INFO org.apache.camel.graalvm.FastCamelContext - Route: route1
> started and consuming from: file://./target/orders
>
> [main] INFO org.apache.camel.graalvm.FastCamelContext - Total 1 routes, of
> which 1 are started
>
> [main] INFO org.apache.camel.graalvm.FastCamelContext - Apache Camel
> 2.23.0-SNAPSHOT (CamelContext: camel-1) started in 0.012 seconds
>
> Note that the binary is quite big: 26 Mb. One of the reason is that it
> contains the whole camel-core code.  This is because the components /
> languages / dataformats are loaded dynamically, so the graalvm tool has to
> be informed of the implementation classes that can be discovered.
> Currently, all of them from camel-core are included, but I think a better
> way would be to discover them through the route model and only include the
> ones that are needed.  I'll work on that.
>

I guess it can also be a job for a maven plugin to include only what
is listed as dependency. We need to move all the core components out
of camel-core of course but this is also a goal for camel 3 :)

> Le jeu. 2 août 2018 à 00:18, Luca Burgazzoli <lb...@gmail.com> a
> écrit :
>
>> That’s very nice start, thx Guillaume
>>
>> On Wed, 1 Aug 2018 at 20:15, Guillaume Nodet <gn...@apache.org> wrote:
>>
>> > I've pushed a branch with graalvm experiments.
>> >    https://github.com/apache/camel/tree/graalvm
>> > In order to build the project, you need to use the GraalVM jdk and use
>> the
>> > following ~/.m2/toolchains.xml adapted to your path:
>> >
>> > <?xml version="1.0" encoding="UTF8"?>
>> >
>> > <toolchains>
>> >
>> >   <toolchain>
>> >
>> >     <type>jdk</type>
>> >
>> >     <provides>
>> >
>> >       <version>1.0.0-rc4</version>
>> >
>> >       <vendor>oracle</vendor>
>> >
>> >       <id>graalvm-ee-1.0.0-rc4</id>
>> >
>> >     </provides>
>> >
>> >     <configuration>
>> >
>> >
>> >
>> >
>> <jdkHome>/Users/gnodet/Downloads/graalvm-ee-1.0.0-rc4/Contents/Home/</jdkHome>
>> >
>> >     </configuration>
>> >
>> >   </toolchain>
>> >
>> > </toolchains>
>> >
>> > Build and run:
>> >  gnodet•camel*(*graalvm*)*» cd platforms/graalvm
>> >  gnodet•camel/platforms/graalvm*(*graalvm*)*» mvn install
>> > ...
>> >  gnodet•camel/platforms/graalvm*(*graalvm*)*»
>> > ./example/target/org.apache.camel.graalvm.main
>> >
>> > [main] INFO org.apache.camel.graalvm.FastCamelContext - Apache Camel
>> > 2.23.0-SNAPSHOT (CamelContext: camel-1) is starting
>> >
>> > [main] INFO org.apache.camel.graalvm.FastCamelContext - StreamCaching is
>> > not in use. If using streams then its recommended to enable stream
>> caching.
>> > See more details at http://camel.apache.org/stream-caching.html
>> >
>> > [main] INFO org.apache.camel.graalvm.FastCamelContext - Total 0 routes,
>> of
>> > which 0 are started
>> >
>> > [main] INFO org.apache.camel.graalvm.FastCamelContext - Apache Camel
>> > 2.23.0-SNAPSHOT (CamelContext: camel-1) started in 0.006 seconds
>> >
>> >  Guillaume
>> >
>> > Le mar. 31 juil. 2018 à 09:44, Guillaume Nodet <gn...@apache.org> a
>> > écrit :
>> >
>> > >
>> > >
>> > > Le lun. 30 juil. 2018 à 17:49, Nicola Ferraro <ni...@gmail.com> a
>> > > écrit :
>> > >
>> > >> Hi Cameleers,
>> > >> it seems from the comments that this "Kamel" subproject is something
>> we
>> > >> want to start and I think that also the main camel core will benefit
>> > from
>> > >> the new features it will bring.
>> > >>
>> > >> I would like to donate the current "Kamel" code to Apache Camel, in
>> > order
>> > >> to have a initial brick to start from.
>> > >>
>> > >> From your reactions, the name "Kamel" seems cool and I'd like to keep
>> > it.
>> > >> However if you talk to people about the "Kamel" project, they won't
>> > >> understand if you're talking about "Kamel with the K" or "Camel with
>> the
>> > >> C".
>> > >> For this reasons I propose to keep "Kamel", but also use "Camel K"
>> > [ˈkæməl
>> > >> keɪ] as friendly name when talking about it. This is in line with the
>> > >> repository we should create for the subproject, that needs to start
>> with
>> > >> "camel-" (it will be "apache/camel-k" in that case).
>> > >>
>> > >> These days I've been experimenting different ideas with Luca
>> Burgazzoli.
>> > >> In
>> > >> particular, there have been some concerns here on the power of a
>> > >> declarative DSL (a new one or the existing XML one) and we've found a
>> > >> strategy that will allow us to use also the Java DSL in Kamel
>> > >> integrations.
>> > >> Without adding too many details here, at integration build time we can
>> > run
>> > >> the user code in a build container and inspect the produced routes to
>> > get
>> > >> metadata for the optimizer. This is one of the first things we should
>> do
>> > >> next.
>> > >>
>> > >> This possibility of running code that produces integrations opened a
>> lot
>> > >> of
>> > >> unexpected paths, that I've written down in the project roadmap (
>> > >> https://github.com/nicolaferraro/integration-operator) and summarized
>> > >> below.
>> > >>
>> > >> One nice feature (kudos to Luca) is that we can simplify life for
>> Kamel
>> > >> users up to the point that they'll just need to write their routes on
>> a
>> > >> Java file and run them with e.g.:
>> > >> "kamel run Routes.java"
>> > >>
>> > >> Where "kamel" is a binary we release within the project. Kubernetes
>> > custom
>> > >> resources will be used under the hood, but the "kamel" binary is a
>> > utility
>> > >> that will provide a user experience comparable to that of current
>> > >> serverless platforms (or even better).
>> > >>
>> > >> But there's not just that.
>> > >>
>> > >> Luca wrote this weekend a prototype for having a polyglot Camel (
>> > >> https://github.com/lburgazzoli/camel-routes-loader).
>> > >>
>> > >> With that, you will be able to e.g. write integrations in groovy and
>> use
>> > >> the same Kamel engine:
>> > >> "kamel run routes.groovy"
>> > >>
>> > >> But also JavaScript:
>> > >>
>> > >>
>> >
>> https://github.com/lburgazzoli/camel-routes-loader/blob/ca986541f7c422ee02c21727cdfe4293d64a364e/src/test/resources/ext/camel/routes.js#L2
>> > >>
>> > >> This is a prototype right now, but a similar approach based on GraalVM
>> > has
>> > >> a lot of potential, because users can not only use their preferred
>> > >> language
>> > >> to write routes and processors, but also bind them to functionalities
>> > >> available in their preferred libraries.
>> > >>
>> > >> How much this will be feasible depends on the adoption of GraalVM, but
>> > I'm
>> > >> seeing many frameworks adding metadata to make GraalVM work with
>> > >> reflection. We've also done some work in Camel, there are some Jiras
>> for
>> > >> it
>> > >> and first tests made by Guillaume signal that it's something feasible,
>> > at
>> > >> least for the Camel core and a subset of components. GraalVM would be
>> > >> important also to reduce memory footprint and improve startup time, as
>> > >> already said.
>> > >>
>> > >
>> > > I'll continue experimenting and I'll report back.
>> > > I did some initial experiments leveraging some modifications I did  to
>> > > speed up the start up time [1] and on a single route [2].
>> > > There may be some limitations down the road of course, but at least it
>> > > shows that it's feasible. Fwiw, the experiments lead to a startup time
>> of
>> > > 14 ms compared to a few hundreds milliseconds (roughly 680 with [1]).
>> > >
>> > >
>> > >>
>> > >> If the GraalVM approach works (it is working for other frameworks),
>> > >> instead
>> > >> of rewriting a subset of Camel in Go (as the original proposal
>> > mentions),
>> > >> we can just sanitize and recompile our existing codebase: this way we
>> > >> fully
>> > >> leverage the strength of Apache Camel.
>> > >>
>> > >
>> > > Right, if we can avoid rewriting Camel, that would be much better !
>> > >
>> > > [1] https://issues.apache.org/jira/browse/CAMEL-12688
>> > > [2]
>> > >
>> >
>> https://github.com/gnodet/openwhisk-runtime-camel/blob/master/camel-openwhisk-example/src/main/java/org/jboss/fuse/openwhisk/camel/example/SimpleCamelFunction.java
>> > >
>> > >
>> > >>
>> > >> Another important thing in the roadmap is actually how we'll integrate
>> > >> with
>> > >> current FaaS platforms. We will evaluate together if it's better to
>> > have a
>> > >> tighter integration with some of them or to leverage knative for some
>> > >> aspects of Kamel...
>> > >>
>> > >> But, let's get started!
>> > >>
>> > >> Il 19 lug 2018 10:45, "Nicola Ferraro" <ni...@gmail.com> ha
>> > scritto:
>> > >>
>> > >> It's clear to me that we need to add support for our existing XML DSL,
>> > >> that
>> > >> is powerful. But there are multiple reasons why I'd like to also
>> "add" a
>> > >> limited yaml/json notation to "Kamel".
>> > >>
>> > >> The first one (and simplest) is that json/yaml is the primary encoding
>> > for
>> > >> all resources exchanged with the Kubernetes API server. I'm not saying
>> > >> that
>> > >> for this reason the Integration resource cannot contain a XML
>> > >> (/Java/Ballerina) section, but from a pure presentational point of
>> view,
>> > >> having the possibility to write simple use cases (even a "hello
>> world")
>> > in
>> > >> pure json/yaml is better than always requiring a mix of json and XML.
>> > >>
>> > >> The second one is simplicity. Writing a optimizer (the module that
>> > >> materializes the integration into running code, choosing a specific
>> > >> runtime
>> > >> and configuration) for a turing-complete language like Java is not
>> easy:
>> > >> even if you manage to create a good parser, it may be able to do
>> > >> optimizations only if you write routes in a particular way, without
>> > >> complex
>> > >> constructs.
>> > >> XML is ok from this point of view. The thing is that both json/yaml
>> and
>> > >> XML
>> > >> are just two different ways to serialize object trees, that can be
>> then
>> > >> statically analyzed.
>> > >> The point is not XML vs json/yaml, it's more about tailoring a new
>> > >> minimalistic DSL to the emerging use cases vs proposing "only" our
>> > classic
>> > >> way of writing integrations. I think XML can be the "advanced" way. We
>> > can
>> > >> experiment optimizations easily with the new DSL, and enable them also
>> > on
>> > >> XML if it's worth.
>> > >>
>> > >> Scripting should be part of the spec, but I'd try to use programming
>> > >> languages only for processing/transformation, not for the route
>> > >> definition.
>> > >>
>> > >> Third one is performance. Apart from the fact that json parsers are in
>> > >> general said to be faster than XML parsers... Given the "fast startup"
>> > >> target that we want to reach, we may think e.g. to translate the new
>> DSL
>> > >> into Java or Go code, then compiling it. This allows doing parsing at
>> > >> build
>> > >> time in order to avoid it on startup. This kind of improvements are
>> much
>> > >> easier with a limited DSL but much more difficult with a existing
>> > >> fully-fledged DSL..
>> > >>
>> > >> Nicola
>> > >>
>> > >>
>> > >>
>> > >> On Wed, Jul 18, 2018 at 11:51 PM Pontus Ullgren <ul...@gmail.com>
>> > >> wrote:
>> > >>
>> > >> > I also like the idea but with some comments.
>> > >> >
>> > >> > As Hiram Chirino I'm not sure YAML/JSON is the best language for
>> this.
>> > >> > Perhaps a more fluent DSL like the Camel Java DSL or perhaps
>> something
>> > >> like
>> > >> > Ballerina language would be better suited ?
>> > >> > Also in my experience even simple integrations, that is simple real
>> > >> world
>> > >> > integration and not just hello world demos, requires
>> > >> > you to add one or more Java class or scripting in addition to the
>> core
>> > >> > components. So for it to be useful there must be some way to add
>> > custom
>> > >> > code in some way for aggregation/enrichment strategies.
>> > >> >
>> > >> > If you go with a GO implementation I would also like some fallback
>> for
>> > >> > using Java since I find it unlikely that the existing base of
>> > components
>> > >> > will be ported any time soon.
>> > >> >
>> > >> > As for the name as other I like the "Kamel"  name however a being a
>> > >> newborn
>> > >> > kotlin fan I know of one pre-existing project with that name
>> > >> > that is close enough to allow for potential confusion.
>> > >> > https://github.com/topicusoverheid/kamel
>> > >> >
>> > >> > Looking forward to see the result.
>> > >> >
>> > >> > // Pontus
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> > On Mon, 16 Jul 2018 at 15:15 Hiram Chirino <hi...@hiramchirino.com>
>> > >> wrote:
>> > >> >
>> > >> > > Love the idea.
>> > >> > >
>> > >> > > Personally, I'd keep using the existing Camel XML DSL if possible.
>> > >> You
>> > >> > can
>> > >> > > still embed it in the CRD.  The operator that deploys it could
>> > >> inspect it
>> > >> > > and figure whats the most optimized runtime that can support the
>> > DSL.
>> > >> > > Perhaps if it's only using the restricted set of camel components
>> > >> > supported
>> > >> > > by camel-go (https://github.com/lburgazzoli/camel-go) then is
>> uses
>> > >> that.
>> > >> > > Otherwise it falls back to using camel spring boot.
>> > >> > >
>> > >> > > For that to work I think we need the different runtime
>> > >> implementations to
>> > >> > > provide a way to ask them: 'hey do you support running this camel
>> > >> route?'
>> > >> > > Not a trivial thing to respond to, it might require a build step
>> in
>> > >> there
>> > >> > > for traditional Camel runtimes.
>> > >> > >
>> > >> > >
>> > >> > > On Fri, Jul 13, 2018 at 4:32 AM Antonin Stefanutti <
>> > >> > antonin@stefanutti.fr>
>> > >> > > wrote:
>> > >> > >
>> > >> > > > Hi Nicola,
>> > >> > > >
>> > >> > > > I love the idea.
>> > >> > > >
>> > >> > > > I just wonder whether YAML/JSON is an expressive enough format
>> in
>> > >> the
>> > >> > > long
>> > >> > > > term. But as you’ve mentioned, starting simple would enable
>> > >> > experimenting
>> > >> > > > some very interesting / promising optimisations. So it seems
>> worth
>> > >> > taking
>> > >> > > > that path, instead of trying to embed a complex DSL or the
>> > existing
>> > >> XML
>> > >> > > DSL
>> > >> > > > into the CRD.
>> > >> > > >
>> > >> > > > Definitely +1
>> > >> > > >
>> > >> > > > > On 13 Jul 2018, at 01:30, Nicola Ferraro <
>> ni.ferraro@gmail.com>
>> > >> > wrote:
>> > >> > > > >
>> > >> > > > > Hi Cameleers,
>> > >> > > > > it's now passed some time since I started thinking about a new
>> > >> > project
>> > >> > > > that
>> > >> > > > > we can begin here at Apache Camel, and I'd like to have your
>> > >> opinion.
>> > >> > > > >
>> > >> > > > > We've already been targeting cloud-native applications with
>> > Camel,
>> > >> > > > > especially on top of Kubernetes, that is becoming "the
>> standard"
>> > >> > cloud
>> > >> > > > > platform. But writing a Camel integration and running it on
>> > >> > Kubernetes
>> > >> > > > > requires some effort: choosing the base platform (spring-boot,
>> > >> karaf,
>> > >> > > > > simple main?), adding health checks (actuator?), packaging a
>> > >> docker
>> > >> > > image
>> > >> > > > > and creating the Kubernetes resources (fabric8-maven-plugin,
>> > >> helm?),
>> > >> > > > > publishing the image on a docker registry, then finally
>> > deploying
>> > >> the
>> > >> > > > > resources on a Kubernetes cluster.
>> > >> > > > >
>> > >> > > > > The resulting integration container is then far from being
>> > optimal
>> > >> > > from a
>> > >> > > > > resource consumption point of view: it is likely that a Camel
>> > >> > > Spring-Boot
>> > >> > > > > application will require at least 200MB of RAM and also some
>> CPU
>> > >> > shares
>> > >> > > > > because of polling threads used by many components.
>> > >> > > > >
>> > >> > > > > In case people use a CI/CD pipeline, it will take also a long
>> > >> time to
>> > >> > > get
>> > >> > > > > from a code update to having a Kubernetes POD up and running.
>> > >> > > > > Apart from compilation and image push/pull time, also startup
>> > >> time is
>> > >> > > > often
>> > >> > > > > ~10 seconds for Camel + Spring-Boot in a container with
>> standard
>> > >> > limits
>> > >> > > > on
>> > >> > > > > resources, making it difficult to propose this combination for
>> > >> > > > "serverless
>> > >> > > > > integration" (this term is becoming increasingly more
>> popular).
>> > >> > > > >
>> > >> > > > > So, my proposal is to start to investigate a "more
>> cloud-native"
>> > >> > > approach
>> > >> > > > > to integration: *making Camel integrations first-class
>> citizens
>> > in
>> > >> > > > > Kubernetes, and making them super fast and lightweight.*
>> > >> > > > >
>> > >> > > > > We can base the project on Kubernetes Custom Resource
>> > Definitions
>> > >> > (CRD)
>> > >> > > > > <
>> > >> > > >
>> > >> > >
>> > >> >
>> > >>
>> >
>> https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/
>> > >> > > > >,
>> > >> > > > > for example a Integration CRD and have a Kubernetes "operator"
>> > >> > > > > <https://coreos.com/operators/> taking care of:
>> > >> > > > > - Optimizing the integration that we want to run
>> > >> > > > > - Packaging in a container
>> > >> > > > > - Running it on Kubernetes
>> > >> > > > > - Managing its entire lifecycle
>> > >> > > > >
>> > >> > > > > A Kubernetes-native integration may look like:
>> > >> > > > >
>> > >> > > > > -------------------
>> > >> > > > > kind: "Integration"
>> > >> > > > > apiVersion: "camel.apache.org/v1alpha1"
>> > >> > > > > metadata:
>> > >> > > > > name: "example"
>> > >> > > > > spec:
>> > >> > > > > replicas: 1
>> > >> > > > > routes:
>> > >> > > > >  - id: timer
>> > >> > > > >    route:
>> > >> > > > >     - type: endpoint
>> > >> > > > >       uri: timer:tick
>> > >> > > > >     - type: endpoint
>> > >> > > > >       uri: log:info
>> > >> > > > > -------------------
>> > >> > > > >
>> > >> > > > > For those who are not familiar with Kubernetes resources, this
>> > >> kind
>> > >> > of
>> > >> > > > > YAML/JSON resource definitions are really common.
>> > >> > > > > The example route is embedded in the Kubernetes resource
>> > >> declaration
>> > >> > > and
>> > >> > > > > follows a basic "flow DSL". We may start from a basic one and
>> > >> evolve
>> > >> > it
>> > >> > > > as
>> > >> > > > > new requirements arrive from the community.
>> > >> > > > >
>> > >> > > > > I've made a very simple (but working) POC here:
>> > >> > > > > https://github.com/nicolaferraro/integration-operator.
>> > >> > > > >
>> > >> > > > > This idea of a "Cloud-Native Camel" on Kubernetes (project
>> > >> codename
>> > >> > can
>> > >> > > > be "
>> > >> > > > > *Kamel*", if you like it :D), will be an enabler for a lot of
>> > nice
>> > >> > > > features.
>> > >> > > > >
>> > >> > > > > For example, we can propose "Kamel" as "ideal" platform for
>> > >> > "serverless
>> > >> > > > > integration" (I see many people reinventing the wheel out
>> > there):
>> > >> the
>> > >> > > > > operator can reduce resource consumption of a single
>> integration
>> > >> by
>> > >> > > > > optimizing the runtime and also pause/resume integrations when
>> > >> they
>> > >> > are
>> > >> > > > not
>> > >> > > > > used, that is the basic idea behind "serverless" (e.g. think
>> to
>> > >> > > > > HTTP-triggered integrations, but not only).
>> > >> > > > > Focusing on serverless will bring more emphasis on push-based
>> > >> > > > notifications
>> > >> > > > > (webhooks, cloud events <https://cloudevents.io/>), that are
>> > >> rarely
>> > >> > > > used in
>> > >> > > > > Camel components, that prefer a poll based approach being it
>> > >> simpler
>> > >> > to
>> > >> > > > use
>> > >> > > > > in classic deployments, but not so good in the cloud, where
>> more
>> > >> > > > resources
>> > >> > > > > become higher direct costs for the users.
>> > >> > > > >
>> > >> > > > > The presence of the simplified DSL enables also experimenting
>> on
>> > >> > > > "*reduced*
>> > >> > > > > subsets of Camel" implemented in languages other than Java,
>> for
>> > >> > example
>> > >> > > > one
>> > >> > > > > language that has a reactive approach on thread scheduling
>> and a
>> > >> > really
>> > >> > > > low
>> > >> > > > > memory footprint, like Go.
>> > >> > > > >
>> > >> > > > > But apart from this kind of experiments (that are valid IMO),
>> > the
>> > >> > > "Kamel"
>> > >> > > > > optimizer will have free room to choose the right platform for
>> > the
>> > >> > > > > integration that the user wants to run, including, in the
>> > future,
>> > >> > doing
>> > >> > > > AOT
>> > >> > > > > compilation using Graal/VM (less memory, faster startup) if
>> the
>> > >> > > features
>> > >> > > > > (components) used in the integration are supporting it (maybe
>> we
>> > >> can
>> > >> > > add
>> > >> > > > > AOT compilation in the roadmap for Camel 3).
>> > >> > > > > A silly optimization: integrations starting from "timer:..."
>> may
>> > >> be
>> > >> > > > > scheduled directly with Kubernetes CronJobs, so they will
>> > consume
>> > >> > > > resources
>> > >> > > > > only when actually running.
>> > >> > > > >
>> > >> > > > > Being the final integrations lightweight and being the DSL
>> > >> > > > > language-independent, we may see a increased adoption of Camel
>> > >> also
>> > >> > as
>> > >> > > > > agile integration layer for not-only-java applications (both
>> > >> "cloud"
>> > >> > > and
>> > >> > > > > "serverless" applications).
>> > >> > > > >
>> > >> > > > > I'm the first one that would like to work on a project ilke
>> > this.
>> > >> > I've
>> > >> > > > > worked on many Kubernetes/Openshift based applications and
>> > >> frameworks
>> > >> > > in
>> > >> > > > > the past years, also on operators and CRDs, and I think this
>> way
>> > >> of
>> > >> > > > > redesigning integrations has a lot of potential.
>> > >> > > > >
>> > >> > > > > Integrations will not be necessarily limited to the simplified
>> > >> DSL,
>> > >> > but
>> > >> > > > we
>> > >> > > > > can add extension points for scripting and even custom
>> libraries
>> > >> > > > (although
>> > >> > > > > limiting the freedom of the optimizer).
>> > >> > > > >
>> > >> > > > > The most important thing: it may become a great project, since
>> > >> it's
>> > >> > > > driven
>> > >> > > > > by a great community.
>> > >> > > > >
>> > >> > > > > So, what do you think? Is it crazy enough?
>> > >> > > > >
>> > >> > > > > Nicola
>> > >> > > >
>> > >> > > >
>> > >> > >
>> > >> > > --
>> > >> > > Hiram Chirino
>> > >> > > Engineering | Red Hat, Inc.
>> > >> > > hchirino@redhat.com | fusesource.com | redhat.com
>> > >> > > skype: hiramchirino | twitter: @hiramchirino
>> > >> > >
>> > >> >
>> > >>
>> > >
>> > >
>> > > --
>> > > ------------------------
>> > > Guillaume Nodet
>> > >
>> > >
>> >
>> > --
>> > ------------------------
>> > Guillaume Nodet
>> >
>> --
>> --
>> Luca Burgazzoli
>>
>
>
> --
> ------------------------
> Guillaume Nodet

Re: A "Kamel" crazy idea

[Guillaume Nodet <gn...@apache.org>]

In order to actually start the route, I forgot to add the required
parameter:

gnodet•camel/platforms/graalvm*(*graalvm*)*»
./example/target/org.apache.camel.graalvm.main -r
org.apache.camel.graalvm.example.SimpleCamelRouteBuilder

[main] INFO org.apache.camel.graalvm.FastCamelContext - Apache Camel
2.23.0-SNAPSHOT (CamelContext: camel-1) is starting

[main] INFO org.apache.camel.graalvm.FastCamelContext - StreamCaching is
not in use. If using streams then its recommended to enable stream caching.
See more details at http://camel.apache.org/stream-caching.html

[main] INFO org.apache.camel.graalvm.FastCamelContext - Route: route1
started and consuming from: file://./target/orders

[main] INFO org.apache.camel.graalvm.FastCamelContext - Total 1 routes, of
which 1 are started

[main] INFO org.apache.camel.graalvm.FastCamelContext - Apache Camel
2.23.0-SNAPSHOT (CamelContext: camel-1) started in 0.012 seconds

Note that the binary is quite big: 26 Mb. One of the reason is that it
contains the whole camel-core code.  This is because the components /
languages / dataformats are loaded dynamically, so the graalvm tool has to
be informed of the implementation classes that can be discovered.
Currently, all of them from camel-core are included, but I think a better
way would be to discover them through the route model and only include the
ones that are needed.  I'll work on that.

Le jeu. 2 août 2018 à 00:18, Luca Burgazzoli <lb...@gmail.com> a
écrit :

> That’s very nice start, thx Guillaume
>
> On Wed, 1 Aug 2018 at 20:15, Guillaume Nodet <gn...@apache.org> wrote:
>
> > I've pushed a branch with graalvm experiments.
> >    https://github.com/apache/camel/tree/graalvm
> > In order to build the project, you need to use the GraalVM jdk and use
> the
> > following ~/.m2/toolchains.xml adapted to your path:
> >
> > <?xml version="1.0" encoding="UTF8"?>
> >
> > <toolchains>
> >
> >   <toolchain>
> >
> >     <type>jdk</type>
> >
> >     <provides>
> >
> >       <version>1.0.0-rc4</version>
> >
> >       <vendor>oracle</vendor>
> >
> >       <id>graalvm-ee-1.0.0-rc4</id>
> >
> >     </provides>
> >
> >     <configuration>
> >
> >
> >
> >
> <jdkHome>/Users/gnodet/Downloads/graalvm-ee-1.0.0-rc4/Contents/Home/</jdkHome>
> >
> >     </configuration>
> >
> >   </toolchain>
> >
> > </toolchains>
> >
> > Build and run:
> >  gnodet•camel*(*graalvm*)*» cd platforms/graalvm
> >  gnodet•camel/platforms/graalvm*(*graalvm*)*» mvn install
> > ...
> >  gnodet•camel/platforms/graalvm*(*graalvm*)*»
> > ./example/target/org.apache.camel.graalvm.main
> >
> > [main] INFO org.apache.camel.graalvm.FastCamelContext - Apache Camel
> > 2.23.0-SNAPSHOT (CamelContext: camel-1) is starting
> >
> > [main] INFO org.apache.camel.graalvm.FastCamelContext - StreamCaching is
> > not in use. If using streams then its recommended to enable stream
> caching.
> > See more details at http://camel.apache.org/stream-caching.html
> >
> > [main] INFO org.apache.camel.graalvm.FastCamelContext - Total 0 routes,
> of
> > which 0 are started
> >
> > [main] INFO org.apache.camel.graalvm.FastCamelContext - Apache Camel
> > 2.23.0-SNAPSHOT (CamelContext: camel-1) started in 0.006 seconds
> >
> >  Guillaume
> >
> > Le mar. 31 juil. 2018 à 09:44, Guillaume Nodet <gn...@apache.org> a
> > écrit :
> >
> > >
> > >
> > > Le lun. 30 juil. 2018 à 17:49, Nicola Ferraro <ni...@gmail.com> a
> > > écrit :
> > >
> > >> Hi Cameleers,
> > >> it seems from the comments that this "Kamel" subproject is something
> we
> > >> want to start and I think that also the main camel core will benefit
> > from
> > >> the new features it will bring.
> > >>
> > >> I would like to donate the current "Kamel" code to Apache Camel, in
> > order
> > >> to have a initial brick to start from.
> > >>
> > >> From your reactions, the name "Kamel" seems cool and I'd like to keep
> > it.
> > >> However if you talk to people about the "Kamel" project, they won't
> > >> understand if you're talking about "Kamel with the K" or "Camel with
> the
> > >> C".
> > >> For this reasons I propose to keep "Kamel", but also use "Camel K"
> > [ˈkæməl
> > >> keɪ] as friendly name when talking about it. This is in line with the
> > >> repository we should create for the subproject, that needs to start
> with
> > >> "camel-" (it will be "apache/camel-k" in that case).
> > >>
> > >> These days I've been experimenting different ideas with Luca
> Burgazzoli.
> > >> In
> > >> particular, there have been some concerns here on the power of a
> > >> declarative DSL (a new one or the existing XML one) and we've found a
> > >> strategy that will allow us to use also the Java DSL in Kamel
> > >> integrations.
> > >> Without adding too many details here, at integration build time we can
> > run
> > >> the user code in a build container and inspect the produced routes to
> > get
> > >> metadata for the optimizer. This is one of the first things we should
> do
> > >> next.
> > >>
> > >> This possibility of running code that produces integrations opened a
> lot
> > >> of
> > >> unexpected paths, that I've written down in the project roadmap (
> > >> https://github.com/nicolaferraro/integration-operator) and summarized
> > >> below.
> > >>
> > >> One nice feature (kudos to Luca) is that we can simplify life for
> Kamel
> > >> users up to the point that they'll just need to write their routes on
> a
> > >> Java file and run them with e.g.:
> > >> "kamel run Routes.java"
> > >>
> > >> Where "kamel" is a binary we release within the project. Kubernetes
> > custom
> > >> resources will be used under the hood, but the "kamel" binary is a
> > utility
> > >> that will provide a user experience comparable to that of current
> > >> serverless platforms (or even better).
> > >>
> > >> But there's not just that.
> > >>
> > >> Luca wrote this weekend a prototype for having a polyglot Camel (
> > >> https://github.com/lburgazzoli/camel-routes-loader).
> > >>
> > >> With that, you will be able to e.g. write integrations in groovy and
> use
> > >> the same Kamel engine:
> > >> "kamel run routes.groovy"
> > >>
> > >> But also JavaScript:
> > >>
> > >>
> >
> https://github.com/lburgazzoli/camel-routes-loader/blob/ca986541f7c422ee02c21727cdfe4293d64a364e/src/test/resources/ext/camel/routes.js#L2
> > >>
> > >> This is a prototype right now, but a similar approach based on GraalVM
> > has
> > >> a lot of potential, because users can not only use their preferred
> > >> language
> > >> to write routes and processors, but also bind them to functionalities
> > >> available in their preferred libraries.
> > >>
> > >> How much this will be feasible depends on the adoption of GraalVM, but
> > I'm
> > >> seeing many frameworks adding metadata to make GraalVM work with
> > >> reflection. We've also done some work in Camel, there are some Jiras
> for
> > >> it
> > >> and first tests made by Guillaume signal that it's something feasible,
> > at
> > >> least for the Camel core and a subset of components. GraalVM would be
> > >> important also to reduce memory footprint and improve startup time, as
> > >> already said.
> > >>
> > >
> > > I'll continue experimenting and I'll report back.
> > > I did some initial experiments leveraging some modifications I did  to
> > > speed up the start up time [1] and on a single route [2].
> > > There may be some limitations down the road of course, but at least it
> > > shows that it's feasible. Fwiw, the experiments lead to a startup time
> of
> > > 14 ms compared to a few hundreds milliseconds (roughly 680 with [1]).
> > >
> > >
> > >>
> > >> If the GraalVM approach works (it is working for other frameworks),
> > >> instead
> > >> of rewriting a subset of Camel in Go (as the original proposal
> > mentions),
> > >> we can just sanitize and recompile our existing codebase: this way we
> > >> fully
> > >> leverage the strength of Apache Camel.
> > >>
> > >
> > > Right, if we can avoid rewriting Camel, that would be much better !
> > >
> > > [1] https://issues.apache.org/jira/browse/CAMEL-12688
> > > [2]
> > >
> >
> https://github.com/gnodet/openwhisk-runtime-camel/blob/master/camel-openwhisk-example/src/main/java/org/jboss/fuse/openwhisk/camel/example/SimpleCamelFunction.java
> > >
> > >
> > >>
> > >> Another important thing in the roadmap is actually how we'll integrate
> > >> with
> > >> current FaaS platforms. We will evaluate together if it's better to
> > have a
> > >> tighter integration with some of them or to leverage knative for some
> > >> aspects of Kamel...
> > >>
> > >> But, let's get started!
> > >>
> > >> Il 19 lug 2018 10:45, "Nicola Ferraro" <ni...@gmail.com> ha
> > scritto:
> > >>
> > >> It's clear to me that we need to add support for our existing XML DSL,
> > >> that
> > >> is powerful. But there are multiple reasons why I'd like to also
> "add" a
> > >> limited yaml/json notation to "Kamel".
> > >>
> > >> The first one (and simplest) is that json/yaml is the primary encoding
> > for
> > >> all resources exchanged with the Kubernetes API server. I'm not saying
> > >> that
> > >> for this reason the Integration resource cannot contain a XML
> > >> (/Java/Ballerina) section, but from a pure presentational point of
> view,
> > >> having the possibility to write simple use cases (even a "hello
> world")
> > in
> > >> pure json/yaml is better than always requiring a mix of json and XML.
> > >>
> > >> The second one is simplicity. Writing a optimizer (the module that
> > >> materializes the integration into running code, choosing a specific
> > >> runtime
> > >> and configuration) for a turing-complete language like Java is not
> easy:
> > >> even if you manage to create a good parser, it may be able to do
> > >> optimizations only if you write routes in a particular way, without
> > >> complex
> > >> constructs.
> > >> XML is ok from this point of view. The thing is that both json/yaml
> and
> > >> XML
> > >> are just two different ways to serialize object trees, that can be
> then
> > >> statically analyzed.
> > >> The point is not XML vs json/yaml, it's more about tailoring a new
> > >> minimalistic DSL to the emerging use cases vs proposing "only" our
> > classic
> > >> way of writing integrations. I think XML can be the "advanced" way. We
> > can
> > >> experiment optimizations easily with the new DSL, and enable them also
> > on
> > >> XML if it's worth.
> > >>
> > >> Scripting should be part of the spec, but I'd try to use programming
> > >> languages only for processing/transformation, not for the route
> > >> definition.
> > >>
> > >> Third one is performance. Apart from the fact that json parsers are in
> > >> general said to be faster than XML parsers... Given the "fast startup"
> > >> target that we want to reach, we may think e.g. to translate the new
> DSL
> > >> into Java or Go code, then compiling it. This allows doing parsing at
> > >> build
> > >> time in order to avoid it on startup. This kind of improvements are
> much
> > >> easier with a limited DSL but much more difficult with a existing
> > >> fully-fledged DSL..
> > >>
> > >> Nicola
> > >>
> > >>
> > >>
> > >> On Wed, Jul 18, 2018 at 11:51 PM Pontus Ullgren <ul...@gmail.com>
> > >> wrote:
> > >>
> > >> > I also like the idea but with some comments.
> > >> >
> > >> > As Hiram Chirino I'm not sure YAML/JSON is the best language for
> this.
> > >> > Perhaps a more fluent DSL like the Camel Java DSL or perhaps
> something
> > >> like
> > >> > Ballerina language would be better suited ?
> > >> > Also in my experience even simple integrations, that is simple real
> > >> world
> > >> > integration and not just hello world demos, requires
> > >> > you to add one or more Java class or scripting in addition to the
> core
> > >> > components. So for it to be useful there must be some way to add
> > custom
> > >> > code in some way for aggregation/enrichment strategies.
> > >> >
> > >> > If you go with a GO implementation I would also like some fallback
> for
> > >> > using Java since I find it unlikely that the existing base of
> > components
> > >> > will be ported any time soon.
> > >> >
> > >> > As for the name as other I like the "Kamel"  name however a being a
> > >> newborn
> > >> > kotlin fan I know of one pre-existing project with that name
> > >> > that is close enough to allow for potential confusion.
> > >> > https://github.com/topicusoverheid/kamel
> > >> >
> > >> > Looking forward to see the result.
> > >> >
> > >> > // Pontus
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > On Mon, 16 Jul 2018 at 15:15 Hiram Chirino <hi...@hiramchirino.com>
> > >> wrote:
> > >> >
> > >> > > Love the idea.
> > >> > >
> > >> > > Personally, I'd keep using the existing Camel XML DSL if possible.
> > >> You
> > >> > can
> > >> > > still embed it in the CRD.  The operator that deploys it could
> > >> inspect it
> > >> > > and figure whats the most optimized runtime that can support the
> > DSL.
> > >> > > Perhaps if it's only using the restricted set of camel components
> > >> > supported
> > >> > > by camel-go (https://github.com/lburgazzoli/camel-go) then is
> uses
> > >> that.
> > >> > > Otherwise it falls back to using camel spring boot.
> > >> > >
> > >> > > For that to work I think we need the different runtime
> > >> implementations to
> > >> > > provide a way to ask them: 'hey do you support running this camel
> > >> route?'
> > >> > > Not a trivial thing to respond to, it might require a build step
> in
> > >> there
> > >> > > for traditional Camel runtimes.
> > >> > >
> > >> > >
> > >> > > On Fri, Jul 13, 2018 at 4:32 AM Antonin Stefanutti <
> > >> > antonin@stefanutti.fr>
> > >> > > wrote:
> > >> > >
> > >> > > > Hi Nicola,
> > >> > > >
> > >> > > > I love the idea.
> > >> > > >
> > >> > > > I just wonder whether YAML/JSON is an expressive enough format
> in
> > >> the
> > >> > > long
> > >> > > > term. But as you’ve mentioned, starting simple would enable
> > >> > experimenting
> > >> > > > some very interesting / promising optimisations. So it seems
> worth
> > >> > taking
> > >> > > > that path, instead of trying to embed a complex DSL or the
> > existing
> > >> XML
> > >> > > DSL
> > >> > > > into the CRD.
> > >> > > >
> > >> > > > Definitely +1
> > >> > > >
> > >> > > > > On 13 Jul 2018, at 01:30, Nicola Ferraro <
> ni.ferraro@gmail.com>
> > >> > wrote:
> > >> > > > >
> > >> > > > > Hi Cameleers,
> > >> > > > > it's now passed some time since I started thinking about a new
> > >> > project
> > >> > > > that
> > >> > > > > we can begin here at Apache Camel, and I'd like to have your
> > >> opinion.
> > >> > > > >
> > >> > > > > We've already been targeting cloud-native applications with
> > Camel,
> > >> > > > > especially on top of Kubernetes, that is becoming "the
> standard"
> > >> > cloud
> > >> > > > > platform. But writing a Camel integration and running it on
> > >> > Kubernetes
> > >> > > > > requires some effort: choosing the base platform (spring-boot,
> > >> karaf,
> > >> > > > > simple main?), adding health checks (actuator?), packaging a
> > >> docker
> > >> > > image
> > >> > > > > and creating the Kubernetes resources (fabric8-maven-plugin,
> > >> helm?),
> > >> > > > > publishing the image on a docker registry, then finally
> > deploying
> > >> the
> > >> > > > > resources on a Kubernetes cluster.
> > >> > > > >
> > >> > > > > The resulting integration container is then far from being
> > optimal
> > >> > > from a
> > >> > > > > resource consumption point of view: it is likely that a Camel
> > >> > > Spring-Boot
> > >> > > > > application will require at least 200MB of RAM and also some
> CPU
> > >> > shares
> > >> > > > > because of polling threads used by many components.
> > >> > > > >
> > >> > > > > In case people use a CI/CD pipeline, it will take also a long
> > >> time to
> > >> > > get
> > >> > > > > from a code update to having a Kubernetes POD up and running.
> > >> > > > > Apart from compilation and image push/pull time, also startup
> > >> time is
> > >> > > > often
> > >> > > > > ~10 seconds for Camel + Spring-Boot in a container with
> standard
> > >> > limits
> > >> > > > on
> > >> > > > > resources, making it difficult to propose this combination for
> > >> > > > "serverless
> > >> > > > > integration" (this term is becoming increasingly more
> popular).
> > >> > > > >
> > >> > > > > So, my proposal is to start to investigate a "more
> cloud-native"
> > >> > > approach
> > >> > > > > to integration: *making Camel integrations first-class
> citizens
> > in
> > >> > > > > Kubernetes, and making them super fast and lightweight.*
> > >> > > > >
> > >> > > > > We can base the project on Kubernetes Custom Resource
> > Definitions
> > >> > (CRD)
> > >> > > > > <
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/
> > >> > > > >,
> > >> > > > > for example a Integration CRD and have a Kubernetes "operator"
> > >> > > > > <https://coreos.com/operators/> taking care of:
> > >> > > > > - Optimizing the integration that we want to run
> > >> > > > > - Packaging in a container
> > >> > > > > - Running it on Kubernetes
> > >> > > > > - Managing its entire lifecycle
> > >> > > > >
> > >> > > > > A Kubernetes-native integration may look like:
> > >> > > > >
> > >> > > > > -------------------
> > >> > > > > kind: "Integration"
> > >> > > > > apiVersion: "camel.apache.org/v1alpha1"
> > >> > > > > metadata:
> > >> > > > > name: "example"
> > >> > > > > spec:
> > >> > > > > replicas: 1
> > >> > > > > routes:
> > >> > > > >  - id: timer
> > >> > > > >    route:
> > >> > > > >     - type: endpoint
> > >> > > > >       uri: timer:tick
> > >> > > > >     - type: endpoint
> > >> > > > >       uri: log:info
> > >> > > > > -------------------
> > >> > > > >
> > >> > > > > For those who are not familiar with Kubernetes resources, this
> > >> kind
> > >> > of
> > >> > > > > YAML/JSON resource definitions are really common.
> > >> > > > > The example route is embedded in the Kubernetes resource
> > >> declaration
> > >> > > and
> > >> > > > > follows a basic "flow DSL". We may start from a basic one and
> > >> evolve
> > >> > it
> > >> > > > as
> > >> > > > > new requirements arrive from the community.
> > >> > > > >
> > >> > > > > I've made a very simple (but working) POC here:
> > >> > > > > https://github.com/nicolaferraro/integration-operator.
> > >> > > > >
> > >> > > > > This idea of a "Cloud-Native Camel" on Kubernetes (project
> > >> codename
> > >> > can
> > >> > > > be "
> > >> > > > > *Kamel*", if you like it :D), will be an enabler for a lot of
> > nice
> > >> > > > features.
> > >> > > > >
> > >> > > > > For example, we can propose "Kamel" as "ideal" platform for
> > >> > "serverless
> > >> > > > > integration" (I see many people reinventing the wheel out
> > there):
> > >> the
> > >> > > > > operator can reduce resource consumption of a single
> integration
> > >> by
> > >> > > > > optimizing the runtime and also pause/resume integrations when
> > >> they
> > >> > are
> > >> > > > not
> > >> > > > > used, that is the basic idea behind "serverless" (e.g. think
> to
> > >> > > > > HTTP-triggered integrations, but not only).
> > >> > > > > Focusing on serverless will bring more emphasis on push-based
> > >> > > > notifications
> > >> > > > > (webhooks, cloud events <https://cloudevents.io/>), that are
> > >> rarely
> > >> > > > used in
> > >> > > > > Camel components, that prefer a poll based approach being it
> > >> simpler
> > >> > to
> > >> > > > use
> > >> > > > > in classic deployments, but not so good in the cloud, where
> more
> > >> > > > resources
> > >> > > > > become higher direct costs for the users.
> > >> > > > >
> > >> > > > > The presence of the simplified DSL enables also experimenting
> on
> > >> > > > "*reduced*
> > >> > > > > subsets of Camel" implemented in languages other than Java,
> for
> > >> > example
> > >> > > > one
> > >> > > > > language that has a reactive approach on thread scheduling
> and a
> > >> > really
> > >> > > > low
> > >> > > > > memory footprint, like Go.
> > >> > > > >
> > >> > > > > But apart from this kind of experiments (that are valid IMO),
> > the
> > >> > > "Kamel"
> > >> > > > > optimizer will have free room to choose the right platform for
> > the
> > >> > > > > integration that the user wants to run, including, in the
> > future,
> > >> > doing
> > >> > > > AOT
> > >> > > > > compilation using Graal/VM (less memory, faster startup) if
> the
> > >> > > features
> > >> > > > > (components) used in the integration are supporting it (maybe
> we
> > >> can
> > >> > > add
> > >> > > > > AOT compilation in the roadmap for Camel 3).
> > >> > > > > A silly optimization: integrations starting from "timer:..."
> may
> > >> be
> > >> > > > > scheduled directly with Kubernetes CronJobs, so they will
> > consume
> > >> > > > resources
> > >> > > > > only when actually running.
> > >> > > > >
> > >> > > > > Being the final integrations lightweight and being the DSL
> > >> > > > > language-independent, we may see a increased adoption of Camel
> > >> also
> > >> > as
> > >> > > > > agile integration layer for not-only-java applications (both
> > >> "cloud"
> > >> > > and
> > >> > > > > "serverless" applications).
> > >> > > > >
> > >> > > > > I'm the first one that would like to work on a project ilke
> > this.
> > >> > I've
> > >> > > > > worked on many Kubernetes/Openshift based applications and
> > >> frameworks
> > >> > > in
> > >> > > > > the past years, also on operators and CRDs, and I think this
> way
> > >> of
> > >> > > > > redesigning integrations has a lot of potential.
> > >> > > > >
> > >> > > > > Integrations will not be necessarily limited to the simplified
> > >> DSL,
> > >> > but
> > >> > > > we
> > >> > > > > can add extension points for scripting and even custom
> libraries
> > >> > > > (although
> > >> > > > > limiting the freedom of the optimizer).
> > >> > > > >
> > >> > > > > The most important thing: it may become a great project, since
> > >> it's
> > >> > > > driven
> > >> > > > > by a great community.
> > >> > > > >
> > >> > > > > So, what do you think? Is it crazy enough?
> > >> > > > >
> > >> > > > > Nicola
> > >> > > >
> > >> > > >
> > >> > >
> > >> > > --
> > >> > > Hiram Chirino
> > >> > > Engineering | Red Hat, Inc.
> > >> > > hchirino@redhat.com | fusesource.com | redhat.com
> > >> > > skype: hiramchirino | twitter: @hiramchirino
> > >> > >
> > >> >
> > >>
> > >
> > >
> > > --
> > > ------------------------
> > > Guillaume Nodet
> > >
> > >
> >
> > --
> > ------------------------
> > Guillaume Nodet
> >
> --
> --
> Luca Burgazzoli
>


-- 
------------------------
Guillaume Nodet

Re: A "Kamel" crazy idea

[Claus Ibsen <cl...@gmail.com>]

On Thu, Aug 2, 2018 at 12:17 AM, Luca Burgazzoli <lb...@gmail.com> wrote:
> That’s very nice start, thx Guillaume
>

Yes this is really awesome to see this prototype in action.



> On Wed, 1 Aug 2018 at 20:15, Guillaume Nodet <gn...@apache.org> wrote:
>
>> I've pushed a branch with graalvm experiments.
>>    https://github.com/apache/camel/tree/graalvm
>> In order to build the project, you need to use the GraalVM jdk and use the
>> following ~/.m2/toolchains.xml adapted to your path:
>>
>> <?xml version="1.0" encoding="UTF8"?>
>>
>> <toolchains>
>>
>>   <toolchain>
>>
>>     <type>jdk</type>
>>
>>     <provides>
>>
>>       <version>1.0.0-rc4</version>
>>
>>       <vendor>oracle</vendor>
>>
>>       <id>graalvm-ee-1.0.0-rc4</id>
>>
>>     </provides>
>>
>>     <configuration>
>>
>>
>>
>> <jdkHome>/Users/gnodet/Downloads/graalvm-ee-1.0.0-rc4/Contents/Home/</jdkHome>
>>
>>     </configuration>
>>
>>   </toolchain>
>>
>> </toolchains>
>>
>> Build and run:
>>  gnodet•camel*(*graalvm*)*» cd platforms/graalvm
>>  gnodet•camel/platforms/graalvm*(*graalvm*)*» mvn install
>> ...
>>  gnodet•camel/platforms/graalvm*(*graalvm*)*»
>> ./example/target/org.apache.camel.graalvm.main
>>
>> [main] INFO org.apache.camel.graalvm.FastCamelContext - Apache Camel
>> 2.23.0-SNAPSHOT (CamelContext: camel-1) is starting
>>
>> [main] INFO org.apache.camel.graalvm.FastCamelContext - StreamCaching is
>> not in use. If using streams then its recommended to enable stream caching.
>> See more details at http://camel.apache.org/stream-caching.html
>>
>> [main] INFO org.apache.camel.graalvm.FastCamelContext - Total 0 routes, of
>> which 0 are started
>>
>> [main] INFO org.apache.camel.graalvm.FastCamelContext - Apache Camel
>> 2.23.0-SNAPSHOT (CamelContext: camel-1) started in 0.006 seconds
>>
>>  Guillaume
>>
>> Le mar. 31 juil. 2018 à 09:44, Guillaume Nodet <gn...@apache.org> a
>> écrit :
>>
>> >
>> >
>> > Le lun. 30 juil. 2018 à 17:49, Nicola Ferraro <ni...@gmail.com> a
>> > écrit :
>> >
>> >> Hi Cameleers,
>> >> it seems from the comments that this "Kamel" subproject is something we
>> >> want to start and I think that also the main camel core will benefit
>> from
>> >> the new features it will bring.
>> >>
>> >> I would like to donate the current "Kamel" code to Apache Camel, in
>> order
>> >> to have a initial brick to start from.
>> >>
>> >> From your reactions, the name "Kamel" seems cool and I'd like to keep
>> it.
>> >> However if you talk to people about the "Kamel" project, they won't
>> >> understand if you're talking about "Kamel with the K" or "Camel with the
>> >> C".
>> >> For this reasons I propose to keep "Kamel", but also use "Camel K"
>> [ˈkæməl
>> >> keɪ] as friendly name when talking about it. This is in line with the
>> >> repository we should create for the subproject, that needs to start with
>> >> "camel-" (it will be "apache/camel-k" in that case).
>> >>
>> >> These days I've been experimenting different ideas with Luca Burgazzoli.
>> >> In
>> >> particular, there have been some concerns here on the power of a
>> >> declarative DSL (a new one or the existing XML one) and we've found a
>> >> strategy that will allow us to use also the Java DSL in Kamel
>> >> integrations.
>> >> Without adding too many details here, at integration build time we can
>> run
>> >> the user code in a build container and inspect the produced routes to
>> get
>> >> metadata for the optimizer. This is one of the first things we should do
>> >> next.
>> >>
>> >> This possibility of running code that produces integrations opened a lot
>> >> of
>> >> unexpected paths, that I've written down in the project roadmap (
>> >> https://github.com/nicolaferraro/integration-operator) and summarized
>> >> below.
>> >>
>> >> One nice feature (kudos to Luca) is that we can simplify life for Kamel
>> >> users up to the point that they'll just need to write their routes on a
>> >> Java file and run them with e.g.:
>> >> "kamel run Routes.java"
>> >>
>> >> Where "kamel" is a binary we release within the project. Kubernetes
>> custom
>> >> resources will be used under the hood, but the "kamel" binary is a
>> utility
>> >> that will provide a user experience comparable to that of current
>> >> serverless platforms (or even better).
>> >>
>> >> But there's not just that.
>> >>
>> >> Luca wrote this weekend a prototype for having a polyglot Camel (
>> >> https://github.com/lburgazzoli/camel-routes-loader).
>> >>
>> >> With that, you will be able to e.g. write integrations in groovy and use
>> >> the same Kamel engine:
>> >> "kamel run routes.groovy"
>> >>
>> >> But also JavaScript:
>> >>
>> >>
>> https://github.com/lburgazzoli/camel-routes-loader/blob/ca986541f7c422ee02c21727cdfe4293d64a364e/src/test/resources/ext/camel/routes.js#L2
>> >>
>> >> This is a prototype right now, but a similar approach based on GraalVM
>> has
>> >> a lot of potential, because users can not only use their preferred
>> >> language
>> >> to write routes and processors, but also bind them to functionalities
>> >> available in their preferred libraries.
>> >>
>> >> How much this will be feasible depends on the adoption of GraalVM, but
>> I'm
>> >> seeing many frameworks adding metadata to make GraalVM work with
>> >> reflection. We've also done some work in Camel, there are some Jiras for
>> >> it
>> >> and first tests made by Guillaume signal that it's something feasible,
>> at
>> >> least for the Camel core and a subset of components. GraalVM would be
>> >> important also to reduce memory footprint and improve startup time, as
>> >> already said.
>> >>
>> >
>> > I'll continue experimenting and I'll report back.
>> > I did some initial experiments leveraging some modifications I did  to
>> > speed up the start up time [1] and on a single route [2].
>> > There may be some limitations down the road of course, but at least it
>> > shows that it's feasible. Fwiw, the experiments lead to a startup time of
>> > 14 ms compared to a few hundreds milliseconds (roughly 680 with [1]).
>> >
>> >
>> >>
>> >> If the GraalVM approach works (it is working for other frameworks),
>> >> instead
>> >> of rewriting a subset of Camel in Go (as the original proposal
>> mentions),
>> >> we can just sanitize and recompile our existing codebase: this way we
>> >> fully
>> >> leverage the strength of Apache Camel.
>> >>
>> >
>> > Right, if we can avoid rewriting Camel, that would be much better !
>> >
>> > [1] https://issues.apache.org/jira/browse/CAMEL-12688
>> > [2]
>> >
>> https://github.com/gnodet/openwhisk-runtime-camel/blob/master/camel-openwhisk-example/src/main/java/org/jboss/fuse/openwhisk/camel/example/SimpleCamelFunction.java
>> >
>> >
>> >>
>> >> Another important thing in the roadmap is actually how we'll integrate
>> >> with
>> >> current FaaS platforms. We will evaluate together if it's better to
>> have a
>> >> tighter integration with some of them or to leverage knative for some
>> >> aspects of Kamel...
>> >>
>> >> But, let's get started!
>> >>
>> >> Il 19 lug 2018 10:45, "Nicola Ferraro" <ni...@gmail.com> ha
>> scritto:
>> >>
>> >> It's clear to me that we need to add support for our existing XML DSL,
>> >> that
>> >> is powerful. But there are multiple reasons why I'd like to also "add" a
>> >> limited yaml/json notation to "Kamel".
>> >>
>> >> The first one (and simplest) is that json/yaml is the primary encoding
>> for
>> >> all resources exchanged with the Kubernetes API server. I'm not saying
>> >> that
>> >> for this reason the Integration resource cannot contain a XML
>> >> (/Java/Ballerina) section, but from a pure presentational point of view,
>> >> having the possibility to write simple use cases (even a "hello world")
>> in
>> >> pure json/yaml is better than always requiring a mix of json and XML.
>> >>
>> >> The second one is simplicity. Writing a optimizer (the module that
>> >> materializes the integration into running code, choosing a specific
>> >> runtime
>> >> and configuration) for a turing-complete language like Java is not easy:
>> >> even if you manage to create a good parser, it may be able to do
>> >> optimizations only if you write routes in a particular way, without
>> >> complex
>> >> constructs.
>> >> XML is ok from this point of view. The thing is that both json/yaml and
>> >> XML
>> >> are just two different ways to serialize object trees, that can be then
>> >> statically analyzed.
>> >> The point is not XML vs json/yaml, it's more about tailoring a new
>> >> minimalistic DSL to the emerging use cases vs proposing "only" our
>> classic
>> >> way of writing integrations. I think XML can be the "advanced" way. We
>> can
>> >> experiment optimizations easily with the new DSL, and enable them also
>> on
>> >> XML if it's worth.
>> >>
>> >> Scripting should be part of the spec, but I'd try to use programming
>> >> languages only for processing/transformation, not for the route
>> >> definition.
>> >>
>> >> Third one is performance. Apart from the fact that json parsers are in
>> >> general said to be faster than XML parsers... Given the "fast startup"
>> >> target that we want to reach, we may think e.g. to translate the new DSL
>> >> into Java or Go code, then compiling it. This allows doing parsing at
>> >> build
>> >> time in order to avoid it on startup. This kind of improvements are much
>> >> easier with a limited DSL but much more difficult with a existing
>> >> fully-fledged DSL..
>> >>
>> >> Nicola
>> >>
>> >>
>> >>
>> >> On Wed, Jul 18, 2018 at 11:51 PM Pontus Ullgren <ul...@gmail.com>
>> >> wrote:
>> >>
>> >> > I also like the idea but with some comments.
>> >> >
>> >> > As Hiram Chirino I'm not sure YAML/JSON is the best language for this.
>> >> > Perhaps a more fluent DSL like the Camel Java DSL or perhaps something
>> >> like
>> >> > Ballerina language would be better suited ?
>> >> > Also in my experience even simple integrations, that is simple real
>> >> world
>> >> > integration and not just hello world demos, requires
>> >> > you to add one or more Java class or scripting in addition to the core
>> >> > components. So for it to be useful there must be some way to add
>> custom
>> >> > code in some way for aggregation/enrichment strategies.
>> >> >
>> >> > If you go with a GO implementation I would also like some fallback for
>> >> > using Java since I find it unlikely that the existing base of
>> components
>> >> > will be ported any time soon.
>> >> >
>> >> > As for the name as other I like the "Kamel"  name however a being a
>> >> newborn
>> >> > kotlin fan I know of one pre-existing project with that name
>> >> > that is close enough to allow for potential confusion.
>> >> > https://github.com/topicusoverheid/kamel
>> >> >
>> >> > Looking forward to see the result.
>> >> >
>> >> > // Pontus
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > On Mon, 16 Jul 2018 at 15:15 Hiram Chirino <hi...@hiramchirino.com>
>> >> wrote:
>> >> >
>> >> > > Love the idea.
>> >> > >
>> >> > > Personally, I'd keep using the existing Camel XML DSL if possible.
>> >> You
>> >> > can
>> >> > > still embed it in the CRD.  The operator that deploys it could
>> >> inspect it
>> >> > > and figure whats the most optimized runtime that can support the
>> DSL.
>> >> > > Perhaps if it's only using the restricted set of camel components
>> >> > supported
>> >> > > by camel-go (https://github.com/lburgazzoli/camel-go) then is uses
>> >> that.
>> >> > > Otherwise it falls back to using camel spring boot.
>> >> > >
>> >> > > For that to work I think we need the different runtime
>> >> implementations to
>> >> > > provide a way to ask them: 'hey do you support running this camel
>> >> route?'
>> >> > > Not a trivial thing to respond to, it might require a build step in
>> >> there
>> >> > > for traditional Camel runtimes.
>> >> > >
>> >> > >
>> >> > > On Fri, Jul 13, 2018 at 4:32 AM Antonin Stefanutti <
>> >> > antonin@stefanutti.fr>
>> >> > > wrote:
>> >> > >
>> >> > > > Hi Nicola,
>> >> > > >
>> >> > > > I love the idea.
>> >> > > >
>> >> > > > I just wonder whether YAML/JSON is an expressive enough format in
>> >> the
>> >> > > long
>> >> > > > term. But as you’ve mentioned, starting simple would enable
>> >> > experimenting
>> >> > > > some very interesting / promising optimisations. So it seems worth
>> >> > taking
>> >> > > > that path, instead of trying to embed a complex DSL or the
>> existing
>> >> XML
>> >> > > DSL
>> >> > > > into the CRD.
>> >> > > >
>> >> > > > Definitely +1
>> >> > > >
>> >> > > > > On 13 Jul 2018, at 01:30, Nicola Ferraro <ni...@gmail.com>
>> >> > wrote:
>> >> > > > >
>> >> > > > > Hi Cameleers,
>> >> > > > > it's now passed some time since I started thinking about a new
>> >> > project
>> >> > > > that
>> >> > > > > we can begin here at Apache Camel, and I'd like to have your
>> >> opinion.
>> >> > > > >
>> >> > > > > We've already been targeting cloud-native applications with
>> Camel,
>> >> > > > > especially on top of Kubernetes, that is becoming "the standard"
>> >> > cloud
>> >> > > > > platform. But writing a Camel integration and running it on
>> >> > Kubernetes
>> >> > > > > requires some effort: choosing the base platform (spring-boot,
>> >> karaf,
>> >> > > > > simple main?), adding health checks (actuator?), packaging a
>> >> docker
>> >> > > image
>> >> > > > > and creating the Kubernetes resources (fabric8-maven-plugin,
>> >> helm?),
>> >> > > > > publishing the image on a docker registry, then finally
>> deploying
>> >> the
>> >> > > > > resources on a Kubernetes cluster.
>> >> > > > >
>> >> > > > > The resulting integration container is then far from being
>> optimal
>> >> > > from a
>> >> > > > > resource consumption point of view: it is likely that a Camel
>> >> > > Spring-Boot
>> >> > > > > application will require at least 200MB of RAM and also some CPU
>> >> > shares
>> >> > > > > because of polling threads used by many components.
>> >> > > > >
>> >> > > > > In case people use a CI/CD pipeline, it will take also a long
>> >> time to
>> >> > > get
>> >> > > > > from a code update to having a Kubernetes POD up and running.
>> >> > > > > Apart from compilation and image push/pull time, also startup
>> >> time is
>> >> > > > often
>> >> > > > > ~10 seconds for Camel + Spring-Boot in a container with standard
>> >> > limits
>> >> > > > on
>> >> > > > > resources, making it difficult to propose this combination for
>> >> > > > "serverless
>> >> > > > > integration" (this term is becoming increasingly more popular).
>> >> > > > >
>> >> > > > > So, my proposal is to start to investigate a "more cloud-native"
>> >> > > approach
>> >> > > > > to integration: *making Camel integrations first-class citizens
>> in
>> >> > > > > Kubernetes, and making them super fast and lightweight.*
>> >> > > > >
>> >> > > > > We can base the project on Kubernetes Custom Resource
>> Definitions
>> >> > (CRD)
>> >> > > > > <
>> >> > > >
>> >> > >
>> >> >
>> >>
>> https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/
>> >> > > > >,
>> >> > > > > for example a Integration CRD and have a Kubernetes "operator"
>> >> > > > > <https://coreos.com/operators/> taking care of:
>> >> > > > > - Optimizing the integration that we want to run
>> >> > > > > - Packaging in a container
>> >> > > > > - Running it on Kubernetes
>> >> > > > > - Managing its entire lifecycle
>> >> > > > >
>> >> > > > > A Kubernetes-native integration may look like:
>> >> > > > >
>> >> > > > > -------------------
>> >> > > > > kind: "Integration"
>> >> > > > > apiVersion: "camel.apache.org/v1alpha1"
>> >> > > > > metadata:
>> >> > > > > name: "example"
>> >> > > > > spec:
>> >> > > > > replicas: 1
>> >> > > > > routes:
>> >> > > > >  - id: timer
>> >> > > > >    route:
>> >> > > > >     - type: endpoint
>> >> > > > >       uri: timer:tick
>> >> > > > >     - type: endpoint
>> >> > > > >       uri: log:info
>> >> > > > > -------------------
>> >> > > > >
>> >> > > > > For those who are not familiar with Kubernetes resources, this
>> >> kind
>> >> > of
>> >> > > > > YAML/JSON resource definitions are really common.
>> >> > > > > The example route is embedded in the Kubernetes resource
>> >> declaration
>> >> > > and
>> >> > > > > follows a basic "flow DSL". We may start from a basic one and
>> >> evolve
>> >> > it
>> >> > > > as
>> >> > > > > new requirements arrive from the community.
>> >> > > > >
>> >> > > > > I've made a very simple (but working) POC here:
>> >> > > > > https://github.com/nicolaferraro/integration-operator.
>> >> > > > >
>> >> > > > > This idea of a "Cloud-Native Camel" on Kubernetes (project
>> >> codename
>> >> > can
>> >> > > > be "
>> >> > > > > *Kamel*", if you like it :D), will be an enabler for a lot of
>> nice
>> >> > > > features.
>> >> > > > >
>> >> > > > > For example, we can propose "Kamel" as "ideal" platform for
>> >> > "serverless
>> >> > > > > integration" (I see many people reinventing the wheel out
>> there):
>> >> the
>> >> > > > > operator can reduce resource consumption of a single integration
>> >> by
>> >> > > > > optimizing the runtime and also pause/resume integrations when
>> >> they
>> >> > are
>> >> > > > not
>> >> > > > > used, that is the basic idea behind "serverless" (e.g. think to
>> >> > > > > HTTP-triggered integrations, but not only).
>> >> > > > > Focusing on serverless will bring more emphasis on push-based
>> >> > > > notifications
>> >> > > > > (webhooks, cloud events <https://cloudevents.io/>), that are
>> >> rarely
>> >> > > > used in
>> >> > > > > Camel components, that prefer a poll based approach being it
>> >> simpler
>> >> > to
>> >> > > > use
>> >> > > > > in classic deployments, but not so good in the cloud, where more
>> >> > > > resources
>> >> > > > > become higher direct costs for the users.
>> >> > > > >
>> >> > > > > The presence of the simplified DSL enables also experimenting on
>> >> > > > "*reduced*
>> >> > > > > subsets of Camel" implemented in languages other than Java, for
>> >> > example
>> >> > > > one
>> >> > > > > language that has a reactive approach on thread scheduling and a
>> >> > really
>> >> > > > low
>> >> > > > > memory footprint, like Go.
>> >> > > > >
>> >> > > > > But apart from this kind of experiments (that are valid IMO),
>> the
>> >> > > "Kamel"
>> >> > > > > optimizer will have free room to choose the right platform for
>> the
>> >> > > > > integration that the user wants to run, including, in the
>> future,
>> >> > doing
>> >> > > > AOT
>> >> > > > > compilation using Graal/VM (less memory, faster startup) if the
>> >> > > features
>> >> > > > > (components) used in the integration are supporting it (maybe we
>> >> can
>> >> > > add
>> >> > > > > AOT compilation in the roadmap for Camel 3).
>> >> > > > > A silly optimization: integrations starting from "timer:..." may
>> >> be
>> >> > > > > scheduled directly with Kubernetes CronJobs, so they will
>> consume
>> >> > > > resources
>> >> > > > > only when actually running.
>> >> > > > >
>> >> > > > > Being the final integrations lightweight and being the DSL
>> >> > > > > language-independent, we may see a increased adoption of Camel
>> >> also
>> >> > as
>> >> > > > > agile integration layer for not-only-java applications (both
>> >> "cloud"
>> >> > > and
>> >> > > > > "serverless" applications).
>> >> > > > >
>> >> > > > > I'm the first one that would like to work on a project ilke
>> this.
>> >> > I've
>> >> > > > > worked on many Kubernetes/Openshift based applications and
>> >> frameworks
>> >> > > in
>> >> > > > > the past years, also on operators and CRDs, and I think this way
>> >> of
>> >> > > > > redesigning integrations has a lot of potential.
>> >> > > > >
>> >> > > > > Integrations will not be necessarily limited to the simplified
>> >> DSL,
>> >> > but
>> >> > > > we
>> >> > > > > can add extension points for scripting and even custom libraries
>> >> > > > (although
>> >> > > > > limiting the freedom of the optimizer).
>> >> > > > >
>> >> > > > > The most important thing: it may become a great project, since
>> >> it's
>> >> > > > driven
>> >> > > > > by a great community.
>> >> > > > >
>> >> > > > > So, what do you think? Is it crazy enough?
>> >> > > > >
>> >> > > > > Nicola
>> >> > > >
>> >> > > >
>> >> > >
>> >> > > --
>> >> > > Hiram Chirino
>> >> > > Engineering | Red Hat, Inc.
>> >> > > hchirino@redhat.com | fusesource.com | redhat.com
>> >> > > skype: hiramchirino | twitter: @hiramchirino
>> >> > >
>> >> >
>> >>
>> >
>> >
>> > --
>> > ------------------------
>> > Guillaume Nodet
>> >
>> >
>>
>> --
>> ------------------------
>> Guillaume Nodet
>>
> --
> --
> Luca Burgazzoli



-- 
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2

Re: A "Kamel" crazy idea

[Luca Burgazzoli <lb...@gmail.com>]

That’s very nice start, thx Guillaume

On Wed, 1 Aug 2018 at 20:15, Guillaume Nodet <gn...@apache.org> wrote:

> I've pushed a branch with graalvm experiments.
>    https://github.com/apache/camel/tree/graalvm
> In order to build the project, you need to use the GraalVM jdk and use the
> following ~/.m2/toolchains.xml adapted to your path:
>
> <?xml version="1.0" encoding="UTF8"?>
>
> <toolchains>
>
>   <toolchain>
>
>     <type>jdk</type>
>
>     <provides>
>
>       <version>1.0.0-rc4</version>
>
>       <vendor>oracle</vendor>
>
>       <id>graalvm-ee-1.0.0-rc4</id>
>
>     </provides>
>
>     <configuration>
>
>
>
> <jdkHome>/Users/gnodet/Downloads/graalvm-ee-1.0.0-rc4/Contents/Home/</jdkHome>
>
>     </configuration>
>
>   </toolchain>
>
> </toolchains>
>
> Build and run:
>  gnodet•camel*(*graalvm*)*» cd platforms/graalvm
>  gnodet•camel/platforms/graalvm*(*graalvm*)*» mvn install
> ...
>  gnodet•camel/platforms/graalvm*(*graalvm*)*»
> ./example/target/org.apache.camel.graalvm.main
>
> [main] INFO org.apache.camel.graalvm.FastCamelContext - Apache Camel
> 2.23.0-SNAPSHOT (CamelContext: camel-1) is starting
>
> [main] INFO org.apache.camel.graalvm.FastCamelContext - StreamCaching is
> not in use. If using streams then its recommended to enable stream caching.
> See more details at http://camel.apache.org/stream-caching.html
>
> [main] INFO org.apache.camel.graalvm.FastCamelContext - Total 0 routes, of
> which 0 are started
>
> [main] INFO org.apache.camel.graalvm.FastCamelContext - Apache Camel
> 2.23.0-SNAPSHOT (CamelContext: camel-1) started in 0.006 seconds
>
>  Guillaume
>
> Le mar. 31 juil. 2018 à 09:44, Guillaume Nodet <gn...@apache.org> a
> écrit :
>
> >
> >
> > Le lun. 30 juil. 2018 à 17:49, Nicola Ferraro <ni...@gmail.com> a
> > écrit :
> >
> >> Hi Cameleers,
> >> it seems from the comments that this "Kamel" subproject is something we
> >> want to start and I think that also the main camel core will benefit
> from
> >> the new features it will bring.
> >>
> >> I would like to donate the current "Kamel" code to Apache Camel, in
> order
> >> to have a initial brick to start from.
> >>
> >> From your reactions, the name "Kamel" seems cool and I'd like to keep
> it.
> >> However if you talk to people about the "Kamel" project, they won't
> >> understand if you're talking about "Kamel with the K" or "Camel with the
> >> C".
> >> For this reasons I propose to keep "Kamel", but also use "Camel K"
> [ˈkæməl
> >> keɪ] as friendly name when talking about it. This is in line with the
> >> repository we should create for the subproject, that needs to start with
> >> "camel-" (it will be "apache/camel-k" in that case).
> >>
> >> These days I've been experimenting different ideas with Luca Burgazzoli.
> >> In
> >> particular, there have been some concerns here on the power of a
> >> declarative DSL (a new one or the existing XML one) and we've found a
> >> strategy that will allow us to use also the Java DSL in Kamel
> >> integrations.
> >> Without adding too many details here, at integration build time we can
> run
> >> the user code in a build container and inspect the produced routes to
> get
> >> metadata for the optimizer. This is one of the first things we should do
> >> next.
> >>
> >> This possibility of running code that produces integrations opened a lot
> >> of
> >> unexpected paths, that I've written down in the project roadmap (
> >> https://github.com/nicolaferraro/integration-operator) and summarized
> >> below.
> >>
> >> One nice feature (kudos to Luca) is that we can simplify life for Kamel
> >> users up to the point that they'll just need to write their routes on a
> >> Java file and run them with e.g.:
> >> "kamel run Routes.java"
> >>
> >> Where "kamel" is a binary we release within the project. Kubernetes
> custom
> >> resources will be used under the hood, but the "kamel" binary is a
> utility
> >> that will provide a user experience comparable to that of current
> >> serverless platforms (or even better).
> >>
> >> But there's not just that.
> >>
> >> Luca wrote this weekend a prototype for having a polyglot Camel (
> >> https://github.com/lburgazzoli/camel-routes-loader).
> >>
> >> With that, you will be able to e.g. write integrations in groovy and use
> >> the same Kamel engine:
> >> "kamel run routes.groovy"
> >>
> >> But also JavaScript:
> >>
> >>
> https://github.com/lburgazzoli/camel-routes-loader/blob/ca986541f7c422ee02c21727cdfe4293d64a364e/src/test/resources/ext/camel/routes.js#L2
> >>
> >> This is a prototype right now, but a similar approach based on GraalVM
> has
> >> a lot of potential, because users can not only use their preferred
> >> language
> >> to write routes and processors, but also bind them to functionalities
> >> available in their preferred libraries.
> >>
> >> How much this will be feasible depends on the adoption of GraalVM, but
> I'm
> >> seeing many frameworks adding metadata to make GraalVM work with
> >> reflection. We've also done some work in Camel, there are some Jiras for
> >> it
> >> and first tests made by Guillaume signal that it's something feasible,
> at
> >> least for the Camel core and a subset of components. GraalVM would be
> >> important also to reduce memory footprint and improve startup time, as
> >> already said.
> >>
> >
> > I'll continue experimenting and I'll report back.
> > I did some initial experiments leveraging some modifications I did  to
> > speed up the start up time [1] and on a single route [2].
> > There may be some limitations down the road of course, but at least it
> > shows that it's feasible. Fwiw, the experiments lead to a startup time of
> > 14 ms compared to a few hundreds milliseconds (roughly 680 with [1]).
> >
> >
> >>
> >> If the GraalVM approach works (it is working for other frameworks),
> >> instead
> >> of rewriting a subset of Camel in Go (as the original proposal
> mentions),
> >> we can just sanitize and recompile our existing codebase: this way we
> >> fully
> >> leverage the strength of Apache Camel.
> >>
> >
> > Right, if we can avoid rewriting Camel, that would be much better !
> >
> > [1] https://issues.apache.org/jira/browse/CAMEL-12688
> > [2]
> >
> https://github.com/gnodet/openwhisk-runtime-camel/blob/master/camel-openwhisk-example/src/main/java/org/jboss/fuse/openwhisk/camel/example/SimpleCamelFunction.java
> >
> >
> >>
> >> Another important thing in the roadmap is actually how we'll integrate
> >> with
> >> current FaaS platforms. We will evaluate together if it's better to
> have a
> >> tighter integration with some of them or to leverage knative for some
> >> aspects of Kamel...
> >>
> >> But, let's get started!
> >>
> >> Il 19 lug 2018 10:45, "Nicola Ferraro" <ni...@gmail.com> ha
> scritto:
> >>
> >> It's clear to me that we need to add support for our existing XML DSL,
> >> that
> >> is powerful. But there are multiple reasons why I'd like to also "add" a
> >> limited yaml/json notation to "Kamel".
> >>
> >> The first one (and simplest) is that json/yaml is the primary encoding
> for
> >> all resources exchanged with the Kubernetes API server. I'm not saying
> >> that
> >> for this reason the Integration resource cannot contain a XML
> >> (/Java/Ballerina) section, but from a pure presentational point of view,
> >> having the possibility to write simple use cases (even a "hello world")
> in
> >> pure json/yaml is better than always requiring a mix of json and XML.
> >>
> >> The second one is simplicity. Writing a optimizer (the module that
> >> materializes the integration into running code, choosing a specific
> >> runtime
> >> and configuration) for a turing-complete language like Java is not easy:
> >> even if you manage to create a good parser, it may be able to do
> >> optimizations only if you write routes in a particular way, without
> >> complex
> >> constructs.
> >> XML is ok from this point of view. The thing is that both json/yaml and
> >> XML
> >> are just two different ways to serialize object trees, that can be then
> >> statically analyzed.
> >> The point is not XML vs json/yaml, it's more about tailoring a new
> >> minimalistic DSL to the emerging use cases vs proposing "only" our
> classic
> >> way of writing integrations. I think XML can be the "advanced" way. We
> can
> >> experiment optimizations easily with the new DSL, and enable them also
> on
> >> XML if it's worth.
> >>
> >> Scripting should be part of the spec, but I'd try to use programming
> >> languages only for processing/transformation, not for the route
> >> definition.
> >>
> >> Third one is performance. Apart from the fact that json parsers are in
> >> general said to be faster than XML parsers... Given the "fast startup"
> >> target that we want to reach, we may think e.g. to translate the new DSL
> >> into Java or Go code, then compiling it. This allows doing parsing at
> >> build
> >> time in order to avoid it on startup. This kind of improvements are much
> >> easier with a limited DSL but much more difficult with a existing
> >> fully-fledged DSL..
> >>
> >> Nicola
> >>
> >>
> >>
> >> On Wed, Jul 18, 2018 at 11:51 PM Pontus Ullgren <ul...@gmail.com>
> >> wrote:
> >>
> >> > I also like the idea but with some comments.
> >> >
> >> > As Hiram Chirino I'm not sure YAML/JSON is the best language for this.
> >> > Perhaps a more fluent DSL like the Camel Java DSL or perhaps something
> >> like
> >> > Ballerina language would be better suited ?
> >> > Also in my experience even simple integrations, that is simple real
> >> world
> >> > integration and not just hello world demos, requires
> >> > you to add one or more Java class or scripting in addition to the core
> >> > components. So for it to be useful there must be some way to add
> custom
> >> > code in some way for aggregation/enrichment strategies.
> >> >
> >> > If you go with a GO implementation I would also like some fallback for
> >> > using Java since I find it unlikely that the existing base of
> components
> >> > will be ported any time soon.
> >> >
> >> > As for the name as other I like the "Kamel"  name however a being a
> >> newborn
> >> > kotlin fan I know of one pre-existing project with that name
> >> > that is close enough to allow for potential confusion.
> >> > https://github.com/topicusoverheid/kamel
> >> >
> >> > Looking forward to see the result.
> >> >
> >> > // Pontus
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > On Mon, 16 Jul 2018 at 15:15 Hiram Chirino <hi...@hiramchirino.com>
> >> wrote:
> >> >
> >> > > Love the idea.
> >> > >
> >> > > Personally, I'd keep using the existing Camel XML DSL if possible.
> >> You
> >> > can
> >> > > still embed it in the CRD.  The operator that deploys it could
> >> inspect it
> >> > > and figure whats the most optimized runtime that can support the
> DSL.
> >> > > Perhaps if it's only using the restricted set of camel components
> >> > supported
> >> > > by camel-go (https://github.com/lburgazzoli/camel-go) then is uses
> >> that.
> >> > > Otherwise it falls back to using camel spring boot.
> >> > >
> >> > > For that to work I think we need the different runtime
> >> implementations to
> >> > > provide a way to ask them: 'hey do you support running this camel
> >> route?'
> >> > > Not a trivial thing to respond to, it might require a build step in
> >> there
> >> > > for traditional Camel runtimes.
> >> > >
> >> > >
> >> > > On Fri, Jul 13, 2018 at 4:32 AM Antonin Stefanutti <
> >> > antonin@stefanutti.fr>
> >> > > wrote:
> >> > >
> >> > > > Hi Nicola,
> >> > > >
> >> > > > I love the idea.
> >> > > >
> >> > > > I just wonder whether YAML/JSON is an expressive enough format in
> >> the
> >> > > long
> >> > > > term. But as you’ve mentioned, starting simple would enable
> >> > experimenting
> >> > > > some very interesting / promising optimisations. So it seems worth
> >> > taking
> >> > > > that path, instead of trying to embed a complex DSL or the
> existing
> >> XML
> >> > > DSL
> >> > > > into the CRD.
> >> > > >
> >> > > > Definitely +1
> >> > > >
> >> > > > > On 13 Jul 2018, at 01:30, Nicola Ferraro <ni...@gmail.com>
> >> > wrote:
> >> > > > >
> >> > > > > Hi Cameleers,
> >> > > > > it's now passed some time since I started thinking about a new
> >> > project
> >> > > > that
> >> > > > > we can begin here at Apache Camel, and I'd like to have your
> >> opinion.
> >> > > > >
> >> > > > > We've already been targeting cloud-native applications with
> Camel,
> >> > > > > especially on top of Kubernetes, that is becoming "the standard"
> >> > cloud
> >> > > > > platform. But writing a Camel integration and running it on
> >> > Kubernetes
> >> > > > > requires some effort: choosing the base platform (spring-boot,
> >> karaf,
> >> > > > > simple main?), adding health checks (actuator?), packaging a
> >> docker
> >> > > image
> >> > > > > and creating the Kubernetes resources (fabric8-maven-plugin,
> >> helm?),
> >> > > > > publishing the image on a docker registry, then finally
> deploying
> >> the
> >> > > > > resources on a Kubernetes cluster.
> >> > > > >
> >> > > > > The resulting integration container is then far from being
> optimal
> >> > > from a
> >> > > > > resource consumption point of view: it is likely that a Camel
> >> > > Spring-Boot
> >> > > > > application will require at least 200MB of RAM and also some CPU
> >> > shares
> >> > > > > because of polling threads used by many components.
> >> > > > >
> >> > > > > In case people use a CI/CD pipeline, it will take also a long
> >> time to
> >> > > get
> >> > > > > from a code update to having a Kubernetes POD up and running.
> >> > > > > Apart from compilation and image push/pull time, also startup
> >> time is
> >> > > > often
> >> > > > > ~10 seconds for Camel + Spring-Boot in a container with standard
> >> > limits
> >> > > > on
> >> > > > > resources, making it difficult to propose this combination for
> >> > > > "serverless
> >> > > > > integration" (this term is becoming increasingly more popular).
> >> > > > >
> >> > > > > So, my proposal is to start to investigate a "more cloud-native"
> >> > > approach
> >> > > > > to integration: *making Camel integrations first-class citizens
> in
> >> > > > > Kubernetes, and making them super fast and lightweight.*
> >> > > > >
> >> > > > > We can base the project on Kubernetes Custom Resource
> Definitions
> >> > (CRD)
> >> > > > > <
> >> > > >
> >> > >
> >> >
> >>
> https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/
> >> > > > >,
> >> > > > > for example a Integration CRD and have a Kubernetes "operator"
> >> > > > > <https://coreos.com/operators/> taking care of:
> >> > > > > - Optimizing the integration that we want to run
> >> > > > > - Packaging in a container
> >> > > > > - Running it on Kubernetes
> >> > > > > - Managing its entire lifecycle
> >> > > > >
> >> > > > > A Kubernetes-native integration may look like:
> >> > > > >
> >> > > > > -------------------
> >> > > > > kind: "Integration"
> >> > > > > apiVersion: "camel.apache.org/v1alpha1"
> >> > > > > metadata:
> >> > > > > name: "example"
> >> > > > > spec:
> >> > > > > replicas: 1
> >> > > > > routes:
> >> > > > >  - id: timer
> >> > > > >    route:
> >> > > > >     - type: endpoint
> >> > > > >       uri: timer:tick
> >> > > > >     - type: endpoint
> >> > > > >       uri: log:info
> >> > > > > -------------------
> >> > > > >
> >> > > > > For those who are not familiar with Kubernetes resources, this
> >> kind
> >> > of
> >> > > > > YAML/JSON resource definitions are really common.
> >> > > > > The example route is embedded in the Kubernetes resource
> >> declaration
> >> > > and
> >> > > > > follows a basic "flow DSL". We may start from a basic one and
> >> evolve
> >> > it
> >> > > > as
> >> > > > > new requirements arrive from the community.
> >> > > > >
> >> > > > > I've made a very simple (but working) POC here:
> >> > > > > https://github.com/nicolaferraro/integration-operator.
> >> > > > >
> >> > > > > This idea of a "Cloud-Native Camel" on Kubernetes (project
> >> codename
> >> > can
> >> > > > be "
> >> > > > > *Kamel*", if you like it :D), will be an enabler for a lot of
> nice
> >> > > > features.
> >> > > > >
> >> > > > > For example, we can propose "Kamel" as "ideal" platform for
> >> > "serverless
> >> > > > > integration" (I see many people reinventing the wheel out
> there):
> >> the
> >> > > > > operator can reduce resource consumption of a single integration
> >> by
> >> > > > > optimizing the runtime and also pause/resume integrations when
> >> they
> >> > are
> >> > > > not
> >> > > > > used, that is the basic idea behind "serverless" (e.g. think to
> >> > > > > HTTP-triggered integrations, but not only).
> >> > > > > Focusing on serverless will bring more emphasis on push-based
> >> > > > notifications
> >> > > > > (webhooks, cloud events <https://cloudevents.io/>), that are
> >> rarely
> >> > > > used in
> >> > > > > Camel components, that prefer a poll based approach being it
> >> simpler
> >> > to
> >> > > > use
> >> > > > > in classic deployments, but not so good in the cloud, where more
> >> > > > resources
> >> > > > > become higher direct costs for the users.
> >> > > > >
> >> > > > > The presence of the simplified DSL enables also experimenting on
> >> > > > "*reduced*
> >> > > > > subsets of Camel" implemented in languages other than Java, for
> >> > example
> >> > > > one
> >> > > > > language that has a reactive approach on thread scheduling and a
> >> > really
> >> > > > low
> >> > > > > memory footprint, like Go.
> >> > > > >
> >> > > > > But apart from this kind of experiments (that are valid IMO),
> the
> >> > > "Kamel"
> >> > > > > optimizer will have free room to choose the right platform for
> the
> >> > > > > integration that the user wants to run, including, in the
> future,
> >> > doing
> >> > > > AOT
> >> > > > > compilation using Graal/VM (less memory, faster startup) if the
> >> > > features
> >> > > > > (components) used in the integration are supporting it (maybe we
> >> can
> >> > > add
> >> > > > > AOT compilation in the roadmap for Camel 3).
> >> > > > > A silly optimization: integrations starting from "timer:..." may
> >> be
> >> > > > > scheduled directly with Kubernetes CronJobs, so they will
> consume
> >> > > > resources
> >> > > > > only when actually running.
> >> > > > >
> >> > > > > Being the final integrations lightweight and being the DSL
> >> > > > > language-independent, we may see a increased adoption of Camel
> >> also
> >> > as
> >> > > > > agile integration layer for not-only-java applications (both
> >> "cloud"
> >> > > and
> >> > > > > "serverless" applications).
> >> > > > >
> >> > > > > I'm the first one that would like to work on a project ilke
> this.
> >> > I've
> >> > > > > worked on many Kubernetes/Openshift based applications and
> >> frameworks
> >> > > in
> >> > > > > the past years, also on operators and CRDs, and I think this way
> >> of
> >> > > > > redesigning integrations has a lot of potential.
> >> > > > >
> >> > > > > Integrations will not be necessarily limited to the simplified
> >> DSL,
> >> > but
> >> > > > we
> >> > > > > can add extension points for scripting and even custom libraries
> >> > > > (although
> >> > > > > limiting the freedom of the optimizer).
> >> > > > >
> >> > > > > The most important thing: it may become a great project, since
> >> it's
> >> > > > driven
> >> > > > > by a great community.
> >> > > > >
> >> > > > > So, what do you think? Is it crazy enough?
> >> > > > >
> >> > > > > Nicola
> >> > > >
> >> > > >
> >> > >
> >> > > --
> >> > > Hiram Chirino
> >> > > Engineering | Red Hat, Inc.
> >> > > hchirino@redhat.com | fusesource.com | redhat.com
> >> > > skype: hiramchirino | twitter: @hiramchirino
> >> > >
> >> >
> >>
> >
> >
> > --
> > ------------------------
> > Guillaume Nodet
> >
> >
>
> --
> ------------------------
> Guillaume Nodet
>
-- 
--
Luca Burgazzoli