You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@camel.apache.org by "squakez (via GitHub)" <gi...@apache.org> on 2023/02/02 11:16:12 UTC
[GitHub] [camel-k] squakez opened a new issue, #4024: Make project generation explicit and generic
squakez opened a new issue, #4024:
URL: https://github.com/apache/camel-k/issues/4024
As we're discussing around Camel K 2.0 and the possibility to make it runtime-generic, and as we're discovering we may need full source at build time in order to build natively, here a possible proposal to enable such features.
Disclaimer: this is a very high level design, any development will require a further and low level assessment. I'd like to discuss the "what" without thinking too much in the "how" (at least at this stage).
At this stage the project generation is done just before the build of the project. But the build is generic as the original scope was to have a base maven project containing the set of dependencies (jars) required by the IntegrationKit to build the container image required at runtime. The source developed by the user was later "injected" into the Camel context by Camel K Runtime project. This approach is limiting us to enable anything else but Camel K Runtime applications. Moreover with this approach the `pod` build strategy requires a circular reference to the Camel K container image to be able to reuse an hidden `kamel builder` command.
It follows a diagram with my proposal:
I propose to separate the project generation and the build so that we can generate the project or even upload the project (ie, from a git repository or from a `camel export`). We can try to reuse (and eventually maintain) the same Camel JBang export in order to have the same output as the user had locally. We can introduce a reconciliation loop on a new Project CR: once the project is generated/uploaded (Project status Ready) we'd move to the Build phase that has to know the project (probably the location which may be in Project.Status).
The Build should be more or less how it is now, and the rest of the flow probably similar as well. Probably we'll need to start supporting the kind of project we support now (Camel K runtime/Camel Quarkus) and verifying the compatibility of the incremental image/reuse of previous IntegrationKits. Once this is smooth enough we can start supporting any further generic Camel application (main, SB) leveraging the same mechanism of IntegrationKits inheritance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-k] essobedo commented on issue #4024: Make project generation explicit and generic
Posted by "essobedo (via GitHub)" <gi...@apache.org>.
essobedo commented on issue #4024:
URL: https://github.com/apache/camel-k/issues/4024#issuecomment-1414123522
IMHO, If you keep it under the Camel umbrella, people will get confused and the project will hardly get the deserved success. So far everything under the Camel umbrella is Camel related.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-k] essobedo commented on issue #4024: Make project generation explicit and generic
Posted by "essobedo (via GitHub)" <gi...@apache.org>.
essobedo commented on issue #4024:
URL: https://github.com/apache/camel-k/issues/4024#issuecomment-1415351195
Back to the GitHub URL feature, don't you believe that losing control of what is built, could allow the execution of malicious code at build time that could, for example, use the K8s API to collect sensitive data especially if the build is made from the operator's pod?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-k] essobedo commented on issue #4024: Make project generation explicit and generic
Posted by "essobedo (via GitHub)" <gi...@apache.org>.
essobedo commented on issue #4024:
URL: https://github.com/apache/camel-k/issues/4024#issuecomment-1415453694
> I'm not particularly worried, consider any pipeline tool (ie Tekton), they have tasks to pull from git and build accordingly. At some point we may even think to integrate them from any given pipeline technology.
Yes, but in the case of a pipeline, you can isolate the runner and limit its accesses rights. Here even if all the builds are delegated to a specific pod, the pod still needs to be able to update the status of the build, doesn't it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-k] lburgazzoli commented on issue #4024: Make project generation explicit and generic
Posted by "lburgazzoli (via GitHub)" <gi...@apache.org>.
lburgazzoli commented on issue #4024:
URL: https://github.com/apache/camel-k/issues/4024#issuecomment-1414442829
@squakez when talking about git, what does that mean exactly ? In my mind, cloning a project from git could and should be supported, pushing to git as a "storage" is very hard to be done right and I don't see the real advantage vs doing an ephemeral project generation in a pipeline.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-k] squakez commented on issue #4024: Make project generation explicit and generic
Posted by "squakez (via GitHub)" <gi...@apache.org>.
squakez commented on issue #4024:
URL: https://github.com/apache/camel-k/issues/4024#issuecomment-1415344671
> > I am all in for this proposal to have the standard maven project with source code instead of injecting it, sound more natural and allows flexibility in the future to tweak the build tool (mvnd?) I don't know the original motivation to have camel-k-runtime to inject the source code (perhaps, to not depend on the pod operator file system). With this move, there will be less code to maintain in camel-k-runtime
>
> I really don't read this proposal this way, and if it were then it would break any major camel-k use case as you would need to trigger a build every time a kamelet binding/integration changes. We had the option to include the sources back in the day and I'm all for supporting this, but it must be an opt in feature (note that including a yaml/xml/js/groovy/kotlin route is not different from attaching it at runtime and such functionality is implemented by camel core, the camel-k-runtime adds supports for advanced use cases required such has knative, cron, master)
Exactly, this would be an optional feature that could simplify onboarding other Camel runtimes. Also, mind that we need to evaluate the feasibility with the actual design in place.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-k] lburgazzoli commented on issue #4024: Make project generation explicit and generic
Posted by "lburgazzoli (via GitHub)" <gi...@apache.org>.
lburgazzoli commented on issue #4024:
URL: https://github.com/apache/camel-k/issues/4024#issuecomment-1414438853
> I am all in for this proposal to have the standard maven project with source code instead of injecting it, sound more natural and allows flexibility in the future to tweak the build tool (mvnd?) I don't know the original motivation to have camel-k-runtime to inject the source code (perhaps, to not depend on the pod operator file system). With this move, there will be less code to maintain in camel-k-runtime.
I really don't read this proposal this way, and if it were then it would break any major camel-k use case as you would need to trigger a build every time a kamelet binding/integration changes. We had the option to include the sources back in the day and I'm all for supporting this, but it must be an opt in feature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-k] squakez commented on issue #4024: Make project generation explicit and generic
Posted by "squakez (via GitHub)" <gi...@apache.org>.
squakez commented on issue #4024:
URL: https://github.com/apache/camel-k/issues/4024#issuecomment-1415345561
> @squakez when talking about git, what does that mean exactly ? In my mind, cloning a project from git could and should be supported, pushing to git as a "storage" is very hard to be done right and I don't see the real advantage vs doing an ephemeral project generation in a pipeline.
Just pulling.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-k] essobedo commented on issue #4024: Make project generation explicit and generic
Posted by "essobedo (via GitHub)" <gi...@apache.org>.
essobedo commented on issue #4024:
URL: https://github.com/apache/camel-k/issues/4024#issuecomment-1413825413
Very interesting idea, here are my first questions/remarks
1. What would be the expected content of your git project, to be able to launch it with the `kamel run` command?
2. Don't you believe that allowing github URLs could be seen as an open door to security issues?
3. Does it mean that the dependency management is delegated to Camel JBang? This would mean, we won't have dependencies in IntegrationKit and Build, is that correct?
4. Having the project workspace in a local directory could be an issue in a cluster, indeed, if the master dies after generating the project you could have a problem later resuming the build from another node. Maybe it should rather be compressed and stored into a ConfigMap, shouldn't it?
5. Why have a dedicated CR for this purpose instead of eventually a new trait?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-k] squakez commented on issue #4024: Make project generation explicit and generic
Posted by "squakez (via GitHub)" <gi...@apache.org>.
squakez commented on issue #4024:
URL: https://github.com/apache/camel-k/issues/4024#issuecomment-1414079395
> I'm getting confused, what would be the interest of keeping inferring the dependencies and storing them in the IK and Build if the maven project is generated by Camel JBang or retrieved from a git clone?
Kit reusability and incremental image. We don't want to rebuild a container image every time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-k] squakez commented on issue #4024: Make project generation explicit and generic
Posted by "squakez (via GitHub)" <gi...@apache.org>.
squakez commented on issue #4024:
URL: https://github.com/apache/camel-k/issues/4024#issuecomment-1415341561
> If we wanted to have Camel-K agnostic to the Java Application launched, would you mind listing what we would need to add? Just to allow me to understand how far we are from this potential objective?
Although we may think on something very generic, my idea is to stick focusing on Camel project aspects exclusively. There are many operations beside the deployment that still need to be very Camel specific.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-k] claudio4j commented on issue #4024: Make project generation explicit and generic
Posted by "claudio4j (via GitHub)" <gi...@apache.org>.
claudio4j commented on issue #4024:
URL: https://github.com/apache/camel-k/issues/4024#issuecomment-1414427476
I am all in for this proposal to have the standard maven project with source code instead of injecting it, sound more natural and allows flexibility in the future to tweak the build tool (mvnd?)
I don't know the original motivation to have camel-k-runtime to inject the source code (perhaps, to not depend on the pod operator file system).
With this move, there will be less code to maintain in camel-k-runtime.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-k] squakez commented on issue #4024: Make project generation explicit and generic
Posted by "squakez (via GitHub)" <gi...@apache.org>.
squakez commented on issue #4024:
URL: https://github.com/apache/camel-k/issues/4024#issuecomment-1415383474
> Back to the GitHub URL feature, don't you believe that losing control of what is built, could allow the execution of malicious code at build time that could, for example, use the K8s API to collect sensitive data especially if the build is made from the operator's pod?
I'm not particularly worried, consider any pipeline tool (ie Tekton), they have tasks to pull from git and build accordingly. At some point we may even think to integrate them from any given pipeline technology.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-k] oscerd commented on issue #4024: Make project generation explicit and generic
Posted by "oscerd (via GitHub)" <gi...@apache.org>.
oscerd commented on issue #4024:
URL: https://github.com/apache/camel-k/issues/4024#issuecomment-1414124887
I think you are going a bit too far.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-k] github-actions[bot] commented on issue #4024: Make project generation explicit and generic
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #4024:
URL: https://github.com/apache/camel-k/issues/4024#issuecomment-1650737461
This issue has been automatically marked as stale due to 90 days of inactivity.
It will be closed if no further activity occurs within 15 days.
If you think that’s incorrect or the issue should never stale, please simply write any comment.
Thanks for your contributions!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-k] squakez commented on issue #4024: Make project generation explicit and generic
Posted by "squakez (via GitHub)" <gi...@apache.org>.
squakez commented on issue #4024:
URL: https://github.com/apache/camel-k/issues/4024#issuecomment-1413574328
cc @davsclaus @oscerd @lburgazzoli @claudio4j @essobedo
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-k] squakez commented on issue #4024: Make project generation explicit and generic
Posted by "squakez (via GitHub)" <gi...@apache.org>.
squakez commented on issue #4024:
URL: https://github.com/apache/camel-k/issues/4024#issuecomment-1413991905
> Very interesting idea, here are my first questions/remarks
>
> 1. What would be the expected content of your git project, to be able to launch it with the `kamel run` command?
>
We should agree on the expected format, for now I can think about the outcome of the `camel export`
> 2. Don't you believe that allowing github URLs could be seen as an open door to security issues?
>
Not really. What we need to analyze is how to secure the access to a Git repo (github was an example, it could be any enterprise source code repo).
> 3. Does it mean that the dependency management is delegated to Camel JBang? This would mean, we won't have dependencies in IntegrationKit and Build, is that correct?
>
We still need both IntegrationKit and Build. What I'd like to analyze is the feasibility of `camel` to produce a Maven project from a Camel route (that's what we're already doing in Camel K, but I'd like to see if we can delegate that part)
> 4. Having the project workspace in a local directory could be an issue in a cluster, indeed, if the master dies after generating the project you could have a problem later resuming the build from another node. Maybe it should rather be compressed and stored into a ConfigMap, shouldn't it?
>
This is already the way it works (we store everything in /tmp/): if the operator crash and it's rescheduled, it should restart a Project. However the possibility to use a stateful storage is something we should evaluate, because it intersect with the requirement to build with `pod` strategy (a bit of a discussion in #3831). At this stage I'd prefer not to go in low level analysis, it would be some implementation detail we should understand. I'd love to use something like a Configmap, but the problem is the size limitation we have as it can only store 1 MB data.
> 5. Why have a dedicated CR for this purpose instead of eventually a new trait?
I think it would simplify the generation and the error that can happen when a Project is created. Also it would simplify the kick off of a Build which should watch for a Project to be Ready.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-k] essobedo commented on issue #4024: Make project generation explicit and generic
Posted by "essobedo (via GitHub)" <gi...@apache.org>.
essobedo commented on issue #4024:
URL: https://github.com/apache/camel-k/issues/4024#issuecomment-1414115612
In case, we provide a GitHub URL, I'm wondering what would we have in the Kit?
With this generic approach, since the main Camel logic is delegated to Camel JBang, we could even imagine having a Camel-K agnostic to the Java Application launched. The project could then be proposed in the incubator with a different name and Camel-K would then be an extension of this new project for Camel where we would have the project builder based on Camel JBang and a specific Camel mechanism that would extract the specific info to put into the kit for reusability and incremental image. Maybe I'm going too far but at some point, it could be a possibility.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-k] squakez commented on issue #4024: Make project generation explicit and generic
Posted by "squakez (via GitHub)" <gi...@apache.org>.
squakez commented on issue #4024:
URL: https://github.com/apache/camel-k/issues/4024#issuecomment-1523172572
Parking this for the moment as it may be impacted by the development proposed in #4281
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-k] essobedo commented on issue #4024: Make project generation explicit and generic
Posted by "essobedo (via GitHub)" <gi...@apache.org>.
essobedo commented on issue #4024:
URL: https://github.com/apache/camel-k/issues/4024#issuecomment-1414055218
I'm getting confused, what would be the interest of keeping inferring the dependencies and storing them in the IK and Build if the maven project is generated by Camel JBang or retrieved from a git clone?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-k] essobedo commented on issue #4024: Make project generation explicit and generic
Posted by "essobedo (via GitHub)" <gi...@apache.org>.
essobedo commented on issue #4024:
URL: https://github.com/apache/camel-k/issues/4024#issuecomment-1414129184
If we wanted to have Camel-K agnostic to the Java Application launched, would you mind listing what we would need to add?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-k] oscerd commented on issue #4024: Make project generation explicit and generic
Posted by "oscerd (via GitHub)" <gi...@apache.org>.
oscerd commented on issue #4024:
URL: https://github.com/apache/camel-k/issues/4024#issuecomment-1414117903
Since it's stuff coming from Camel it will eventually under the Apache Camel umbrella, not passing through incubator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org