You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@camel.apache.org by "mertdotcc (via GitHub)" <gi...@apache.org> on 2023/03/28 05:01:27 UTC
[GitHub] [camel-k] mertdotcc opened a new issue, #4173: Horizontal scaling ability for the operators
mertdotcc opened a new issue, #4173:
URL: https://github.com/apache/camel-k/issues/4173
(I am aware that there are already discussions and existing issues regarding multi-tenancy and multi-operator configuration, and that these have been added to the roadmap for version 2 as well. I am opening this issue since I think it's slightly different than what's open out there. We can merge this in with an existing issue, or close it, I'd be fine with it. I just want to start a discussion and get different people's opinions.)
Right now we have 8-10 different integrations we are deploying with a native image. We have a memory-optimized node with `16Gi` of memory and we use taints and annotations to make sure that only the operator runs in that node and no other pod.
We first tried running our operator with `--operator-resources requests.memory=4096Mi` and `--operator-resources limits.memory=4096Mi` flags (this was before we created our operator-only memory-optimized node) as per the documentation [suggests](https://github.com/apache/camel-k/blob/6f0037aef8a87d56509f2168cb510cd08d564d46/resources/traits.yaml#L1237), but the native image build times were dreadfully slow for our needs.
Then we created our memory-optimized node and deployed our operator with `--operator-resources requests.memory=8192Mi` and `--operator-resources limits.memory=12288Mi` flags.
The current situation is that the operator uses basically all the memory we throw at it, in this case, all 12 gigs. We acknowledge that the native image build process is resource and time intensive and we can live with that fact. The problem starts when 2 developers who work on 2 different integrations deploy their changes and our ArgoCD picks those changes up and initiates new builds for those 2 integrations... I am not sure whether there is a queue mechanism or whether those 2 native image builds are happening at the same time (because the operator logs get scrambled at this moment) but what we would ideally like to see is the following: we have one operator running at all times (or even better, 2-3 operators running at the same time for HA purposes - when that is supported of course) but whenever there are 2-3 or more images that need to be built at any given time, a spot node instance is created, an operator is deployed there, the build runs and finishes there, the image gets
pushed into our registry, and then the node instance gets killed. This would not only be super cost-effective (always makes it easier to sell to upper management) but also ensure that when there are 2 images that need to be built (or are being built) the wait time doesn't increase from ~10 minutes to 20+ minutes (as it currently happens with our use case).
What are your thoughts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-k] squakez commented on issue #4173: Horizontal scaling ability for operators
Posted by "squakez (via GitHub)" <gi...@apache.org>.
squakez commented on issue #4173:
URL: https://github.com/apache/camel-k/issues/4173#issuecomment-1486376744
I think that version 2.x approach is going to radically change this (and hopefully solve it). In Camel K 2 a Builder `pod` is going to be initiated when a native image is required. There won't be the possibility to run a native build in an operator `routine` any longer (this was required to break the dependency between the operator and the Quarkus build tooling).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-k] squakez commented on issue #4173: Horizontal scaling ability for operators
Posted by "squakez (via GitHub)" <gi...@apache.org>.
squakez commented on issue #4173:
URL: https://github.com/apache/camel-k/issues/4173#issuecomment-1608893412
I think we can close this as the horizontally scalability is given by the fact that a native build is forced now to run externally with `pod` strategy. Non native can still run in `routine` or the user can chose to run them in a `pod` as well. Feel free to reopen if there is still something pending.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-k] squakez closed issue #4173: Horizontal scaling ability for operators
Posted by "squakez (via GitHub)" <gi...@apache.org>.
squakez closed issue #4173: Horizontal scaling ability for operators
URL: https://github.com/apache/camel-k/issues/4173
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-k] squakez commented on issue #4173: Horizontal scaling ability for operators
Posted by "squakez (via GitHub)" <gi...@apache.org>.
squakez commented on issue #4173:
URL: https://github.com/apache/camel-k/issues/4173#issuecomment-1486836320
> Interesting. But the operator will keep building the non-native images, correct?
Of course. Feel free to give it yourself a try with 2.0.0 nightly: https://github.com/apache/camel-k/releases/tag/2.0.0-nightly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-k] mertdotcc commented on issue #4173: Horizontal scaling ability for operators
Posted by "mertdotcc (via GitHub)" <gi...@apache.org>.
mertdotcc commented on issue #4173:
URL: https://github.com/apache/camel-k/issues/4173#issuecomment-1486777182
Interesting. But the operator will keep building the non-native images, correct?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-k] github-actions[bot] commented on issue #4173: Horizontal scaling ability for operators
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #4173:
URL: https://github.com/apache/camel-k/issues/4173#issuecomment-1608520740
This issue has been automatically marked as stale due to 90 days of inactivity.
It will be closed if no further activity occurs within 15 days.
If you think that’s incorrect or the issue should never stale, please simply write any comment.
Thanks for your contributions!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org