You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@camel.apache.org by GitBox <gi...@apache.org> on 2021/10/26 13:06:48 UTC
[GitHub] [camel-k] astefanutti opened a new pull request #2719: feat: Integrate Camel readiness checks into Integration readiness condition
astefanutti opened a new pull request #2719:
URL: https://github.com/apache/camel-k/pull/2719
This PR probes the readiness checks exposed by the Camel runtime (Camel Quarkus / MicroProfile Health / SmallRye Health), to reconcile the Integration phase and readiness condition. It aims at surfacing the response from the Camel readiness checks, and exposing useful information, like error messages, so that the Integration status can serve as a single interface for higher level controllers.
As details from the readiness probe responses are not accessible from the Pod(s) status, the readiness probes are directly called by the operator, each time the Integration Pod(s) readiness condition change.
This follows up #2682, so that readiness / error status reconciliation now covers the entire lifecycle of an Integration.
TODO:
- [ ] Decouple liveness and readiness probes in the _container_ trait: #1610.
**Release Note**
```release-note
feat: Integrate Camel readiness checks into Integration readiness condition
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-k] davsclaus commented on pull request #2719: feat: Integrate Camel readiness checks into Integration readiness condition
Posted by GitBox <gi...@apache.org>.
davsclaus commented on pull request #2719:
URL: https://github.com/apache/camel-k/pull/2719#issuecomment-952611078
Yes whether a route can startup vs the consumer is connected is component specific.
Some components have built in their own recovery so they startup the route, and then will automatic self-heal / failover etc. Such as JMS, SQL etc.
And some components does not and fail starting the route if so.
As Luca says then very likely the best solution is to add component level health check, so we can add the logic needed per component (we can have a default readiness that is based on consumer is started).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-k] davsclaus commented on pull request #2719: feat: Integrate Camel readiness checks into Integration readiness condition
Posted by GitBox <gi...@apache.org>.
davsclaus commented on pull request #2719:
URL: https://github.com/apache/camel-k/pull/2719#issuecomment-953923372
Good idea about a trait for health checks. This can then make configuring this easier for end users.
The success threshold is a nice touch, as today when a route/consumer is successful again after 1 attempt its UP. We can add similar threshold as we have for failure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-k] davsclaus commented on pull request #2719: feat: Integrate Camel readiness checks into Integration readiness condition
Posted by GitBox <gi...@apache.org>.
davsclaus commented on pull request #2719:
URL: https://github.com/apache/camel-k/pull/2719#issuecomment-953671818
Okay have a prototype for camel health-check in camel-telegram that reports it as DOWN or UP
You can set the threshold in the standard way today, so either global with a * or by route id, (pattern)
camel.health.config[*].failure-threshold = 10
or via route id
camel.health.config[myRoute].failure-threshold = 10
There is no threshold by default, so we may consider something special for this, or let camel-k auto assign a default value or something.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-k] astefanutti commented on pull request #2719: feat: Integrate Camel readiness checks into Integration readiness condition
Posted by GitBox <gi...@apache.org>.
astefanutti commented on pull request #2719:
URL: https://github.com/apache/camel-k/pull/2719#issuecomment-954759554
I've updated the logic to call the health probe via the API server proxy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-k] astefanutti commented on pull request #2719: feat: Report runtime health checks into Integration readiness condition
Posted by GitBox <gi...@apache.org>.
astefanutti commented on pull request #2719:
URL: https://github.com/apache/camel-k/pull/2719#issuecomment-971382704
I think it's ready. We'll be able to iterate as soon as we upgrade to Camel 3.13+, to leverage the new features developed by @davsclaus, add more test cases, and fix bugs 😇.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-k] davsclaus edited a comment on pull request #2719: feat: Integrate Camel readiness checks into Integration readiness condition
Posted by GitBox <gi...@apache.org>.
davsclaus edited a comment on pull request #2719:
URL: https://github.com/apache/camel-k/pull/2719#issuecomment-953923372
Good idea about a trait for health checks. This can then make configuring this easier for end users.
The success threshold is a nice touch, as today when a route/consumer is successful again after 1 attempt its UP. We can add similar threshold as we have for failure.
https://issues.apache.org/jira/browse/CAMEL-17143
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-k] astefanutti edited a comment on pull request #2719: feat: Integrate Camel readiness checks into Integration readiness condition
Posted by GitBox <gi...@apache.org>.
astefanutti edited a comment on pull request #2719:
URL: https://github.com/apache/camel-k/pull/2719#issuecomment-951991555
While that PR proposes to use the Camel readiness checks as a channel / interface to peek into the runtime status, I've identified the following open points / gaps:
* `camel-microprofile-health`: The error details are not propagated during the conversion from the Camel health check result (`org.apache.camel.health.HealthCheck.Result`) to the MP Health response (`org.eclipse.microprofile.health.HealthCheckResponse`): https://github.com/apache/camel/blob/80b92e3624ae5db59a1a24a441f1b10b39eaa1a5/components/camel-microprofile/camel-microprofile-health/src/main/java/org/apache/camel/microprofile/health/AbstractCamelMicroProfileHealthCheck.java#L46
* `camel-health`: It seems the default `RouteHealthCheck` generally associates the check status to the route status, that is `UP` (resp. `DOWN`) when the route is started (resp. when the route is stopped). So, by default, with a consumer like `from("telegram?authorizationToken=WRONG_TOKEN")`, the route starts, so health is `UP`, despite the connection keeps failing. A `RoutePolicy` that stops the route after an number of retries may be a solution. Also the error details do not seem to be propagated: https://github.com/apache/camel/blob/1b374423fb9371075184d5ab2d9c3b15015b2334/core/camel-health/src/main/java/org/apache/camel/impl/health/RouteHealthCheck.java#L45.
@lburgazzoli @nicolaferraro @jamesnetherton @davsclaus would you be kind to shed your expertise / opinion on this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-k] davsclaus edited a comment on pull request #2719: feat: Integrate Camel readiness checks into Integration readiness condition
Posted by GitBox <gi...@apache.org>.
davsclaus edited a comment on pull request #2719:
URL: https://github.com/apache/camel-k/pull/2719#issuecomment-952614464
@astefanutti for the 1st bullet we need a JIRA ticket about this. Then maybe @jamesnetherton can take a look, seems like we can copy over the message/error to MP.
There are also some other details such as
builder.detail("route.id", route.getId());
builder.detail("route.status", status.name());
builder.detail("route.context.name", context.getName());
And then some general information for counters, eg number of checks and failures in row etc. See the base class source code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-k] astefanutti commented on pull request #2719: feat: Integrate Camel readiness checks into Integration readiness condition
Posted by GitBox <gi...@apache.org>.
astefanutti commented on pull request #2719:
URL: https://github.com/apache/camel-k/pull/2719#issuecomment-953706548
> Okay have a prototype for camel health-check in camel-telegram that reports it as DOWN or UP
Wow that was fast! There is also @croway that has a PoC for propagating the error details in camel-microprofile-health.
> You can set the threshold in the standard way today, so either global with a * or by route id, (pattern)
>
> camel.health.config[*].failure-threshold = 10
>
> or via route id
>
> camel.health.config[myRoute].failure-threshold = 10
>
> There is no threshold by default, so we may consider something special for this, or let camel-k auto assign a default value or something.
Yes, I think we could have a _health_ trait that would provide users the ability to configure theses, and auto assign sensible defaults there. I think encapsulating the health configuration into a dedicated trait would also help disentangling the container trait.
Also it could be useful to have a `success-threshold` parameter, akin to Kubernetes health probes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-k] davsclaus commented on pull request #2719: feat: Integrate Camel readiness checks into Integration readiness condition
Posted by GitBox <gi...@apache.org>.
davsclaus commented on pull request #2719:
URL: https://github.com/apache/camel-k/pull/2719#issuecomment-954762172
Another ticket to allow to control Camel to auto stop un-healthy routes
https://issues.apache.org/jira/browse/CAMEL-17148
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-k] astefanutti edited a comment on pull request #2719: feat: Integrate Camel readiness checks into Integration readiness condition
Posted by GitBox <gi...@apache.org>.
astefanutti edited a comment on pull request #2719:
URL: https://github.com/apache/camel-k/pull/2719#issuecomment-952818715
> Maybe tunneling the request e.g. via the apiserver proxy could make it work in any configuration. Wdyt @astefanutti ?
Ah right, that's a very good point. I took the shortest path and mimicked what the kubelet does, but the operator Pod is indeed subjected to network policies. Let me rework it based on the API server proxy. Ultimately, it would be possible to have the reason reported into the Pod readiness condition directly, alike the termination message, but in the interim, relying on the API server proxy should do it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-k] jamesnetherton commented on pull request #2719: feat: Integrate Camel readiness checks into Integration readiness condition
Posted by GitBox <gi...@apache.org>.
jamesnetherton commented on pull request #2719:
URL: https://github.com/apache/camel-k/pull/2719#issuecomment-952020467
> The error details are not propagated during the conversion from the Camel health check result to the MP Health response
Yeah, it'd be nice to implement that. Should be a simple enough enhancement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-k] astefanutti commented on pull request #2719: feat: Integrate Camel readiness checks into Integration readiness condition
Posted by GitBox <gi...@apache.org>.
astefanutti commented on pull request #2719:
URL: https://github.com/apache/camel-k/pull/2719#issuecomment-951991555
While that PR proposes to use the Camel readiness checks as a channel / interface to peek into the runtime status, I've identified the following open points / gaps:
* `camel-microprofile-health`: The error details are not propagated during the conversion from the Camel health check result (`org.apache.camel.health.HealthCheck.Result`) to the MP Health response (`org.eclipse.microprofile.health.HealthCheckResponse`): https://github.com/apache/camel/blob/80b92e3624ae5db59a1a24a441f1b10b39eaa1a5/components/camel-microprofile/camel-microprofile-health/src/main/java/org/apache/camel/microprofile/health/AbstractCamelMicroProfileHealthCheck.java#L46
* `camel-health`: It seems the default `RouteHealthCheck` generally associates the check status to the route status, that is `UP` (resp. `DOWN`) when the route is started (resp. when the route is stopped). So, by default, with the consumer like `from("telegram?authorizationToken=WRONG_TOKEN")`, the route starts, so health is `UP`, despite the connection keeps failing. A `RoutePolicy` that stops the route after an number of retries may be a solution. Also the error details do not seem to be propagated: https://github.com/apache/camel/blob/1b374423fb9371075184d5ab2d9c3b15015b2334/core/camel-health/src/main/java/org/apache/camel/impl/health/RouteHealthCheck.java#L45.
@lburgazzoli @nicolaferraro @jamesnetherton @davsclaus would you be kind to shed your expertise / opinion on this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-k] astefanutti commented on pull request #2719: feat: Integrate Camel readiness checks into Integration readiness condition
Posted by GitBox <gi...@apache.org>.
astefanutti commented on pull request #2719:
URL: https://github.com/apache/camel-k/pull/2719#issuecomment-952818715
> Maybe tunneling the request e.g. via the apiserver proxy could make it work in any configuration. Wdyt @astefanutti ?
Ah right, that's a very good point. I took the shortest path and mimic what the kubelet does, but the operator Pod is indeed subjected to network policies. Let me rework it based on the API server proxy. Ultimately, it would be possible to have the reason reported into the Pod readiness condition directly, alike the termination message, but in the interim, relying on the API server proxy should do it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-k] davsclaus commented on pull request #2719: feat: Integrate Camel readiness checks into Integration readiness condition
Posted by GitBox <gi...@apache.org>.
davsclaus commented on pull request #2719:
URL: https://github.com/apache/camel-k/pull/2719#issuecomment-952612483
There are some tickets already for component level health checks
https://issues.apache.org/jira/browse/CAMEL-16975
https://issues.apache.org/jira/browse/CAMEL-16976
https://issues.apache.org/jira/browse/CAMEL-15133
And also some way of checking the caused error in the consumer if its connectivity error or a business error
https://issues.apache.org/jira/browse/CAMEL-16977
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-k] astefanutti commented on pull request #2719: feat: Integrate Camel readiness checks into Integration readiness condition
Posted by GitBox <gi...@apache.org>.
astefanutti commented on pull request #2719:
URL: https://github.com/apache/camel-k/pull/2719#issuecomment-952759236
@davsclaus, thanks a lot, these tickets capture exactly what would be needed 👍🏼.
I've created [CAMEL-17138](https://issues.apache.org/jira/browse/CAMEL-17138) to track the propagation of the Camel health check result details into the MP Health responses.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-k] astefanutti commented on pull request #2719: feat: Integrate Camel readiness checks into Integration readiness condition
Posted by GitBox <gi...@apache.org>.
astefanutti commented on pull request #2719:
URL: https://github.com/apache/camel-k/pull/2719#issuecomment-964104224
The new _health_ trait and the fix for #1610 is addressed via #2740.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-k] astefanutti merged pull request #2719: feat: Report runtime health checks into Integration readiness condition
Posted by GitBox <gi...@apache.org>.
astefanutti merged pull request #2719:
URL: https://github.com/apache/camel-k/pull/2719
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-k] davsclaus commented on pull request #2719: feat: Integrate Camel readiness checks into Integration readiness condition
Posted by GitBox <gi...@apache.org>.
davsclaus commented on pull request #2719:
URL: https://github.com/apache/camel-k/pull/2719#issuecomment-952614464
@astefanutti for the 1st bullet we need a JIRA ticket about this. Then maybe @jamesnetherton can take a look, seems like we can copy over the message/error to MP
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-k] nicolaferraro commented on pull request #2719: feat: Integrate Camel readiness checks into Integration readiness condition
Posted by GitBox <gi...@apache.org>.
nicolaferraro commented on pull request #2719:
URL: https://github.com/apache/camel-k/pull/2719#issuecomment-952788067
I'm thinking to a multi-tenant cluster with strict network policies that disallow cross-namespace connections, and a Camel K operator deployed globally. It seems from the code that this would result in a kind of "health unavailable" (if the connection error is catched).
Maybe tunneling the request e.g. via the apiserver proxy could make it work in any configuration. Wdyt @astefanutti ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-k] lburgazzoli commented on pull request #2719: feat: Integrate Camel readiness checks into Integration readiness condition
Posted by GitBox <gi...@apache.org>.
lburgazzoli commented on pull request #2719:
URL: https://github.com/apache/camel-k/pull/2719#issuecomment-952052482
> * `camel-health`: It seems the default `RouteHealthCheck` generally associates the check status to the route status, that is `UP` (resp. `DOWN`) when the route is started (resp. when the route is stopped). So, by default, with the consumer like `from("telegram?authorizationToken=WRONG_TOKEN")`, the route starts, so health is `UP`, despite the connection keeps failing.
This is a long standing issue and I think we need to start adding health check at component level
> Also the error details do not seem to be propagated
This needs to be fixed too
Mind opening some issue on camel ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org