[jira] [Updated] (CAMEL-24051) camel-test-infra: add an "observability" service (Prometheus + VictoriaTraces + VictoriaLogs + Perses, Apache-2.0 stack) for local dev and E2E, mirroring OpenShift's observability stack

["Federico Mariani (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/CAMEL-24051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Federico Mariani updated CAMEL-24051:
-------------------------------------
    Description: 
h3. Problem / Motivation

Camel's local dev loop ({{camel run --dev}}) and the Kubernetes export path already wire up {{camel-observability-services}} (Micrometer/Prometheus metrics, health checks, OpenTelemetry tracing via the bundled {{camel-opentelemetry2}}) when targeting Kubernetes/OpenShift. However, there is no local backend to receive or visualize that telemetry: {{camel infra run}} can start backing services like Postgres or Artemis, but nothing analogous exists for observability.

As a result, developers only find out whether traces, metrics, and logs actually work correctly (correct span correlation, meaningful dashboards, sane metric cardinality, etc.) once the application is deployed to a real OpenShift cluster, which typically provides the Red Hat build of OpenTelemetry, the Tempo Operator, Loki/OpenShift Logging, and the Cluster Observability Operator (Grafana). Telemetry problems surface late, in a shared cluster environment, rather than at dev/E2E time on a laptop or in CI.

h3. Licensing note (why not grafana/otel-lgtm)

The obvious candidate for this is Grafana's {{grafana/otel-lgtm}} all-in-one image. It was *ruled out*: Grafana, Loki, and Tempo were relicensed from Apache 2.0 to AGPLv3 in 2021. AGPLv3 is Category X on the ASF's third-party license policy, so an ASF project cannot depend on it even as opt-in test/dev tooling without a real legal-discuss@apache.org sign-off, and Red Hat's OSPO treats AGPL as off-limits even more strictly, including for test-only tooling in downstream builds. Only Prometheus and the OpenTelemetry Collector in that bundle are actually Apache 2.0.

h3. Proposed Solution -- Apache-2.0-only stack (validated with a working PoC, see below)

Use an equivalent stack built entirely from Apache-2.0-licensed components:
* *Metrics*: Prometheus (unchanged -- already Apache 2.0, already the {{camel-observability-services}} default via {{/observe/metrics}}).
* *Traces*: https://github.com/VictoriaMetrics/VictoriaTraces[VictoriaTraces] (Apache 2.0) -- ingests OTLP natively and exposes a Jaeger Query Service-compatible JSON API. Functionally a drop-in swap for Tempo.
* *Logs*: https://github.com/VictoriaMetrics/VictoriaLogs[VictoriaLogs] (Apache 2.0) -- LogsQL query language, accepts OTLP logs directly via {{/insert/opentelemetry/v1/logs}}. Swap for Loki.
* *Dashboard*: https://perses.dev[Perses] (Apache 2.0, CNCF sandbox project) -- dashboard-as-code tool backed by Amadeus/SAP/Red Hat, and notably *already powers OpenShift's own new traces UI*, which makes it a much safer downstream bet than Grafana. Its default plugin set already ships Prometheus, Jaeger, and VictoriaLogs datasource kinds with no extra plugin install.

Adding this also revisits the fate of the existing {{camel-test-infra-jaeger}} module (added in CAMEL-23641): since VictoriaTraces exposes a Jaeger-compatible query API, {{camel-test-infra-jaeger}} does *not* need to be deprecated as originally floated -- it can keep serving its current role, or the new service can point at it directly, instead of being replaced outright. This should be decided during implementation, not assumed here.

h3. Signal-level design rationale (matching real K8s/OCP behavior where it matters)

* *Traces -- push*: matches OCP's own model (Red Hat build of OpenTelemetry pushes OTLP spans to a Collector). No divergence between local and cluster behavior.
* *Metrics -- pull*: OCP's real default is Prometheus scraping via {{ServiceMonitor}}/{{PodMonitor}}, not app-side push. {{camel-observability-services}}' existing Prometheus-pull default already matches this -- the local service's job is to supply a scrape target pointed at the running app, mirroring what a {{ServiceMonitor}} does on-cluster, *not* to switch metrics to OTLP push.
* *Logs -- local shortcut, not literal OCP parity*: real OpenShift logging does not have the app push logs at all -- Vector tails container stdout/stderr and ships to LokiStack; Loki itself still doesn't natively ingest OTLP. Locally, pushing OTLP logs straight from the OpenTelemetry Java agent's logback/log4j2 instrumentation into VictoriaLogs is validated and simple, but it is a different data path than production and should be documented as a pragmatic local-dev shortcut, not a faithful mirror of the cluster's log-collection mechanism.
* *Trace/log correlation*: validated to work *automatically* via the OTel Java agent's logback-appender instrumentation ({{-Dotel.instrumentation.logback-appender.enabled=true}}) -- every ingested log line carries real {{trace_id}}/{{span_id}} with zero app-side MDC configuration. The {{camel-mdc}}-based approach documented in {{camel-opentelemetry2}}'s docs is not required for this path.

h3. Empirical validation (PoC results)

A working proof-of-concept was built and torn down; all 4 signals (metrics, traces, logs, and trace/log correlation) were confirmed working end-to-end with real data from a real {{camel-observability-services}}-enabled Camel route, and Perses accepted all 3 datasource definitions with no schema errors. Key findings to carry into implementation:

* VictoriaTraces' OTLP gRPC receiver is *disabled by default* -- must pass {{-otlpGRPCListenAddr=:4317 -otlpGRPC.tls=false}}.
* VictoriaTraces' Jaeger-compatible API is served at {{/select/jaeger/api/...}}, not the bare {{/api/...}} its docs imply.
* *Port collision*: Perses defaults to port 8080, the same default port used by Camel/Spring Boot/Quarkus apps. Fix is to remap Perses, not the app -- confirmed via {{-web.listen-address :3000}} (also verified live), keeping the app on its normal default port. Suggest defaulting the test-infra service to port 3000 for Perses, matching the Grafana/otel-lgtm convention devs may already be used to.
* Minimal working docker commands (see also PoC files, not retained beyond this session):
{code}
docker network create camel-obs
docker run -d --name victoria-traces --network camel-obs -p 8428:8428 -p 4317:4317 \
  victoriametrics/victoria-traces:latest -httpListenAddr=:8428 -otlpGRPCListenAddr=:4317 -otlpGRPC.tls=false
docker run -d --name victoria-logs --network camel-obs -p 9428:9428 \
  victoriametrics/victoria-logs:latest -httpListenAddr=:9428
docker run -d --name prometheus --network camel-obs -p 9090:9090 \
  -v "$(pwd)/prometheus.yml:/etc/prometheus/prometheus.yml:ro" --add-host=host.docker.internal:host-gateway \
  prom/prometheus:latest
docker run -d --name perses --network camel-obs -p 3000:3000 \
  persesdev/perses:latest -web.listen-address :3000
{code}
* A minimal {{camel-observability-services}}-enabled JBang app, run via {{camel run}} (a bare {{RouteBuilder}} has no {{main}}, so plain {{jbang run}} does not work), with the OpenTelemetry Java agent supplying traces+logs export, confirmed all of: Prometheus scraping {{/observe/metrics}}, a real Camel-level {{timer}} span in VictoriaTraces, and OTLP log ingestion into VictoriaLogs correlated back to that trace.

h3. Scope

* New test-infra module/service(s) following existing test-infra patterns (e.g. {{*InfraService}}, {{*LocalContainerService}}, {{*ServiceFactory}}), composing the *4 separate containers* above -- there is no single prebuilt image bundling Prometheus + VictoriaTraces + VictoriaLogs + Perses the way {{grafana/otel-lgtm}} bundles its stack.
* Wire the new service into the {{camel-jbang-core}} {{infra}} command so {{camel infra run observability}} works like other {{camel infra run <service>}} commands, including supplying the Prometheus scrape-target wiring against the running app.
* Decide the relationship to the existing {{camel-test-infra-jaeger}} module (coexist vs. supersede) as part of implementation.
* Out of scope for this ticket: changes to {{camel-observability-services}}' default export behavior itself (e.g. any future default-on MDC/log-correlation improvement is a separate, runtime-behavior-changing ticket), and cluster-side OpenShift/Kubernetes tooling (traits, {{camel kubernetes doctor}}, PodMonitor/PrometheusRule generation).

h3. Open items

* Perses' live query-through-datasource-proxy path was not confirmed during the PoC (datasource definitions were accepted, but querying data through Perses' proxy returned errors under the guessed path conventions) -- needs checking against Perses source/docs during implementation.


  was:
h3. Problem / Motivation

Camel's local dev loop ({{camel run --dev}}) and the Kubernetes export path already wire up {{camel-observability-services}} (Micrometer/Prometheus metrics, health checks, OpenTelemetry tracing) when targeting Kubernetes/OpenShift. However, there is no local backend to receive or visualize that telemetry: {{camel infra run}} can start backing services like Postgres or Artemis, but nothing analogous exists for observability.

As a result, developers only find out whether traces, metrics, and logs actually work correctly (correct span correlation, meaningful dashboards, sane metric cardinality, etc.) once the application is deployed to a real OpenShift cluster, which typically provides the Red Hat build of OpenTelemetry, the Tempo Operator, Loki/OpenShift Logging, and the Cluster Observability Operator (Grafana). Telemetry problems surface late, in a shared cluster environment, rather than at dev/E2E time on a laptop or in CI.

h3. Proposed Solution

Add a new {{camel-test-infra}} service, invocable as {{camel infra run observability}} (same UX as the existing {{camel infra run postgres}} / {{camel infra run artemis}} services), that starts a local all-in-one observability stack via Testcontainers -- for example the {{grafana/otel-lgtm}} image, which bundles an OTel collector plus Prometheus, Tempo, Loki, and Grafana.

The service should be wired to the same OTLP endpoint / ports / paths that {{camel-observability-services}} already exports (metrics, health, tracing), so that the dashboards, traces, and logs a developer sees locally when running {{camel run --dev}} against this service are the same shape as what they will see after deploying to OpenShift. This closes the local-vs-production observability gap and gives a fast, zero-cluster feedback loop for validating telemetry during the inner dev loop and in E2E test workflows.

h3. Scope

* New test-infra module/service following the existing test-infra patterns (e.g. an {{*InfraService}}, {{*LocalContainerService}}, and {{*ServiceFactory}} for the {{grafana/otel-lgtm}} container).
* Wire the new service into the {{camel-jbang-core}} {{infra}} command so {{camel infra run observability}} works like other {{camel infra run <service>}} commands.
* Out of scope for this ticket: any changes to {{camel-observability-services}} itself, and cluster-side OpenShift/Kubernetes tooling (traits, {{camel kubernetes doctor}}, PodMonitor/PrometheusRule generation) -- those are separate, larger efforts.

        Summary: camel-test-infra: add an "observability" service (Prometheus + VictoriaTraces + VictoriaLogs + Perses, Apache-2.0 stack) for local dev and E2E, mirroring OpenShift's observability stack  (was: camel-test-infra: add an "observability" service (Grafana/Tempo/Loki OTel-LGTM stack) for local dev and E2E, mirroring OpenShift's observability stack)

> camel-test-infra: add an "observability" service (Prometheus + VictoriaTraces + VictoriaLogs + Perses, Apache-2.0 stack) for local dev and E2E, mirroring OpenShift's observability stack
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: CAMEL-24051
>                 URL: https://issues.apache.org/jira/browse/CAMEL-24051
>             Project: Camel
>          Issue Type: New Feature
>          Components: test-infra
>            Reporter: Federico Mariani
>            Assignee: Federico Mariani
>            Priority: Major
>
> h3. Problem / Motivation
> Camel's local dev loop ({{camel run --dev}}) and the Kubernetes export path already wire up {{camel-observability-services}} (Micrometer/Prometheus metrics, health checks, OpenTelemetry tracing via the bundled {{camel-opentelemetry2}}) when targeting Kubernetes/OpenShift. However, there is no local backend to receive or visualize that telemetry: {{camel infra run}} can start backing services like Postgres or Artemis, but nothing analogous exists for observability.
> As a result, developers only find out whether traces, metrics, and logs actually work correctly (correct span correlation, meaningful dashboards, sane metric cardinality, etc.) once the application is deployed to a real OpenShift cluster, which typically provides the Red Hat build of OpenTelemetry, the Tempo Operator, Loki/OpenShift Logging, and the Cluster Observability Operator (Grafana). Telemetry problems surface late, in a shared cluster environment, rather than at dev/E2E time on a laptop or in CI.
> h3. Licensing note (why not grafana/otel-lgtm)
> The obvious candidate for this is Grafana's {{grafana/otel-lgtm}} all-in-one image. It was *ruled out*: Grafana, Loki, and Tempo were relicensed from Apache 2.0 to AGPLv3 in 2021. AGPLv3 is Category X on the ASF's third-party license policy, so an ASF project cannot depend on it even as opt-in test/dev tooling without a real legal-discuss@apache.org sign-off, and Red Hat's OSPO treats AGPL as off-limits even more strictly, including for test-only tooling in downstream builds. Only Prometheus and the OpenTelemetry Collector in that bundle are actually Apache 2.0.
> h3. Proposed Solution -- Apache-2.0-only stack (validated with a working PoC, see below)
> Use an equivalent stack built entirely from Apache-2.0-licensed components:
> * *Metrics*: Prometheus (unchanged -- already Apache 2.0, already the {{camel-observability-services}} default via {{/observe/metrics}}).
> * *Traces*: https://github.com/VictoriaMetrics/VictoriaTraces[VictoriaTraces] (Apache 2.0) -- ingests OTLP natively and exposes a Jaeger Query Service-compatible JSON API. Functionally a drop-in swap for Tempo.
> * *Logs*: https://github.com/VictoriaMetrics/VictoriaLogs[VictoriaLogs] (Apache 2.0) -- LogsQL query language, accepts OTLP logs directly via {{/insert/opentelemetry/v1/logs}}. Swap for Loki.
> * *Dashboard*: https://perses.dev[Perses] (Apache 2.0, CNCF sandbox project) -- dashboard-as-code tool backed by Amadeus/SAP/Red Hat, and notably *already powers OpenShift's own new traces UI*, which makes it a much safer downstream bet than Grafana. Its default plugin set already ships Prometheus, Jaeger, and VictoriaLogs datasource kinds with no extra plugin install.
> Adding this also revisits the fate of the existing {{camel-test-infra-jaeger}} module (added in CAMEL-23641): since VictoriaTraces exposes a Jaeger-compatible query API, {{camel-test-infra-jaeger}} does *not* need to be deprecated as originally floated -- it can keep serving its current role, or the new service can point at it directly, instead of being replaced outright. This should be decided during implementation, not assumed here.
> h3. Signal-level design rationale (matching real K8s/OCP behavior where it matters)
> * *Traces -- push*: matches OCP's own model (Red Hat build of OpenTelemetry pushes OTLP spans to a Collector). No divergence between local and cluster behavior.
> * *Metrics -- pull*: OCP's real default is Prometheus scraping via {{ServiceMonitor}}/{{PodMonitor}}, not app-side push. {{camel-observability-services}}' existing Prometheus-pull default already matches this -- the local service's job is to supply a scrape target pointed at the running app, mirroring what a {{ServiceMonitor}} does on-cluster, *not* to switch metrics to OTLP push.
> * *Logs -- local shortcut, not literal OCP parity*: real OpenShift logging does not have the app push logs at all -- Vector tails container stdout/stderr and ships to LokiStack; Loki itself still doesn't natively ingest OTLP. Locally, pushing OTLP logs straight from the OpenTelemetry Java agent's logback/log4j2 instrumentation into VictoriaLogs is validated and simple, but it is a different data path than production and should be documented as a pragmatic local-dev shortcut, not a faithful mirror of the cluster's log-collection mechanism.
> * *Trace/log correlation*: validated to work *automatically* via the OTel Java agent's logback-appender instrumentation ({{-Dotel.instrumentation.logback-appender.enabled=true}}) -- every ingested log line carries real {{trace_id}}/{{span_id}} with zero app-side MDC configuration. The {{camel-mdc}}-based approach documented in {{camel-opentelemetry2}}'s docs is not required for this path.
> h3. Empirical validation (PoC results)
> A working proof-of-concept was built and torn down; all 4 signals (metrics, traces, logs, and trace/log correlation) were confirmed working end-to-end with real data from a real {{camel-observability-services}}-enabled Camel route, and Perses accepted all 3 datasource definitions with no schema errors. Key findings to carry into implementation:
> * VictoriaTraces' OTLP gRPC receiver is *disabled by default* -- must pass {{-otlpGRPCListenAddr=:4317 -otlpGRPC.tls=false}}.
> * VictoriaTraces' Jaeger-compatible API is served at {{/select/jaeger/api/...}}, not the bare {{/api/...}} its docs imply.
> * *Port collision*: Perses defaults to port 8080, the same default port used by Camel/Spring Boot/Quarkus apps. Fix is to remap Perses, not the app -- confirmed via {{-web.listen-address :3000}} (also verified live), keeping the app on its normal default port. Suggest defaulting the test-infra service to port 3000 for Perses, matching the Grafana/otel-lgtm convention devs may already be used to.
> * Minimal working docker commands (see also PoC files, not retained beyond this session):
> {code}
> docker network create camel-obs
> docker run -d --name victoria-traces --network camel-obs -p 8428:8428 -p 4317:4317 \
>   victoriametrics/victoria-traces:latest -httpListenAddr=:8428 -otlpGRPCListenAddr=:4317 -otlpGRPC.tls=false
> docker run -d --name victoria-logs --network camel-obs -p 9428:9428 \
>   victoriametrics/victoria-logs:latest -httpListenAddr=:9428
> docker run -d --name prometheus --network camel-obs -p 9090:9090 \
>   -v "$(pwd)/prometheus.yml:/etc/prometheus/prometheus.yml:ro" --add-host=host.docker.internal:host-gateway \
>   prom/prometheus:latest
> docker run -d --name perses --network camel-obs -p 3000:3000 \
>   persesdev/perses:latest -web.listen-address :3000
> {code}
> * A minimal {{camel-observability-services}}-enabled JBang app, run via {{camel run}} (a bare {{RouteBuilder}} has no {{main}}, so plain {{jbang run}} does not work), with the OpenTelemetry Java agent supplying traces+logs export, confirmed all of: Prometheus scraping {{/observe/metrics}}, a real Camel-level {{timer}} span in VictoriaTraces, and OTLP log ingestion into VictoriaLogs correlated back to that trace.
> h3. Scope
> * New test-infra module/service(s) following existing test-infra patterns (e.g. {{*InfraService}}, {{*LocalContainerService}}, {{*ServiceFactory}}), composing the *4 separate containers* above -- there is no single prebuilt image bundling Prometheus + VictoriaTraces + VictoriaLogs + Perses the way {{grafana/otel-lgtm}} bundles its stack.
> * Wire the new service into the {{camel-jbang-core}} {{infra}} command so {{camel infra run observability}} works like other {{camel infra run <service>}} commands, including supplying the Prometheus scrape-target wiring against the running app.
> * Decide the relationship to the existing {{camel-test-infra-jaeger}} module (coexist vs. supersede) as part of implementation.
> * Out of scope for this ticket: changes to {{camel-observability-services}}' default export behavior itself (e.g. any future default-on MDC/log-correlation improvement is a separate, runtime-behavior-changing ticket), and cluster-side OpenShift/Kubernetes tooling (traits, {{camel kubernetes doctor}}, PodMonitor/PrometheusRule generation).
> h3. Open items
> * Perses' live query-through-datasource-proxy path was not confirmed during the PoC (datasource definitions were accepted, but querying data through Perses' proxy returned errors under the guessed path conventions) -- needs checking against Perses source/docs during implementation.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)