[jira] [Commented] (CAMEL-24051) camel-test-infra: add an "observability" service (Prometheus + VictoriaTraces + VictoriaLogs + Perses, Apache-2.0 stack) for local dev and E2E, mirroring OpenShift's observability stack

["Federico Mariani (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CAMEL-24051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18096143#comment-18096143 ] 

Federico Mariani commented on CAMEL-24051:
------------------------------------------

Good point, but I still think the observability test infra service makes some sense, in particular:
* Is much closer to a production (k8s/ocp) scenario
* While TUI traces and jaeger already handle the traces, with the observability test infra we can:
** Validate that the metrics integration with prometheus works
** Validate OTLP log export
** And Perses, a unified UI

> camel-test-infra: add an "observability" service (Prometheus + VictoriaTraces + VictoriaLogs + Perses, Apache-2.0 stack) for local dev and E2E, mirroring OpenShift's observability stack
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: CAMEL-24051
>                 URL: https://issues.apache.org/jira/browse/CAMEL-24051
>             Project: Camel
>          Issue Type: New Feature
>          Components: test-infra
>            Reporter: Federico Mariani
>            Assignee: Federico Mariani
>            Priority: Major
>
> h3. Problem / Motivation
> Camel's local dev loop ({{camel run --dev}}) and the Kubernetes export path already wire up {{camel-observability-services}} (Micrometer/Prometheus metrics, health checks, OpenTelemetry tracing via the bundled {{camel-opentelemetry2}}) when targeting Kubernetes/OpenShift. However, there is no local backend to receive or visualize that telemetry: {{camel infra run}} can start backing services like Postgres or Artemis, but nothing analogous exists for observability.
> As a result, developers only find out whether traces, metrics, and logs actually work correctly (correct span correlation, meaningful dashboards, sane metric cardinality, etc.) once the application is deployed to a real OpenShift cluster, which typically provides the Red Hat build of OpenTelemetry, the Tempo Operator, Loki/OpenShift Logging, and the Cluster Observability Operator (Grafana). Telemetry problems surface late, in a shared cluster environment, rather than at dev/E2E time on a laptop or in CI.
> h3. Licensing note (why not grafana/otel-lgtm)
> The obvious candidate for this is Grafana's {{grafana/otel-lgtm}} all-in-one image. It was *ruled out*: Grafana, Loki, and Tempo were relicensed from Apache 2.0 to AGPLv3 in 2021. AGPLv3 is Category X on the ASF's third-party license policy, so an ASF project cannot depend on it even as opt-in test/dev tooling without a real legal-discuss@apache.org sign-off, and Red Hat's OSPO treats AGPL as off-limits even more strictly, including for test-only tooling in downstream builds. Only Prometheus and the OpenTelemetry Collector in that bundle are actually Apache 2.0.
> h3. Proposed Solution -- Apache-2.0-only stack (validated with a working PoC, see below)
> Use an equivalent stack built entirely from Apache-2.0-licensed components:
> * *Metrics*: Prometheus (unchanged -- already Apache 2.0, already the {{camel-observability-services}} default via {{/observe/metrics}}).
> * *Traces*: https://github.com/VictoriaMetrics/VictoriaTraces[VictoriaTraces] (Apache 2.0) -- ingests OTLP natively and exposes a Jaeger Query Service-compatible JSON API. Functionally a drop-in swap for Tempo.
> * *Logs*: https://github.com/VictoriaMetrics/VictoriaLogs[VictoriaLogs] (Apache 2.0) -- LogsQL query language, accepts OTLP logs directly via {{/insert/opentelemetry/v1/logs}}. Swap for Loki.
> * *Dashboard*: https://perses.dev[Perses] (Apache 2.0, CNCF sandbox project) -- dashboard-as-code tool backed by Amadeus/SAP/Red Hat, and notably *already powers OpenShift's own new traces UI*, which makes it a much safer downstream bet than Grafana. Its default plugin set already ships Prometheus, Jaeger, and VictoriaLogs datasource kinds with no extra plugin install.
> Adding this also revisits the fate of the existing {{camel-test-infra-jaeger}} module (added in CAMEL-23641): since VictoriaTraces exposes a Jaeger-compatible query API, {{camel-test-infra-jaeger}} does *not* need to be deprecated as originally floated -- it can keep serving its current role, or the new service can point at it directly, instead of being replaced outright. This should be decided during implementation, not assumed here.
> h3. Signal-level design rationale (matching real K8s/OCP behavior where it matters)
> * *Traces -- push*: matches OCP's own model (Red Hat build of OpenTelemetry pushes OTLP spans to a Collector). No divergence between local and cluster behavior.
> * *Metrics -- pull*: OCP's real default is Prometheus scraping via {{ServiceMonitor}}/{{PodMonitor}}, not app-side push. {{camel-observability-services}}' existing Prometheus-pull default already matches this -- the local service's job is to supply a scrape target pointed at the running app, mirroring what a {{ServiceMonitor}} does on-cluster, *not* to switch metrics to OTLP push.
> * *Logs -- local shortcut, not literal OCP parity*: real OpenShift logging does not have the app push logs at all -- Vector tails container stdout/stderr and ships to LokiStack; Loki itself still doesn't natively ingest OTLP. Locally, pushing OTLP logs straight from the OpenTelemetry Java agent's logback/log4j2 instrumentation into VictoriaLogs is validated and simple, but it is a different data path than production and should be documented as a pragmatic local-dev shortcut, not a faithful mirror of the cluster's log-collection mechanism.
> * *Trace/log correlation*: validated to work *automatically* via the OTel Java agent's logback-appender instrumentation ({{-Dotel.instrumentation.logback-appender.enabled=true}}) -- every ingested log line carries real {{trace_id}}/{{span_id}} with zero app-side MDC configuration. The {{camel-mdc}}-based approach documented in {{camel-opentelemetry2}}'s docs is not required for this path.
> h3. Empirical validation (PoC results)
> A working proof-of-concept was built and torn down; all 4 signals (metrics, traces, logs, and trace/log correlation) were confirmed working end-to-end with real data from a real {{camel-observability-services}}-enabled Camel route, and Perses accepted all 3 datasource definitions with no schema errors. Key findings to carry into implementation:
> * VictoriaTraces' OTLP gRPC receiver is *disabled by default* -- must pass {{-otlpGRPCListenAddr=:4317 -otlpGRPC.tls=false}}.
> * VictoriaTraces' Jaeger-compatible API is served at {{/select/jaeger/api/...}}, not the bare {{/api/...}} its docs imply.
> * *Port collision*: Perses defaults to port 8080, the same default port used by Camel/Spring Boot/Quarkus apps. Fix is to remap Perses, not the app -- confirmed via {{-web.listen-address :3000}} (also verified live), keeping the app on its normal default port. Suggest defaulting the test-infra service to port 3000 for Perses, matching the Grafana/otel-lgtm convention devs may already be used to.
> * Minimal working docker commands (see also PoC files, not retained beyond this session):
> {code}
> docker network create camel-obs
> docker run -d --name victoria-traces --network camel-obs -p 8428:8428 -p 4317:4317 \
>   victoriametrics/victoria-traces:latest -httpListenAddr=:8428 -otlpGRPCListenAddr=:4317 -otlpGRPC.tls=false
> docker run -d --name victoria-logs --network camel-obs -p 9428:9428 \
>   victoriametrics/victoria-logs:latest -httpListenAddr=:9428
> docker run -d --name prometheus --network camel-obs -p 9090:9090 \
>   -v "$(pwd)/prometheus.yml:/etc/prometheus/prometheus.yml:ro" --add-host=host.docker.internal:host-gateway \
>   prom/prometheus:latest
> docker run -d --name perses --network camel-obs -p 3000:3000 \
>   persesdev/perses:latest -web.listen-address :3000
> {code}
> * A minimal {{camel-observability-services}}-enabled JBang app, run via {{camel run}} (a bare {{RouteBuilder}} has no {{main}}, so plain {{jbang run}} does not work), with the OpenTelemetry Java agent supplying traces+logs export, confirmed all of: Prometheus scraping {{/observe/metrics}}, a real Camel-level {{timer}} span in VictoriaTraces, and OTLP log ingestion into VictoriaLogs correlated back to that trace.
> h3. Scope
> * New test-infra module/service(s) following existing test-infra patterns (e.g. {{*InfraService}}, {{*LocalContainerService}}, {{*ServiceFactory}}), composing the *4 separate containers* above -- there is no single prebuilt image bundling Prometheus + VictoriaTraces + VictoriaLogs + Perses the way {{grafana/otel-lgtm}} bundles its stack.
> * Wire the new service into the {{camel-jbang-core}} {{infra}} command so {{camel infra run observability}} works like other {{camel infra run <service>}} commands, including supplying the Prometheus scrape-target wiring against the running app.
> * Decide the relationship to the existing {{camel-test-infra-jaeger}} module (coexist vs. supersede) as part of implementation.
> * Out of scope for this ticket: changes to {{camel-observability-services}}' default export behavior itself (e.g. any future default-on MDC/log-correlation improvement is a separate, runtime-behavior-changing ticket), and cluster-side OpenShift/Kubernetes tooling (traits, {{camel kubernetes doctor}}, PodMonitor/PrometheusRule generation).
> h3. Open items
> * Perses' live query-through-datasource-proxy path was not confirmed during the PoC (datasource definitions were accepted, but querying data through Perses' proxy returned errors under the guessed path conventions) -- needs checking against Perses source/docs during implementation.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)