You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by Roberto Camelk <be...@gmail.com> on 2021/11/25 11:25:30 UTC
Apache Camel-k metrics in rancher embedded prometheus
I have a Kubernetes running Rancher 2.4.3. I have the cluster
monitoring enabled in rancher, so that exists a Prometheus instance
running, so as a Prometheus Operator.
Recently I deployed a Apache Camel-K operator, and now I want to
enable the prometheus integration for collect metrics about my camel
routes.
So, my Camel-K operator is running in namescape camel-k and the
rancher embedded prometheus stack in cattle-prometheus namespace.
I just have launched my route with the trait --trait
prometheus.enabled=true, but the camel metrics aren't listing at my
prometheus.
Anyone knows why or what I need to configure to my camel-k route
deploy it's metrics at the rancher embedded prometheus?
Re: Apache Camel-k metrics in rancher embedded prometheus
Posted by Antonin Stefanutti <an...@stefanutti.fr.INVALID>.
It generally takes some time for metrics to "show up", depending on the Prometheus instance configuration:
https://github.com/prometheus-operator/prometheus-operator/blob/787f54b055f797464d5832ace1c7f8318321c87a/Documentation/api.md
That may explain why restarting seems to fix the issue, as it forces a scrape cycle.
You may have to lower settings like scrapeInterval, evaluationInterval... These are probably better asking the Prometheus project, as it's already beyond my knowledge and the scope of the Camel K project.
> On 29 Nov 2021, at 16:04, Roberto Camelk <be...@gmail.com> wrote:
>
> Antonin, hi.
>
> I'm getting a problem in my prometheus operator about the podmonitor
> discovery....
>
> When I start a new integration, it's not discovered by the prometheus
> operator! I need to restart the pod operator to my integration to be
> discovered and scraped
>
> This is my Prometheus object in k8s:
>
>
> apiVersion: monitoring.coreos.com/v1
> kind: Prometheus
> metadata:
> annotations:
> kubectl.kubernetes.io/last-applied-configuration: |
> {"apiVersion":"monitoring.coreos.com/v1","kind":"Prometheus","metadata":{"annotations":{"project.cattle.io/namespaces":"[\"cattle-prometheus\",\"cattle-system\",\"kube-node-lease\",\"kube-public\",\"kube-sy$
> project.cattle.io/namespaces:
> '["cattle-prometheus","cattle-system","kube-node-lease","kube-public","kube-system","security-scan"]'
> generation: 23
> labels:
> app: prometheus
> chart: prometheus-0.0.1
> heritage: Tiller
> io.cattle.field/appId: cluster-monitoring
> release: cluster-monitoring
> name: cluster-monitoring
> namespace: cattle-prometheus
> resourceVersion: "68969602"
> spec:
> additionalAlertManagerConfigs:
> key: additional-alertmanager-configs.yaml
> name: prometheus-cluster-monitoring-additional-alertmanager-configs
> additionalScrapeConfigs:
> key: additional-scrape-configs.yaml
> name: prometheus-cluster-monitoring-additional-scrape-configs
> affinity:
> podAntiAffinity:
> preferredDuringSchedulingIgnoredDuringExecution:
> - podAffinityTerm:
> labelSelector:
> matchLabels:
> app: prometheus
> prometheus: cluster-monitoring
> topologyKey: kubernetes.io/hostname
> weight: 100
> arbitraryFSAccessThroughSMs: {}
> baseImage: rancher/prom-prometheus
> configMaps:
> - prometheus-cluster-monitoring-nginx
> containers:
> - command:
> - /bin/sh
> - -c
> - cp /nginx/run-sh.tmpl /var/cache/nginx/nginx-start.sh; chmod +x
> /var/cache/nginx/nginx-start.sh;
> /var/cache/nginx/nginx-start.sh
> env:
> - name: POD_IP
> valueFrom:
> fieldRef:
> fieldPath: status.podIP
> image: rancher/nginx:1.17.4-alpine
> name: prometheus-proxy
> ports:
> - containerPort: 8080
> name: nginx-http
> protocol: TCP
> resources:
> limits:
> cpu: 100m
> memory: 100Mi
> requests:
> cpu: 50m
> memory: 50Mi
> securityContext:
> runAsGroup: 101
> runAsUser: 101
> volumeMounts:
> - mountPath: /nginx
> name: configmap-prometheus-cluster-monitoring-nginx
> - mountPath: /var/cache/nginx
> name: nginx-home
> - args:
> - --proxy-url=http://127.0.0.1:9090
> - --listen-address=$(POD_IP):9090
> - --filter-reader-labels=prometheus
> - --filter-reader-labels=prometheus_replica
> command:
> - prometheus-auth
> env:
> - name: POD_IP
> valueFrom:
> fieldRef:
> fieldPath: status.podIP
> image: rancher/prometheus-auth:v0.2.0
> livenessProbe:
> failureThreshold: 6
> httpGet:
> path: /-/healthy
> port: web
> scheme: HTTP
> initialDelaySeconds: 300
> periodSeconds: 10
> successThreshold: 1
> timeoutSeconds: 10
> name: prometheus-agent
> ports:
> - containerPort: 9090
> name: web
> protocol: TCP
> readinessProbe:
> failureThreshold: 10
> httpGet:
> path: /-/ready
> port: web
> scheme: HTTP
> initialDelaySeconds: 60
> periodSeconds: 10
> successThreshold: 1
> timeoutSeconds: 10
> resources:
> limits:
> cpu: 500m
> memory: 200Mi
> requests:
> cpu: 100m
> memory: 100Mi
> evaluationInterval: 60s
> externalLabels:
> prometheus_from: arquitetura
> listenLocal: true
> logFormat: logfmt
> logLevel: info
> nodeSelector:
> kubernetes.io/os: linux
> podMetadata:
> labels:
> labels:
> app: prometheus
> chart: prometheus-0.0.1
> release: cluster-monitoring
> podMonitorNamespaceSelector: {}
> podMonitorSelector:
> matchLabels:
> app: camel-k
> replicas: 1
> resources:
> limits:
> cpu: "1"
> memory: 1000Mi
> requests:
> cpu: 750m
> memory: 750Mi
> retention: 12h
> ruleNamespaceSelector:
> matchExpressions:
> - key: field.cattle.io/projectId
> operator: In
> values:
> - p-gzj4v
> - key: field.cattle.io/projectId
> operator: In
> values:
> - p-gzj4v
> ruleSelector:
> matchExpressions:
> - key: source
> operator: In
> values:
> - rancher-alert
> - rancher-monitoring
> rules:
> alert: {}
> scrapeInterval: 60s
> secrets:
> - exporter-etcd-cert
> securityContext:
> fsGroup: 2000
> runAsNonRoot: true
> runAsUser: 1000
> serviceAccountName: cluster-monitoring
> serviceMonitorNamespaceSelector:
> matchExpressions:
> - key: field.cattle.io/projectId
> operator: In
> values:
> - p-gzj4v
> - key: field.cattle.io/projectId
> operator: In
> values:
> - p-gzj4v
> serviceMonitorSelector: {}
> tolerations:
> - effect: NoSchedule
> key: cattle.io/os
> operator: Equal
> value: linux
> version: v2.17.2
> volumes:
> - emptyDir: {}
> name: nginx-home
>
> On Fri, Nov 26, 2021 at 11:41 AM Roberto Camelk
> <be...@gmail.com> wrote:
>>
>> Thanks again Antonin.
>>
>> You save my black-friday!
>>
>> On Fri, Nov 26, 2021 at 11:02 AM Antonin Stefanutti
>> <an...@stefanutti.fr.invalid> wrote:
>>>
>>> Great!
>>>
>>> For the metrics, there should be the following registered by the MicroProfile Metrics Camel extension:
>>>
>>> https://camel.apache.org/camel-quarkus/2.4.x/reference/extensions/microprofile-metrics.html#_usage
>>>
>>> However, the final name is determined by the MicroProfile Metrics specification, so for example the following metric:
>>>
>>> camel.context.exchanges.completed.total
>>>
>>> Have its name translated to:
>>>
>>> application_camel_context_exchanges_completed_total
>>>
>>> To check also if the metrics endpoint do have the metrics registered, you can run:
>>>
>>> $ kubectl exec deployment/<integration_name> -- curl -s http://localhost:8080/q/metrics | grep application_camel_route_exchanges_completed_total
>>>
>>> # HELP application_camel_route_exchanges_completed_total The total number of completed exchanges for a route or Camel Context
>>> # TYPE application_camel_route_exchanges_completed_total counter
>>> application_camel_route_exchanges_completed_total{camelContext="camel-1",routeId="route1"} 33.0
>>>
>>>> On 26 Nov 2021, at 14:34, Roberto Camelk <be...@gmail.com> wrote:
>>>>
>>>> Thanks a lot again Antonin. I owe you a beer or coffee!!!
>>>>
>>>> This worked 100%.
>>>>
>>>> To end, another question.... after this, prometheus start scraping the
>>>> metrics from my running route pod, but...
>>>>
>>>> Some metrics IDs are not there, some metrics starting with
>>>> "org_apache_camel_" as I have see in this grafana dashboard sample:
>>>> https://github.com/weimeilin79/camel-k-example-prometheus/blob/ca347f0b8b702b84b7129dfcfcb3b84eff4e2f73/grafana/SampleCamelDashboard.json#L145
>>>>
>>>> The other metrics (camel_k_* -
>>>> https://camel.apache.org/camel-k/next/observability/monitoring/operator.html#metrics)
>>>> are there as so the jvm metrics
>>>> (https://github.com/eclipse/microprofile-metrics/blob/master/spec/src/main/asciidoc/required-metrics.adoc#required-metrics).
>>>>
>>>> Is this correct? Why am I not getting that "org_apache_camel_" metrics?
>>>>
>>>> On Fri, Nov 26, 2021 at 9:13 AM Antonin Stefanutti
>>>> <an...@stefanutti.fr.invalid> wrote:
>>>>>
>>>>> It's likely that you need to configure the Prometheus instance, by editing the Prometheus resource, e.g.:
>>>>>
>>>>> apiVersion: monitoring.coreos.com/v1
>>>>> kind: Prometheus
>>>>> metadata:
>>>>> name: prometheus
>>>>> spec:
>>>>> serviceAccountName: prometheus
>>>>> podMonitorSelector:
>>>>> matchLabels:
>>>>> app: camel-k
>>>>> podMonitorNamespaceSelector: {}
>>>>>
>>>>> It's important to have podMonitorNamespaceSelector: {} to discover all the namespaces, otherwise it's only the resource's namespace.
>>>>>
>>>>> Also you can use the Prometheus trait to set the labels on the Integration PodMonitor accordingly, e.g.:
>>>>>
>>>>> $ kamel run -t prometheus.enabled=true -t prometheus.pod-monitor-labels="app=camel-k"
>>>>>
>>>>> There are some examples in the Prometheus operator documentation:
>>>>>
>>>>> https://github.com/prometheus-operator/prometheus-operator/tree/v0.52.1/example/user-guides/getting-started
>>>>>
>>>>> And the troubleshooting guide at:
>>>>>
>>>>> https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/troubleshooting.md
>>>>>
>>>>>
>>>>>> On 26 Nov 2021, at 12:32, Roberto Camelk <be...@gmail.com> wrote:
>>>>>>
>>>>>> Antonin. Thanks !
>>>>>>
>>>>>> But I continue stuck, please let me share some extra info about my
>>>>>> prometheus operator log:
>>>>>>
>>>>>> level=debug ts=2021-11-26T11:12:05.837624346Z caller=operator.go:1840
>>>>>> component=prometheusoperator msg="filtering namespaces to select
>>>>>> PodMonitors from" namespaces=cattle-prometheus
>>>>>> namespace=cattle-prometheus prometheus=cluster-monitoring
>>>>>> 26/11/2021 08:12:05 level=debug ts=2021-11-26T11:12:05.837687534Z
>>>>>> caller=operator.go:1853 component=prometheusoperator msg="selected
>>>>>> PodMonitors" podmonitors= namespace=cattle-prometheus
>>>>>> prometheus=cluster-monitoring
>>>>>> 26/11/2021 08:12:05 level=debug ts=2021-11-26T11:12:05.942216811Z
>>>>>> caller=operator.go:1677 component=prometheusoperator msg="updating
>>>>>> Prometheus configuration secret skipped, no configuration change"
>>>>>> 26/11/2021 08:12:05 level=debug ts=2021-11-26T11:12:05.950980834Z
>>>>>> caller=operator.go:1776 component=prometheusoperator msg="filtering
>>>>>> namespaces to select ServiceMonitors from"
>>>>>> namespaces=cattle-prometheus,cattle-system,kube-node-lease,kube-public,security-scan,kube-system
>>>>>> namespace=cattle-prometheus prometheus=cluster-monitoring
>>>>>> 26/11/2021 08:12:05 level=debug ts=2021-11-26T11:12:05.951162973Z
>>>>>> caller=operator.go:1810 component=prometheusoperator msg="selected
>>>>>> ServiceMonitors"
>>>>>> servicemonitors=cattle-prometheus/grafana-cluster-monitoring,cattle-prometheus/exporter-kube-etcd-cluster-monitoring,cattle-prometheus/exporter-node-cluster-monitoring,cattle-prometheus/exporter-kube-controller-manager-cluster-monitoring,cattle-prometheus/exporter-kube-state-cluster-monitoring,cattle-prometheus/prometheus-cluster-monitoring,cattle-prometheus/prometheus-operator-monitoring-operator,cattle-prometheus/exporter-fluentd-cluster-monitoring,cattle-prometheus/exporter-kubelets-cluster-monitoring,cattle-prometheus/exporter-kube-scheduler-cluster-monitoring,cattle-prometheus/exporter-kubernetes-cluster-monitoring
>>>>>> namespace=cattle-prometheus prometheus=cluster-monitoring
>>>>>> 26/11/2021 08:12:05 level=debug ts=2021-11-26T11:12:05.977550133Z
>>>>>> caller=operator.go:1741 component=prometheusoperator msg="updated
>>>>>> tlsAssetsSecret" secretname=prometheus-cluster-monitoring-tls-assets
>>>>>> 26/11/2021 08:12:06 level=debug ts=2021-11-26T11:12:06.022196407Z
>>>>>> caller=operator.go:1169 component=prometheusoperator msg="new
>>>>>> statefulset generation inputs match current, skipping any actions"
>>>>>> 26/11/2021 08:12:26 level=debug ts=2021-11-26T11:12:26.321817491Z
>>>>>> caller=operator.go:734 component=prometheusoperator msg="PodMonitor
>>>>>> added"
>>>>>> 26/11/2021 08:12:27 level=debug ts=2021-11-26T11:12:27.755854021Z
>>>>>> caller=operator.go:748 component=prometheusoperator msg="PodMonitor
>>>>>> updated"
>>>>>> 26/11/2021 08:12:46 level=debug ts=2021-11-26T11:12:46.453112794Z
>>>>>> caller=operator.go:748 component=prometheusoperator msg="PodMonitor
>>>>>> updated"
>>>>>> 26/11/2021 08:17:35 level=debug ts=2021-11-26T11:17:35.194031009Z
>>>>>> caller=operator.go:759 component=prometheusoperator msg="PodMonitor
>>>>>> delete"
>>>>>>
>>>>>> This last 4 lines is about my camel-k route, that I ran and stopped.
>>>>>>
>>>>>> So, there are some problems, I think, about the logs above about the
>>>>>> podmonitor selectors telling: "selected PodMonitors" podmonitors=
>>>>>> namespace=cattle-prometheus prometheus=cluster-monitoring
>>>>>>
>>>>>> My camel-k route is running at namespace "platform" and has no label
>>>>>> like "prometheus=cluster-monitoring".
>>>>>>
>>>>>> Do you know how can I fix this? Adding additional scrape configs to
>>>>>> prometheus can solve this? Can you provide a snipet code?
>>>>>>
>>>>>> On Thu, Nov 25, 2021 at 9:56 AM Antonin Stefanutti
>>>>>> <an...@stefanutti.fr.invalid> wrote:
>>>>>>>
>>>>>>> When run an Integration with `kamel run -t prometheus.enabled=true`, a PodMonitor resource is created for the Prometheus operator to reconcile and configure Prometheus to scrape the Integration metrics endpoint.
>>>>>>>
>>>>>>> The PodMonitor metadata must match that of the Prometheus operator, like the namespace, the labels, ...
>>>>>>>
>>>>>>> Some documentation is available at:
>>>>>>>
>>>>>>> https://camel.apache.org/camel-k/1.7.x/observability/monitoring/integration.html#_discovery
>>>>>>>
>>>>>>> That contains some links to the Prometheus operator documentation for troubleshooting why the metrics endpoint is not discovered.
>>>>>>>
>>>>>>>> On 25 Nov 2021, at 12:25, Roberto Camelk <be...@gmail.com> wrote:
>>>>>>>>
>>>>>>>> I have a Kubernetes running Rancher 2.4.3. I have the cluster
>>>>>>>> monitoring enabled in rancher, so that exists a Prometheus instance
>>>>>>>> running, so as a Prometheus Operator.
>>>>>>>>
>>>>>>>> Recently I deployed a Apache Camel-K operator, and now I want to
>>>>>>>> enable the prometheus integration for collect metrics about my camel
>>>>>>>> routes.
>>>>>>>>
>>>>>>>> So, my Camel-K operator is running in namescape camel-k and the
>>>>>>>> rancher embedded prometheus stack in cattle-prometheus namespace.
>>>>>>>>
>>>>>>>> I just have launched my route with the trait --trait
>>>>>>>> prometheus.enabled=true, but the camel metrics aren't listing at my
>>>>>>>> prometheus.
>>>>>>>>
>>>>>>>> Anyone knows why or what I need to configure to my camel-k route
>>>>>>>> deploy it's metrics at the rancher embedded prometheus?
>>>>>>>
>>>>>
>>>
Re: Apache Camel-k metrics in rancher embedded prometheus
Posted by Roberto Camelk <be...@gmail.com>.
Antonin, hi.
I'm getting a problem in my prometheus operator about the podmonitor
discovery....
When I start a new integration, it's not discovered by the prometheus
operator! I need to restart the pod operator to my integration to be
discovered and scraped
This is my Prometheus object in k8s:
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"monitoring.coreos.com/v1","kind":"Prometheus","metadata":{"annotations":{"project.cattle.io/namespaces":"[\"cattle-prometheus\",\"cattle-system\",\"kube-node-lease\",\"kube-public\",\"kube-sy$
project.cattle.io/namespaces:
'["cattle-prometheus","cattle-system","kube-node-lease","kube-public","kube-system","security-scan"]'
generation: 23
labels:
app: prometheus
chart: prometheus-0.0.1
heritage: Tiller
io.cattle.field/appId: cluster-monitoring
release: cluster-monitoring
name: cluster-monitoring
namespace: cattle-prometheus
resourceVersion: "68969602"
spec:
additionalAlertManagerConfigs:
key: additional-alertmanager-configs.yaml
name: prometheus-cluster-monitoring-additional-alertmanager-configs
additionalScrapeConfigs:
key: additional-scrape-configs.yaml
name: prometheus-cluster-monitoring-additional-scrape-configs
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
labelSelector:
matchLabels:
app: prometheus
prometheus: cluster-monitoring
topologyKey: kubernetes.io/hostname
weight: 100
arbitraryFSAccessThroughSMs: {}
baseImage: rancher/prom-prometheus
configMaps:
- prometheus-cluster-monitoring-nginx
containers:
- command:
- /bin/sh
- -c
- cp /nginx/run-sh.tmpl /var/cache/nginx/nginx-start.sh; chmod +x
/var/cache/nginx/nginx-start.sh;
/var/cache/nginx/nginx-start.sh
env:
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
image: rancher/nginx:1.17.4-alpine
name: prometheus-proxy
ports:
- containerPort: 8080
name: nginx-http
protocol: TCP
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 50m
memory: 50Mi
securityContext:
runAsGroup: 101
runAsUser: 101
volumeMounts:
- mountPath: /nginx
name: configmap-prometheus-cluster-monitoring-nginx
- mountPath: /var/cache/nginx
name: nginx-home
- args:
- --proxy-url=http://127.0.0.1:9090
- --listen-address=$(POD_IP):9090
- --filter-reader-labels=prometheus
- --filter-reader-labels=prometheus_replica
command:
- prometheus-auth
env:
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
image: rancher/prometheus-auth:v0.2.0
livenessProbe:
failureThreshold: 6
httpGet:
path: /-/healthy
port: web
scheme: HTTP
initialDelaySeconds: 300
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 10
name: prometheus-agent
ports:
- containerPort: 9090
name: web
protocol: TCP
readinessProbe:
failureThreshold: 10
httpGet:
path: /-/ready
port: web
scheme: HTTP
initialDelaySeconds: 60
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 10
resources:
limits:
cpu: 500m
memory: 200Mi
requests:
cpu: 100m
memory: 100Mi
evaluationInterval: 60s
externalLabels:
prometheus_from: arquitetura
listenLocal: true
logFormat: logfmt
logLevel: info
nodeSelector:
kubernetes.io/os: linux
podMetadata:
labels:
labels:
app: prometheus
chart: prometheus-0.0.1
release: cluster-monitoring
podMonitorNamespaceSelector: {}
podMonitorSelector:
matchLabels:
app: camel-k
replicas: 1
resources:
limits:
cpu: "1"
memory: 1000Mi
requests:
cpu: 750m
memory: 750Mi
retention: 12h
ruleNamespaceSelector:
matchExpressions:
- key: field.cattle.io/projectId
operator: In
values:
- p-gzj4v
- key: field.cattle.io/projectId
operator: In
values:
- p-gzj4v
ruleSelector:
matchExpressions:
- key: source
operator: In
values:
- rancher-alert
- rancher-monitoring
rules:
alert: {}
scrapeInterval: 60s
secrets:
- exporter-etcd-cert
securityContext:
fsGroup: 2000
runAsNonRoot: true
runAsUser: 1000
serviceAccountName: cluster-monitoring
serviceMonitorNamespaceSelector:
matchExpressions:
- key: field.cattle.io/projectId
operator: In
values:
- p-gzj4v
- key: field.cattle.io/projectId
operator: In
values:
- p-gzj4v
serviceMonitorSelector: {}
tolerations:
- effect: NoSchedule
key: cattle.io/os
operator: Equal
value: linux
version: v2.17.2
volumes:
- emptyDir: {}
name: nginx-home
On Fri, Nov 26, 2021 at 11:41 AM Roberto Camelk
<be...@gmail.com> wrote:
>
> Thanks again Antonin.
>
> You save my black-friday!
>
> On Fri, Nov 26, 2021 at 11:02 AM Antonin Stefanutti
> <an...@stefanutti.fr.invalid> wrote:
> >
> > Great!
> >
> > For the metrics, there should be the following registered by the MicroProfile Metrics Camel extension:
> >
> > https://camel.apache.org/camel-quarkus/2.4.x/reference/extensions/microprofile-metrics.html#_usage
> >
> > However, the final name is determined by the MicroProfile Metrics specification, so for example the following metric:
> >
> > camel.context.exchanges.completed.total
> >
> > Have its name translated to:
> >
> > application_camel_context_exchanges_completed_total
> >
> > To check also if the metrics endpoint do have the metrics registered, you can run:
> >
> > $ kubectl exec deployment/<integration_name> -- curl -s http://localhost:8080/q/metrics | grep application_camel_route_exchanges_completed_total
> >
> > # HELP application_camel_route_exchanges_completed_total The total number of completed exchanges for a route or Camel Context
> > # TYPE application_camel_route_exchanges_completed_total counter
> > application_camel_route_exchanges_completed_total{camelContext="camel-1",routeId="route1"} 33.0
> >
> > > On 26 Nov 2021, at 14:34, Roberto Camelk <be...@gmail.com> wrote:
> > >
> > > Thanks a lot again Antonin. I owe you a beer or coffee!!!
> > >
> > > This worked 100%.
> > >
> > > To end, another question.... after this, prometheus start scraping the
> > > metrics from my running route pod, but...
> > >
> > > Some metrics IDs are not there, some metrics starting with
> > > "org_apache_camel_" as I have see in this grafana dashboard sample:
> > > https://github.com/weimeilin79/camel-k-example-prometheus/blob/ca347f0b8b702b84b7129dfcfcb3b84eff4e2f73/grafana/SampleCamelDashboard.json#L145
> > >
> > > The other metrics (camel_k_* -
> > > https://camel.apache.org/camel-k/next/observability/monitoring/operator.html#metrics)
> > > are there as so the jvm metrics
> > > (https://github.com/eclipse/microprofile-metrics/blob/master/spec/src/main/asciidoc/required-metrics.adoc#required-metrics).
> > >
> > > Is this correct? Why am I not getting that "org_apache_camel_" metrics?
> > >
> > > On Fri, Nov 26, 2021 at 9:13 AM Antonin Stefanutti
> > > <an...@stefanutti.fr.invalid> wrote:
> > >>
> > >> It's likely that you need to configure the Prometheus instance, by editing the Prometheus resource, e.g.:
> > >>
> > >> apiVersion: monitoring.coreos.com/v1
> > >> kind: Prometheus
> > >> metadata:
> > >> name: prometheus
> > >> spec:
> > >> serviceAccountName: prometheus
> > >> podMonitorSelector:
> > >> matchLabels:
> > >> app: camel-k
> > >> podMonitorNamespaceSelector: {}
> > >>
> > >> It's important to have podMonitorNamespaceSelector: {} to discover all the namespaces, otherwise it's only the resource's namespace.
> > >>
> > >> Also you can use the Prometheus trait to set the labels on the Integration PodMonitor accordingly, e.g.:
> > >>
> > >> $ kamel run -t prometheus.enabled=true -t prometheus.pod-monitor-labels="app=camel-k"
> > >>
> > >> There are some examples in the Prometheus operator documentation:
> > >>
> > >> https://github.com/prometheus-operator/prometheus-operator/tree/v0.52.1/example/user-guides/getting-started
> > >>
> > >> And the troubleshooting guide at:
> > >>
> > >> https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/troubleshooting.md
> > >>
> > >>
> > >>> On 26 Nov 2021, at 12:32, Roberto Camelk <be...@gmail.com> wrote:
> > >>>
> > >>> Antonin. Thanks !
> > >>>
> > >>> But I continue stuck, please let me share some extra info about my
> > >>> prometheus operator log:
> > >>>
> > >>> level=debug ts=2021-11-26T11:12:05.837624346Z caller=operator.go:1840
> > >>> component=prometheusoperator msg="filtering namespaces to select
> > >>> PodMonitors from" namespaces=cattle-prometheus
> > >>> namespace=cattle-prometheus prometheus=cluster-monitoring
> > >>> 26/11/2021 08:12:05 level=debug ts=2021-11-26T11:12:05.837687534Z
> > >>> caller=operator.go:1853 component=prometheusoperator msg="selected
> > >>> PodMonitors" podmonitors= namespace=cattle-prometheus
> > >>> prometheus=cluster-monitoring
> > >>> 26/11/2021 08:12:05 level=debug ts=2021-11-26T11:12:05.942216811Z
> > >>> caller=operator.go:1677 component=prometheusoperator msg="updating
> > >>> Prometheus configuration secret skipped, no configuration change"
> > >>> 26/11/2021 08:12:05 level=debug ts=2021-11-26T11:12:05.950980834Z
> > >>> caller=operator.go:1776 component=prometheusoperator msg="filtering
> > >>> namespaces to select ServiceMonitors from"
> > >>> namespaces=cattle-prometheus,cattle-system,kube-node-lease,kube-public,security-scan,kube-system
> > >>> namespace=cattle-prometheus prometheus=cluster-monitoring
> > >>> 26/11/2021 08:12:05 level=debug ts=2021-11-26T11:12:05.951162973Z
> > >>> caller=operator.go:1810 component=prometheusoperator msg="selected
> > >>> ServiceMonitors"
> > >>> servicemonitors=cattle-prometheus/grafana-cluster-monitoring,cattle-prometheus/exporter-kube-etcd-cluster-monitoring,cattle-prometheus/exporter-node-cluster-monitoring,cattle-prometheus/exporter-kube-controller-manager-cluster-monitoring,cattle-prometheus/exporter-kube-state-cluster-monitoring,cattle-prometheus/prometheus-cluster-monitoring,cattle-prometheus/prometheus-operator-monitoring-operator,cattle-prometheus/exporter-fluentd-cluster-monitoring,cattle-prometheus/exporter-kubelets-cluster-monitoring,cattle-prometheus/exporter-kube-scheduler-cluster-monitoring,cattle-prometheus/exporter-kubernetes-cluster-monitoring
> > >>> namespace=cattle-prometheus prometheus=cluster-monitoring
> > >>> 26/11/2021 08:12:05 level=debug ts=2021-11-26T11:12:05.977550133Z
> > >>> caller=operator.go:1741 component=prometheusoperator msg="updated
> > >>> tlsAssetsSecret" secretname=prometheus-cluster-monitoring-tls-assets
> > >>> 26/11/2021 08:12:06 level=debug ts=2021-11-26T11:12:06.022196407Z
> > >>> caller=operator.go:1169 component=prometheusoperator msg="new
> > >>> statefulset generation inputs match current, skipping any actions"
> > >>> 26/11/2021 08:12:26 level=debug ts=2021-11-26T11:12:26.321817491Z
> > >>> caller=operator.go:734 component=prometheusoperator msg="PodMonitor
> > >>> added"
> > >>> 26/11/2021 08:12:27 level=debug ts=2021-11-26T11:12:27.755854021Z
> > >>> caller=operator.go:748 component=prometheusoperator msg="PodMonitor
> > >>> updated"
> > >>> 26/11/2021 08:12:46 level=debug ts=2021-11-26T11:12:46.453112794Z
> > >>> caller=operator.go:748 component=prometheusoperator msg="PodMonitor
> > >>> updated"
> > >>> 26/11/2021 08:17:35 level=debug ts=2021-11-26T11:17:35.194031009Z
> > >>> caller=operator.go:759 component=prometheusoperator msg="PodMonitor
> > >>> delete"
> > >>>
> > >>> This last 4 lines is about my camel-k route, that I ran and stopped.
> > >>>
> > >>> So, there are some problems, I think, about the logs above about the
> > >>> podmonitor selectors telling: "selected PodMonitors" podmonitors=
> > >>> namespace=cattle-prometheus prometheus=cluster-monitoring
> > >>>
> > >>> My camel-k route is running at namespace "platform" and has no label
> > >>> like "prometheus=cluster-monitoring".
> > >>>
> > >>> Do you know how can I fix this? Adding additional scrape configs to
> > >>> prometheus can solve this? Can you provide a snipet code?
> > >>>
> > >>> On Thu, Nov 25, 2021 at 9:56 AM Antonin Stefanutti
> > >>> <an...@stefanutti.fr.invalid> wrote:
> > >>>>
> > >>>> When run an Integration with `kamel run -t prometheus.enabled=true`, a PodMonitor resource is created for the Prometheus operator to reconcile and configure Prometheus to scrape the Integration metrics endpoint.
> > >>>>
> > >>>> The PodMonitor metadata must match that of the Prometheus operator, like the namespace, the labels, ...
> > >>>>
> > >>>> Some documentation is available at:
> > >>>>
> > >>>> https://camel.apache.org/camel-k/1.7.x/observability/monitoring/integration.html#_discovery
> > >>>>
> > >>>> That contains some links to the Prometheus operator documentation for troubleshooting why the metrics endpoint is not discovered.
> > >>>>
> > >>>>> On 25 Nov 2021, at 12:25, Roberto Camelk <be...@gmail.com> wrote:
> > >>>>>
> > >>>>> I have a Kubernetes running Rancher 2.4.3. I have the cluster
> > >>>>> monitoring enabled in rancher, so that exists a Prometheus instance
> > >>>>> running, so as a Prometheus Operator.
> > >>>>>
> > >>>>> Recently I deployed a Apache Camel-K operator, and now I want to
> > >>>>> enable the prometheus integration for collect metrics about my camel
> > >>>>> routes.
> > >>>>>
> > >>>>> So, my Camel-K operator is running in namescape camel-k and the
> > >>>>> rancher embedded prometheus stack in cattle-prometheus namespace.
> > >>>>>
> > >>>>> I just have launched my route with the trait --trait
> > >>>>> prometheus.enabled=true, but the camel metrics aren't listing at my
> > >>>>> prometheus.
> > >>>>>
> > >>>>> Anyone knows why or what I need to configure to my camel-k route
> > >>>>> deploy it's metrics at the rancher embedded prometheus?
> > >>>>
> > >>
> >
Re: Apache Camel-k metrics in rancher embedded prometheus
Posted by Roberto Camelk <be...@gmail.com>.
Thanks again Antonin.
You save my black-friday!
On Fri, Nov 26, 2021 at 11:02 AM Antonin Stefanutti
<an...@stefanutti.fr.invalid> wrote:
>
> Great!
>
> For the metrics, there should be the following registered by the MicroProfile Metrics Camel extension:
>
> https://camel.apache.org/camel-quarkus/2.4.x/reference/extensions/microprofile-metrics.html#_usage
>
> However, the final name is determined by the MicroProfile Metrics specification, so for example the following metric:
>
> camel.context.exchanges.completed.total
>
> Have its name translated to:
>
> application_camel_context_exchanges_completed_total
>
> To check also if the metrics endpoint do have the metrics registered, you can run:
>
> $ kubectl exec deployment/<integration_name> -- curl -s http://localhost:8080/q/metrics | grep application_camel_route_exchanges_completed_total
>
> # HELP application_camel_route_exchanges_completed_total The total number of completed exchanges for a route or Camel Context
> # TYPE application_camel_route_exchanges_completed_total counter
> application_camel_route_exchanges_completed_total{camelContext="camel-1",routeId="route1"} 33.0
>
> > On 26 Nov 2021, at 14:34, Roberto Camelk <be...@gmail.com> wrote:
> >
> > Thanks a lot again Antonin. I owe you a beer or coffee!!!
> >
> > This worked 100%.
> >
> > To end, another question.... after this, prometheus start scraping the
> > metrics from my running route pod, but...
> >
> > Some metrics IDs are not there, some metrics starting with
> > "org_apache_camel_" as I have see in this grafana dashboard sample:
> > https://github.com/weimeilin79/camel-k-example-prometheus/blob/ca347f0b8b702b84b7129dfcfcb3b84eff4e2f73/grafana/SampleCamelDashboard.json#L145
> >
> > The other metrics (camel_k_* -
> > https://camel.apache.org/camel-k/next/observability/monitoring/operator.html#metrics)
> > are there as so the jvm metrics
> > (https://github.com/eclipse/microprofile-metrics/blob/master/spec/src/main/asciidoc/required-metrics.adoc#required-metrics).
> >
> > Is this correct? Why am I not getting that "org_apache_camel_" metrics?
> >
> > On Fri, Nov 26, 2021 at 9:13 AM Antonin Stefanutti
> > <an...@stefanutti.fr.invalid> wrote:
> >>
> >> It's likely that you need to configure the Prometheus instance, by editing the Prometheus resource, e.g.:
> >>
> >> apiVersion: monitoring.coreos.com/v1
> >> kind: Prometheus
> >> metadata:
> >> name: prometheus
> >> spec:
> >> serviceAccountName: prometheus
> >> podMonitorSelector:
> >> matchLabels:
> >> app: camel-k
> >> podMonitorNamespaceSelector: {}
> >>
> >> It's important to have podMonitorNamespaceSelector: {} to discover all the namespaces, otherwise it's only the resource's namespace.
> >>
> >> Also you can use the Prometheus trait to set the labels on the Integration PodMonitor accordingly, e.g.:
> >>
> >> $ kamel run -t prometheus.enabled=true -t prometheus.pod-monitor-labels="app=camel-k"
> >>
> >> There are some examples in the Prometheus operator documentation:
> >>
> >> https://github.com/prometheus-operator/prometheus-operator/tree/v0.52.1/example/user-guides/getting-started
> >>
> >> And the troubleshooting guide at:
> >>
> >> https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/troubleshooting.md
> >>
> >>
> >>> On 26 Nov 2021, at 12:32, Roberto Camelk <be...@gmail.com> wrote:
> >>>
> >>> Antonin. Thanks !
> >>>
> >>> But I continue stuck, please let me share some extra info about my
> >>> prometheus operator log:
> >>>
> >>> level=debug ts=2021-11-26T11:12:05.837624346Z caller=operator.go:1840
> >>> component=prometheusoperator msg="filtering namespaces to select
> >>> PodMonitors from" namespaces=cattle-prometheus
> >>> namespace=cattle-prometheus prometheus=cluster-monitoring
> >>> 26/11/2021 08:12:05 level=debug ts=2021-11-26T11:12:05.837687534Z
> >>> caller=operator.go:1853 component=prometheusoperator msg="selected
> >>> PodMonitors" podmonitors= namespace=cattle-prometheus
> >>> prometheus=cluster-monitoring
> >>> 26/11/2021 08:12:05 level=debug ts=2021-11-26T11:12:05.942216811Z
> >>> caller=operator.go:1677 component=prometheusoperator msg="updating
> >>> Prometheus configuration secret skipped, no configuration change"
> >>> 26/11/2021 08:12:05 level=debug ts=2021-11-26T11:12:05.950980834Z
> >>> caller=operator.go:1776 component=prometheusoperator msg="filtering
> >>> namespaces to select ServiceMonitors from"
> >>> namespaces=cattle-prometheus,cattle-system,kube-node-lease,kube-public,security-scan,kube-system
> >>> namespace=cattle-prometheus prometheus=cluster-monitoring
> >>> 26/11/2021 08:12:05 level=debug ts=2021-11-26T11:12:05.951162973Z
> >>> caller=operator.go:1810 component=prometheusoperator msg="selected
> >>> ServiceMonitors"
> >>> servicemonitors=cattle-prometheus/grafana-cluster-monitoring,cattle-prometheus/exporter-kube-etcd-cluster-monitoring,cattle-prometheus/exporter-node-cluster-monitoring,cattle-prometheus/exporter-kube-controller-manager-cluster-monitoring,cattle-prometheus/exporter-kube-state-cluster-monitoring,cattle-prometheus/prometheus-cluster-monitoring,cattle-prometheus/prometheus-operator-monitoring-operator,cattle-prometheus/exporter-fluentd-cluster-monitoring,cattle-prometheus/exporter-kubelets-cluster-monitoring,cattle-prometheus/exporter-kube-scheduler-cluster-monitoring,cattle-prometheus/exporter-kubernetes-cluster-monitoring
> >>> namespace=cattle-prometheus prometheus=cluster-monitoring
> >>> 26/11/2021 08:12:05 level=debug ts=2021-11-26T11:12:05.977550133Z
> >>> caller=operator.go:1741 component=prometheusoperator msg="updated
> >>> tlsAssetsSecret" secretname=prometheus-cluster-monitoring-tls-assets
> >>> 26/11/2021 08:12:06 level=debug ts=2021-11-26T11:12:06.022196407Z
> >>> caller=operator.go:1169 component=prometheusoperator msg="new
> >>> statefulset generation inputs match current, skipping any actions"
> >>> 26/11/2021 08:12:26 level=debug ts=2021-11-26T11:12:26.321817491Z
> >>> caller=operator.go:734 component=prometheusoperator msg="PodMonitor
> >>> added"
> >>> 26/11/2021 08:12:27 level=debug ts=2021-11-26T11:12:27.755854021Z
> >>> caller=operator.go:748 component=prometheusoperator msg="PodMonitor
> >>> updated"
> >>> 26/11/2021 08:12:46 level=debug ts=2021-11-26T11:12:46.453112794Z
> >>> caller=operator.go:748 component=prometheusoperator msg="PodMonitor
> >>> updated"
> >>> 26/11/2021 08:17:35 level=debug ts=2021-11-26T11:17:35.194031009Z
> >>> caller=operator.go:759 component=prometheusoperator msg="PodMonitor
> >>> delete"
> >>>
> >>> This last 4 lines is about my camel-k route, that I ran and stopped.
> >>>
> >>> So, there are some problems, I think, about the logs above about the
> >>> podmonitor selectors telling: "selected PodMonitors" podmonitors=
> >>> namespace=cattle-prometheus prometheus=cluster-monitoring
> >>>
> >>> My camel-k route is running at namespace "platform" and has no label
> >>> like "prometheus=cluster-monitoring".
> >>>
> >>> Do you know how can I fix this? Adding additional scrape configs to
> >>> prometheus can solve this? Can you provide a snipet code?
> >>>
> >>> On Thu, Nov 25, 2021 at 9:56 AM Antonin Stefanutti
> >>> <an...@stefanutti.fr.invalid> wrote:
> >>>>
> >>>> When run an Integration with `kamel run -t prometheus.enabled=true`, a PodMonitor resource is created for the Prometheus operator to reconcile and configure Prometheus to scrape the Integration metrics endpoint.
> >>>>
> >>>> The PodMonitor metadata must match that of the Prometheus operator, like the namespace, the labels, ...
> >>>>
> >>>> Some documentation is available at:
> >>>>
> >>>> https://camel.apache.org/camel-k/1.7.x/observability/monitoring/integration.html#_discovery
> >>>>
> >>>> That contains some links to the Prometheus operator documentation for troubleshooting why the metrics endpoint is not discovered.
> >>>>
> >>>>> On 25 Nov 2021, at 12:25, Roberto Camelk <be...@gmail.com> wrote:
> >>>>>
> >>>>> I have a Kubernetes running Rancher 2.4.3. I have the cluster
> >>>>> monitoring enabled in rancher, so that exists a Prometheus instance
> >>>>> running, so as a Prometheus Operator.
> >>>>>
> >>>>> Recently I deployed a Apache Camel-K operator, and now I want to
> >>>>> enable the prometheus integration for collect metrics about my camel
> >>>>> routes.
> >>>>>
> >>>>> So, my Camel-K operator is running in namescape camel-k and the
> >>>>> rancher embedded prometheus stack in cattle-prometheus namespace.
> >>>>>
> >>>>> I just have launched my route with the trait --trait
> >>>>> prometheus.enabled=true, but the camel metrics aren't listing at my
> >>>>> prometheus.
> >>>>>
> >>>>> Anyone knows why or what I need to configure to my camel-k route
> >>>>> deploy it's metrics at the rancher embedded prometheus?
> >>>>
> >>
>
Re: Apache Camel-k metrics in rancher embedded prometheus
Posted by Antonin Stefanutti <an...@stefanutti.fr.INVALID>.
Great!
For the metrics, there should be the following registered by the MicroProfile Metrics Camel extension:
https://camel.apache.org/camel-quarkus/2.4.x/reference/extensions/microprofile-metrics.html#_usage
However, the final name is determined by the MicroProfile Metrics specification, so for example the following metric:
camel.context.exchanges.completed.total
Have its name translated to:
application_camel_context_exchanges_completed_total
To check also if the metrics endpoint do have the metrics registered, you can run:
$ kubectl exec deployment/<integration_name> -- curl -s http://localhost:8080/q/metrics | grep application_camel_route_exchanges_completed_total
# HELP application_camel_route_exchanges_completed_total The total number of completed exchanges for a route or Camel Context
# TYPE application_camel_route_exchanges_completed_total counter
application_camel_route_exchanges_completed_total{camelContext="camel-1",routeId="route1"} 33.0
> On 26 Nov 2021, at 14:34, Roberto Camelk <be...@gmail.com> wrote:
>
> Thanks a lot again Antonin. I owe you a beer or coffee!!!
>
> This worked 100%.
>
> To end, another question.... after this, prometheus start scraping the
> metrics from my running route pod, but...
>
> Some metrics IDs are not there, some metrics starting with
> "org_apache_camel_" as I have see in this grafana dashboard sample:
> https://github.com/weimeilin79/camel-k-example-prometheus/blob/ca347f0b8b702b84b7129dfcfcb3b84eff4e2f73/grafana/SampleCamelDashboard.json#L145
>
> The other metrics (camel_k_* -
> https://camel.apache.org/camel-k/next/observability/monitoring/operator.html#metrics)
> are there as so the jvm metrics
> (https://github.com/eclipse/microprofile-metrics/blob/master/spec/src/main/asciidoc/required-metrics.adoc#required-metrics).
>
> Is this correct? Why am I not getting that "org_apache_camel_" metrics?
>
> On Fri, Nov 26, 2021 at 9:13 AM Antonin Stefanutti
> <an...@stefanutti.fr.invalid> wrote:
>>
>> It's likely that you need to configure the Prometheus instance, by editing the Prometheus resource, e.g.:
>>
>> apiVersion: monitoring.coreos.com/v1
>> kind: Prometheus
>> metadata:
>> name: prometheus
>> spec:
>> serviceAccountName: prometheus
>> podMonitorSelector:
>> matchLabels:
>> app: camel-k
>> podMonitorNamespaceSelector: {}
>>
>> It's important to have podMonitorNamespaceSelector: {} to discover all the namespaces, otherwise it's only the resource's namespace.
>>
>> Also you can use the Prometheus trait to set the labels on the Integration PodMonitor accordingly, e.g.:
>>
>> $ kamel run -t prometheus.enabled=true -t prometheus.pod-monitor-labels="app=camel-k"
>>
>> There are some examples in the Prometheus operator documentation:
>>
>> https://github.com/prometheus-operator/prometheus-operator/tree/v0.52.1/example/user-guides/getting-started
>>
>> And the troubleshooting guide at:
>>
>> https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/troubleshooting.md
>>
>>
>>> On 26 Nov 2021, at 12:32, Roberto Camelk <be...@gmail.com> wrote:
>>>
>>> Antonin. Thanks !
>>>
>>> But I continue stuck, please let me share some extra info about my
>>> prometheus operator log:
>>>
>>> level=debug ts=2021-11-26T11:12:05.837624346Z caller=operator.go:1840
>>> component=prometheusoperator msg="filtering namespaces to select
>>> PodMonitors from" namespaces=cattle-prometheus
>>> namespace=cattle-prometheus prometheus=cluster-monitoring
>>> 26/11/2021 08:12:05 level=debug ts=2021-11-26T11:12:05.837687534Z
>>> caller=operator.go:1853 component=prometheusoperator msg="selected
>>> PodMonitors" podmonitors= namespace=cattle-prometheus
>>> prometheus=cluster-monitoring
>>> 26/11/2021 08:12:05 level=debug ts=2021-11-26T11:12:05.942216811Z
>>> caller=operator.go:1677 component=prometheusoperator msg="updating
>>> Prometheus configuration secret skipped, no configuration change"
>>> 26/11/2021 08:12:05 level=debug ts=2021-11-26T11:12:05.950980834Z
>>> caller=operator.go:1776 component=prometheusoperator msg="filtering
>>> namespaces to select ServiceMonitors from"
>>> namespaces=cattle-prometheus,cattle-system,kube-node-lease,kube-public,security-scan,kube-system
>>> namespace=cattle-prometheus prometheus=cluster-monitoring
>>> 26/11/2021 08:12:05 level=debug ts=2021-11-26T11:12:05.951162973Z
>>> caller=operator.go:1810 component=prometheusoperator msg="selected
>>> ServiceMonitors"
>>> servicemonitors=cattle-prometheus/grafana-cluster-monitoring,cattle-prometheus/exporter-kube-etcd-cluster-monitoring,cattle-prometheus/exporter-node-cluster-monitoring,cattle-prometheus/exporter-kube-controller-manager-cluster-monitoring,cattle-prometheus/exporter-kube-state-cluster-monitoring,cattle-prometheus/prometheus-cluster-monitoring,cattle-prometheus/prometheus-operator-monitoring-operator,cattle-prometheus/exporter-fluentd-cluster-monitoring,cattle-prometheus/exporter-kubelets-cluster-monitoring,cattle-prometheus/exporter-kube-scheduler-cluster-monitoring,cattle-prometheus/exporter-kubernetes-cluster-monitoring
>>> namespace=cattle-prometheus prometheus=cluster-monitoring
>>> 26/11/2021 08:12:05 level=debug ts=2021-11-26T11:12:05.977550133Z
>>> caller=operator.go:1741 component=prometheusoperator msg="updated
>>> tlsAssetsSecret" secretname=prometheus-cluster-monitoring-tls-assets
>>> 26/11/2021 08:12:06 level=debug ts=2021-11-26T11:12:06.022196407Z
>>> caller=operator.go:1169 component=prometheusoperator msg="new
>>> statefulset generation inputs match current, skipping any actions"
>>> 26/11/2021 08:12:26 level=debug ts=2021-11-26T11:12:26.321817491Z
>>> caller=operator.go:734 component=prometheusoperator msg="PodMonitor
>>> added"
>>> 26/11/2021 08:12:27 level=debug ts=2021-11-26T11:12:27.755854021Z
>>> caller=operator.go:748 component=prometheusoperator msg="PodMonitor
>>> updated"
>>> 26/11/2021 08:12:46 level=debug ts=2021-11-26T11:12:46.453112794Z
>>> caller=operator.go:748 component=prometheusoperator msg="PodMonitor
>>> updated"
>>> 26/11/2021 08:17:35 level=debug ts=2021-11-26T11:17:35.194031009Z
>>> caller=operator.go:759 component=prometheusoperator msg="PodMonitor
>>> delete"
>>>
>>> This last 4 lines is about my camel-k route, that I ran and stopped.
>>>
>>> So, there are some problems, I think, about the logs above about the
>>> podmonitor selectors telling: "selected PodMonitors" podmonitors=
>>> namespace=cattle-prometheus prometheus=cluster-monitoring
>>>
>>> My camel-k route is running at namespace "platform" and has no label
>>> like "prometheus=cluster-monitoring".
>>>
>>> Do you know how can I fix this? Adding additional scrape configs to
>>> prometheus can solve this? Can you provide a snipet code?
>>>
>>> On Thu, Nov 25, 2021 at 9:56 AM Antonin Stefanutti
>>> <an...@stefanutti.fr.invalid> wrote:
>>>>
>>>> When run an Integration with `kamel run -t prometheus.enabled=true`, a PodMonitor resource is created for the Prometheus operator to reconcile and configure Prometheus to scrape the Integration metrics endpoint.
>>>>
>>>> The PodMonitor metadata must match that of the Prometheus operator, like the namespace, the labels, ...
>>>>
>>>> Some documentation is available at:
>>>>
>>>> https://camel.apache.org/camel-k/1.7.x/observability/monitoring/integration.html#_discovery
>>>>
>>>> That contains some links to the Prometheus operator documentation for troubleshooting why the metrics endpoint is not discovered.
>>>>
>>>>> On 25 Nov 2021, at 12:25, Roberto Camelk <be...@gmail.com> wrote:
>>>>>
>>>>> I have a Kubernetes running Rancher 2.4.3. I have the cluster
>>>>> monitoring enabled in rancher, so that exists a Prometheus instance
>>>>> running, so as a Prometheus Operator.
>>>>>
>>>>> Recently I deployed a Apache Camel-K operator, and now I want to
>>>>> enable the prometheus integration for collect metrics about my camel
>>>>> routes.
>>>>>
>>>>> So, my Camel-K operator is running in namescape camel-k and the
>>>>> rancher embedded prometheus stack in cattle-prometheus namespace.
>>>>>
>>>>> I just have launched my route with the trait --trait
>>>>> prometheus.enabled=true, but the camel metrics aren't listing at my
>>>>> prometheus.
>>>>>
>>>>> Anyone knows why or what I need to configure to my camel-k route
>>>>> deploy it's metrics at the rancher embedded prometheus?
>>>>
>>
Re: Apache Camel-k metrics in rancher embedded prometheus
Posted by Roberto Camelk <be...@gmail.com>.
Thanks a lot again Antonin. I owe you a beer or coffee!!!
This worked 100%.
To end, another question.... after this, prometheus start scraping the
metrics from my running route pod, but...
Some metrics IDs are not there, some metrics starting with
"org_apache_camel_" as I have see in this grafana dashboard sample:
https://github.com/weimeilin79/camel-k-example-prometheus/blob/ca347f0b8b702b84b7129dfcfcb3b84eff4e2f73/grafana/SampleCamelDashboard.json#L145
The other metrics (camel_k_* -
https://camel.apache.org/camel-k/next/observability/monitoring/operator.html#metrics)
are there as so the jvm metrics
(https://github.com/eclipse/microprofile-metrics/blob/master/spec/src/main/asciidoc/required-metrics.adoc#required-metrics).
Is this correct? Why am I not getting that "org_apache_camel_" metrics?
On Fri, Nov 26, 2021 at 9:13 AM Antonin Stefanutti
<an...@stefanutti.fr.invalid> wrote:
>
> It's likely that you need to configure the Prometheus instance, by editing the Prometheus resource, e.g.:
>
> apiVersion: monitoring.coreos.com/v1
> kind: Prometheus
> metadata:
> name: prometheus
> spec:
> serviceAccountName: prometheus
> podMonitorSelector:
> matchLabels:
> app: camel-k
> podMonitorNamespaceSelector: {}
>
> It's important to have podMonitorNamespaceSelector: {} to discover all the namespaces, otherwise it's only the resource's namespace.
>
> Also you can use the Prometheus trait to set the labels on the Integration PodMonitor accordingly, e.g.:
>
> $ kamel run -t prometheus.enabled=true -t prometheus.pod-monitor-labels="app=camel-k"
>
> There are some examples in the Prometheus operator documentation:
>
> https://github.com/prometheus-operator/prometheus-operator/tree/v0.52.1/example/user-guides/getting-started
>
> And the troubleshooting guide at:
>
> https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/troubleshooting.md
>
>
> > On 26 Nov 2021, at 12:32, Roberto Camelk <be...@gmail.com> wrote:
> >
> > Antonin. Thanks !
> >
> > But I continue stuck, please let me share some extra info about my
> > prometheus operator log:
> >
> > level=debug ts=2021-11-26T11:12:05.837624346Z caller=operator.go:1840
> > component=prometheusoperator msg="filtering namespaces to select
> > PodMonitors from" namespaces=cattle-prometheus
> > namespace=cattle-prometheus prometheus=cluster-monitoring
> > 26/11/2021 08:12:05 level=debug ts=2021-11-26T11:12:05.837687534Z
> > caller=operator.go:1853 component=prometheusoperator msg="selected
> > PodMonitors" podmonitors= namespace=cattle-prometheus
> > prometheus=cluster-monitoring
> > 26/11/2021 08:12:05 level=debug ts=2021-11-26T11:12:05.942216811Z
> > caller=operator.go:1677 component=prometheusoperator msg="updating
> > Prometheus configuration secret skipped, no configuration change"
> > 26/11/2021 08:12:05 level=debug ts=2021-11-26T11:12:05.950980834Z
> > caller=operator.go:1776 component=prometheusoperator msg="filtering
> > namespaces to select ServiceMonitors from"
> > namespaces=cattle-prometheus,cattle-system,kube-node-lease,kube-public,security-scan,kube-system
> > namespace=cattle-prometheus prometheus=cluster-monitoring
> > 26/11/2021 08:12:05 level=debug ts=2021-11-26T11:12:05.951162973Z
> > caller=operator.go:1810 component=prometheusoperator msg="selected
> > ServiceMonitors"
> > servicemonitors=cattle-prometheus/grafana-cluster-monitoring,cattle-prometheus/exporter-kube-etcd-cluster-monitoring,cattle-prometheus/exporter-node-cluster-monitoring,cattle-prometheus/exporter-kube-controller-manager-cluster-monitoring,cattle-prometheus/exporter-kube-state-cluster-monitoring,cattle-prometheus/prometheus-cluster-monitoring,cattle-prometheus/prometheus-operator-monitoring-operator,cattle-prometheus/exporter-fluentd-cluster-monitoring,cattle-prometheus/exporter-kubelets-cluster-monitoring,cattle-prometheus/exporter-kube-scheduler-cluster-monitoring,cattle-prometheus/exporter-kubernetes-cluster-monitoring
> > namespace=cattle-prometheus prometheus=cluster-monitoring
> > 26/11/2021 08:12:05 level=debug ts=2021-11-26T11:12:05.977550133Z
> > caller=operator.go:1741 component=prometheusoperator msg="updated
> > tlsAssetsSecret" secretname=prometheus-cluster-monitoring-tls-assets
> > 26/11/2021 08:12:06 level=debug ts=2021-11-26T11:12:06.022196407Z
> > caller=operator.go:1169 component=prometheusoperator msg="new
> > statefulset generation inputs match current, skipping any actions"
> > 26/11/2021 08:12:26 level=debug ts=2021-11-26T11:12:26.321817491Z
> > caller=operator.go:734 component=prometheusoperator msg="PodMonitor
> > added"
> > 26/11/2021 08:12:27 level=debug ts=2021-11-26T11:12:27.755854021Z
> > caller=operator.go:748 component=prometheusoperator msg="PodMonitor
> > updated"
> > 26/11/2021 08:12:46 level=debug ts=2021-11-26T11:12:46.453112794Z
> > caller=operator.go:748 component=prometheusoperator msg="PodMonitor
> > updated"
> > 26/11/2021 08:17:35 level=debug ts=2021-11-26T11:17:35.194031009Z
> > caller=operator.go:759 component=prometheusoperator msg="PodMonitor
> > delete"
> >
> > This last 4 lines is about my camel-k route, that I ran and stopped.
> >
> > So, there are some problems, I think, about the logs above about the
> > podmonitor selectors telling: "selected PodMonitors" podmonitors=
> > namespace=cattle-prometheus prometheus=cluster-monitoring
> >
> > My camel-k route is running at namespace "platform" and has no label
> > like "prometheus=cluster-monitoring".
> >
> > Do you know how can I fix this? Adding additional scrape configs to
> > prometheus can solve this? Can you provide a snipet code?
> >
> > On Thu, Nov 25, 2021 at 9:56 AM Antonin Stefanutti
> > <an...@stefanutti.fr.invalid> wrote:
> >>
> >> When run an Integration with `kamel run -t prometheus.enabled=true`, a PodMonitor resource is created for the Prometheus operator to reconcile and configure Prometheus to scrape the Integration metrics endpoint.
> >>
> >> The PodMonitor metadata must match that of the Prometheus operator, like the namespace, the labels, ...
> >>
> >> Some documentation is available at:
> >>
> >> https://camel.apache.org/camel-k/1.7.x/observability/monitoring/integration.html#_discovery
> >>
> >> That contains some links to the Prometheus operator documentation for troubleshooting why the metrics endpoint is not discovered.
> >>
> >>> On 25 Nov 2021, at 12:25, Roberto Camelk <be...@gmail.com> wrote:
> >>>
> >>> I have a Kubernetes running Rancher 2.4.3. I have the cluster
> >>> monitoring enabled in rancher, so that exists a Prometheus instance
> >>> running, so as a Prometheus Operator.
> >>>
> >>> Recently I deployed a Apache Camel-K operator, and now I want to
> >>> enable the prometheus integration for collect metrics about my camel
> >>> routes.
> >>>
> >>> So, my Camel-K operator is running in namescape camel-k and the
> >>> rancher embedded prometheus stack in cattle-prometheus namespace.
> >>>
> >>> I just have launched my route with the trait --trait
> >>> prometheus.enabled=true, but the camel metrics aren't listing at my
> >>> prometheus.
> >>>
> >>> Anyone knows why or what I need to configure to my camel-k route
> >>> deploy it's metrics at the rancher embedded prometheus?
> >>
>
Re: Apache Camel-k metrics in rancher embedded prometheus
Posted by Antonin Stefanutti <an...@stefanutti.fr.INVALID>.
It's likely that you need to configure the Prometheus instance, by editing the Prometheus resource, e.g.:
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
name: prometheus
spec:
serviceAccountName: prometheus
podMonitorSelector:
matchLabels:
app: camel-k
podMonitorNamespaceSelector: {}
It's important to have podMonitorNamespaceSelector: {} to discover all the namespaces, otherwise it's only the resource's namespace.
Also you can use the Prometheus trait to set the labels on the Integration PodMonitor accordingly, e.g.:
$ kamel run -t prometheus.enabled=true -t prometheus.pod-monitor-labels="app=camel-k"
There are some examples in the Prometheus operator documentation:
https://github.com/prometheus-operator/prometheus-operator/tree/v0.52.1/example/user-guides/getting-started
And the troubleshooting guide at:
https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/troubleshooting.md
> On 26 Nov 2021, at 12:32, Roberto Camelk <be...@gmail.com> wrote:
>
> Antonin. Thanks !
>
> But I continue stuck, please let me share some extra info about my
> prometheus operator log:
>
> level=debug ts=2021-11-26T11:12:05.837624346Z caller=operator.go:1840
> component=prometheusoperator msg="filtering namespaces to select
> PodMonitors from" namespaces=cattle-prometheus
> namespace=cattle-prometheus prometheus=cluster-monitoring
> 26/11/2021 08:12:05 level=debug ts=2021-11-26T11:12:05.837687534Z
> caller=operator.go:1853 component=prometheusoperator msg="selected
> PodMonitors" podmonitors= namespace=cattle-prometheus
> prometheus=cluster-monitoring
> 26/11/2021 08:12:05 level=debug ts=2021-11-26T11:12:05.942216811Z
> caller=operator.go:1677 component=prometheusoperator msg="updating
> Prometheus configuration secret skipped, no configuration change"
> 26/11/2021 08:12:05 level=debug ts=2021-11-26T11:12:05.950980834Z
> caller=operator.go:1776 component=prometheusoperator msg="filtering
> namespaces to select ServiceMonitors from"
> namespaces=cattle-prometheus,cattle-system,kube-node-lease,kube-public,security-scan,kube-system
> namespace=cattle-prometheus prometheus=cluster-monitoring
> 26/11/2021 08:12:05 level=debug ts=2021-11-26T11:12:05.951162973Z
> caller=operator.go:1810 component=prometheusoperator msg="selected
> ServiceMonitors"
> servicemonitors=cattle-prometheus/grafana-cluster-monitoring,cattle-prometheus/exporter-kube-etcd-cluster-monitoring,cattle-prometheus/exporter-node-cluster-monitoring,cattle-prometheus/exporter-kube-controller-manager-cluster-monitoring,cattle-prometheus/exporter-kube-state-cluster-monitoring,cattle-prometheus/prometheus-cluster-monitoring,cattle-prometheus/prometheus-operator-monitoring-operator,cattle-prometheus/exporter-fluentd-cluster-monitoring,cattle-prometheus/exporter-kubelets-cluster-monitoring,cattle-prometheus/exporter-kube-scheduler-cluster-monitoring,cattle-prometheus/exporter-kubernetes-cluster-monitoring
> namespace=cattle-prometheus prometheus=cluster-monitoring
> 26/11/2021 08:12:05 level=debug ts=2021-11-26T11:12:05.977550133Z
> caller=operator.go:1741 component=prometheusoperator msg="updated
> tlsAssetsSecret" secretname=prometheus-cluster-monitoring-tls-assets
> 26/11/2021 08:12:06 level=debug ts=2021-11-26T11:12:06.022196407Z
> caller=operator.go:1169 component=prometheusoperator msg="new
> statefulset generation inputs match current, skipping any actions"
> 26/11/2021 08:12:26 level=debug ts=2021-11-26T11:12:26.321817491Z
> caller=operator.go:734 component=prometheusoperator msg="PodMonitor
> added"
> 26/11/2021 08:12:27 level=debug ts=2021-11-26T11:12:27.755854021Z
> caller=operator.go:748 component=prometheusoperator msg="PodMonitor
> updated"
> 26/11/2021 08:12:46 level=debug ts=2021-11-26T11:12:46.453112794Z
> caller=operator.go:748 component=prometheusoperator msg="PodMonitor
> updated"
> 26/11/2021 08:17:35 level=debug ts=2021-11-26T11:17:35.194031009Z
> caller=operator.go:759 component=prometheusoperator msg="PodMonitor
> delete"
>
> This last 4 lines is about my camel-k route, that I ran and stopped.
>
> So, there are some problems, I think, about the logs above about the
> podmonitor selectors telling: "selected PodMonitors" podmonitors=
> namespace=cattle-prometheus prometheus=cluster-monitoring
>
> My camel-k route is running at namespace "platform" and has no label
> like "prometheus=cluster-monitoring".
>
> Do you know how can I fix this? Adding additional scrape configs to
> prometheus can solve this? Can you provide a snipet code?
>
> On Thu, Nov 25, 2021 at 9:56 AM Antonin Stefanutti
> <an...@stefanutti.fr.invalid> wrote:
>>
>> When run an Integration with `kamel run -t prometheus.enabled=true`, a PodMonitor resource is created for the Prometheus operator to reconcile and configure Prometheus to scrape the Integration metrics endpoint.
>>
>> The PodMonitor metadata must match that of the Prometheus operator, like the namespace, the labels, ...
>>
>> Some documentation is available at:
>>
>> https://camel.apache.org/camel-k/1.7.x/observability/monitoring/integration.html#_discovery
>>
>> That contains some links to the Prometheus operator documentation for troubleshooting why the metrics endpoint is not discovered.
>>
>>> On 25 Nov 2021, at 12:25, Roberto Camelk <be...@gmail.com> wrote:
>>>
>>> I have a Kubernetes running Rancher 2.4.3. I have the cluster
>>> monitoring enabled in rancher, so that exists a Prometheus instance
>>> running, so as a Prometheus Operator.
>>>
>>> Recently I deployed a Apache Camel-K operator, and now I want to
>>> enable the prometheus integration for collect metrics about my camel
>>> routes.
>>>
>>> So, my Camel-K operator is running in namescape camel-k and the
>>> rancher embedded prometheus stack in cattle-prometheus namespace.
>>>
>>> I just have launched my route with the trait --trait
>>> prometheus.enabled=true, but the camel metrics aren't listing at my
>>> prometheus.
>>>
>>> Anyone knows why or what I need to configure to my camel-k route
>>> deploy it's metrics at the rancher embedded prometheus?
>>
Re: Apache Camel-k metrics in rancher embedded prometheus
Posted by Roberto Camelk <be...@gmail.com>.
Antonin. Thanks !
But I continue stuck, please let me share some extra info about my
prometheus operator log:
level=debug ts=2021-11-26T11:12:05.837624346Z caller=operator.go:1840
component=prometheusoperator msg="filtering namespaces to select
PodMonitors from" namespaces=cattle-prometheus
namespace=cattle-prometheus prometheus=cluster-monitoring
26/11/2021 08:12:05 level=debug ts=2021-11-26T11:12:05.837687534Z
caller=operator.go:1853 component=prometheusoperator msg="selected
PodMonitors" podmonitors= namespace=cattle-prometheus
prometheus=cluster-monitoring
26/11/2021 08:12:05 level=debug ts=2021-11-26T11:12:05.942216811Z
caller=operator.go:1677 component=prometheusoperator msg="updating
Prometheus configuration secret skipped, no configuration change"
26/11/2021 08:12:05 level=debug ts=2021-11-26T11:12:05.950980834Z
caller=operator.go:1776 component=prometheusoperator msg="filtering
namespaces to select ServiceMonitors from"
namespaces=cattle-prometheus,cattle-system,kube-node-lease,kube-public,security-scan,kube-system
namespace=cattle-prometheus prometheus=cluster-monitoring
26/11/2021 08:12:05 level=debug ts=2021-11-26T11:12:05.951162973Z
caller=operator.go:1810 component=prometheusoperator msg="selected
ServiceMonitors"
servicemonitors=cattle-prometheus/grafana-cluster-monitoring,cattle-prometheus/exporter-kube-etcd-cluster-monitoring,cattle-prometheus/exporter-node-cluster-monitoring,cattle-prometheus/exporter-kube-controller-manager-cluster-monitoring,cattle-prometheus/exporter-kube-state-cluster-monitoring,cattle-prometheus/prometheus-cluster-monitoring,cattle-prometheus/prometheus-operator-monitoring-operator,cattle-prometheus/exporter-fluentd-cluster-monitoring,cattle-prometheus/exporter-kubelets-cluster-monitoring,cattle-prometheus/exporter-kube-scheduler-cluster-monitoring,cattle-prometheus/exporter-kubernetes-cluster-monitoring
namespace=cattle-prometheus prometheus=cluster-monitoring
26/11/2021 08:12:05 level=debug ts=2021-11-26T11:12:05.977550133Z
caller=operator.go:1741 component=prometheusoperator msg="updated
tlsAssetsSecret" secretname=prometheus-cluster-monitoring-tls-assets
26/11/2021 08:12:06 level=debug ts=2021-11-26T11:12:06.022196407Z
caller=operator.go:1169 component=prometheusoperator msg="new
statefulset generation inputs match current, skipping any actions"
26/11/2021 08:12:26 level=debug ts=2021-11-26T11:12:26.321817491Z
caller=operator.go:734 component=prometheusoperator msg="PodMonitor
added"
26/11/2021 08:12:27 level=debug ts=2021-11-26T11:12:27.755854021Z
caller=operator.go:748 component=prometheusoperator msg="PodMonitor
updated"
26/11/2021 08:12:46 level=debug ts=2021-11-26T11:12:46.453112794Z
caller=operator.go:748 component=prometheusoperator msg="PodMonitor
updated"
26/11/2021 08:17:35 level=debug ts=2021-11-26T11:17:35.194031009Z
caller=operator.go:759 component=prometheusoperator msg="PodMonitor
delete"
This last 4 lines is about my camel-k route, that I ran and stopped.
So, there are some problems, I think, about the logs above about the
podmonitor selectors telling: "selected PodMonitors" podmonitors=
namespace=cattle-prometheus prometheus=cluster-monitoring
My camel-k route is running at namespace "platform" and has no label
like "prometheus=cluster-monitoring".
Do you know how can I fix this? Adding additional scrape configs to
prometheus can solve this? Can you provide a snipet code?
On Thu, Nov 25, 2021 at 9:56 AM Antonin Stefanutti
<an...@stefanutti.fr.invalid> wrote:
>
> When run an Integration with `kamel run -t prometheus.enabled=true`, a PodMonitor resource is created for the Prometheus operator to reconcile and configure Prometheus to scrape the Integration metrics endpoint.
>
> The PodMonitor metadata must match that of the Prometheus operator, like the namespace, the labels, ...
>
> Some documentation is available at:
>
> https://camel.apache.org/camel-k/1.7.x/observability/monitoring/integration.html#_discovery
>
> That contains some links to the Prometheus operator documentation for troubleshooting why the metrics endpoint is not discovered.
>
> > On 25 Nov 2021, at 12:25, Roberto Camelk <be...@gmail.com> wrote:
> >
> > I have a Kubernetes running Rancher 2.4.3. I have the cluster
> > monitoring enabled in rancher, so that exists a Prometheus instance
> > running, so as a Prometheus Operator.
> >
> > Recently I deployed a Apache Camel-K operator, and now I want to
> > enable the prometheus integration for collect metrics about my camel
> > routes.
> >
> > So, my Camel-K operator is running in namescape camel-k and the
> > rancher embedded prometheus stack in cattle-prometheus namespace.
> >
> > I just have launched my route with the trait --trait
> > prometheus.enabled=true, but the camel metrics aren't listing at my
> > prometheus.
> >
> > Anyone knows why or what I need to configure to my camel-k route
> > deploy it's metrics at the rancher embedded prometheus?
>
Re: Apache Camel-k metrics in rancher embedded prometheus
Posted by Antonin Stefanutti <an...@stefanutti.fr.INVALID>.
When run an Integration with `kamel run -t prometheus.enabled=true`, a PodMonitor resource is created for the Prometheus operator to reconcile and configure Prometheus to scrape the Integration metrics endpoint.
The PodMonitor metadata must match that of the Prometheus operator, like the namespace, the labels, ...
Some documentation is available at:
https://camel.apache.org/camel-k/1.7.x/observability/monitoring/integration.html#_discovery
That contains some links to the Prometheus operator documentation for troubleshooting why the metrics endpoint is not discovered.
> On 25 Nov 2021, at 12:25, Roberto Camelk <be...@gmail.com> wrote:
>
> I have a Kubernetes running Rancher 2.4.3. I have the cluster
> monitoring enabled in rancher, so that exists a Prometheus instance
> running, so as a Prometheus Operator.
>
> Recently I deployed a Apache Camel-K operator, and now I want to
> enable the prometheus integration for collect metrics about my camel
> routes.
>
> So, my Camel-K operator is running in namescape camel-k and the
> rancher embedded prometheus stack in cattle-prometheus namespace.
>
> I just have launched my route with the trait --trait
> prometheus.enabled=true, but the camel metrics aren't listing at my
> prometheus.
>
> Anyone knows why or what I need to configure to my camel-k route
> deploy it's metrics at the rancher embedded prometheus?