[jira] [Updated] (FLINK-39541) Improve operator metrics documentation and bundle additional metric reporters

["Dennis-Mircea Ciupitu (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-39541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Dennis-Mircea Ciupitu updated FLINK-39541:
------------------------------------------
    Labels: pull-request-available  (was: )

> Improve operator metrics documentation and bundle additional metric reporters
> -----------------------------------------------------------------------------
>
>                 Key: FLINK-39541
>                 URL: https://issues.apache.org/jira/browse/FLINK-39541
>             Project: Flink
>          Issue Type: Improvement
>          Components: Kubernetes Operator
>    Affects Versions: kubernetes-operator-1.14.0
>            Reporter: Dennis-Mircea Ciupitu
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: kubernetes-operator-1.15.0
>
>
> h1. Summary
> The current {{Metrics and Logging}} page of the Flink Kubernetes Operator is terse and, in several places, ambiguous or incomplete. It mixes {{FlinkDeployment}} / {{FlinkSessionJob}} / {{FlinkBlueGreenDeployment}} metrics in a single flat markdown table, does not cover {{FlinkStateSnapshot}} or autoscaler metrics at all, provides no explanation of how scope formats translate into reporter output, and does not describe the operator-side configuration surface for metric reporters. On top of that, the operator image bundles only a subset of the Flink metric reporter plugins commonly used in production.
> This improvement rewrites the page end-to-end, extends the operator image with two additional bundled reporters, and clarifies in code (javadoc) the relationship between operator-scoped and Flink-core metric configuration.
> h1. Motivation
>  * The existing {{Flink Resource Metrics}} markdown table conflates scope (System / Namespace) with resource type ({{{}FlinkDeployment{}}} / {{FlinkSessionJob}} / {{{}FlinkBlueGreenDeployment{}}}) and is hard to read. Several metric groups emitted by the operator are not listed at all.
>  * Autoscaler metrics ({{{}AutoScaler.scalings{}}}, {{{}AutoScaler.errors{}}}, {{{}AutoScaler.balanced{}}}, and the per-vertex {{{}AutoScaler.jobVertexID.<jobVertexID>.<ScalingMetric>.{Current,Average{}}}} gauges) are emitted in code but completely undocumented. Users have no reference for the placeholder values ({{{}LAG{}}}, {{{}LOAD{}}}, {{{}TRUE_PROCESSING_RATE{}}}, {{{}TARGET_DATA_RATE{}}}, etc.) or for which of them also emit an {{{}.Average{}}}.
>  * {{FlinkStateSnapshot}} checkpoint/savepoint gauges ({{{}Checkpoint.Count{}}}, {{{}Checkpoint.State.<SnapshotState>.Count{}}}, {{{}Savepoint.Count{}}}, {{{}Savepoint.State.<SnapshotState>.Count{}}}) are emitted by the snapshot tracker but not documented.
>  * Namespace-level {{{}FlinkMinorVersion.<FlinkMinorVersion>.Count{}}}, {{ResourceUsage.StateSize}} and {{FlinkDeployment.JmDeploymentStatus.<Status>.Count}} are either missing or only mentioned in passing.
>  * There is no explanation of how scope formats ({{{}<host>.k8soperator.<namespace>.<name>.system{}}}, etc.) map to actual reporter output, and in particular no explanation of the asymmetry between non-labeling reporters (SLF4J, JMX, Graphite, …) and labeling reporters (Prometheus, Datadog, InfluxDB, …), even though the latter drop scope-format literals like system / namespace / resource and surface variables as labels.
>  * The {{JOSDK Metrics}} subsection claims metrics are forwarded but does not link to the upstream reference where users can look the names up.
>  * The operator image bundles SLF4J, Prometheus, JMX, Graphite, InfluxDB, Datadog and StatsD plugins, but not Dropwizard or OpenTelemetry, forcing users to build a custom image for both common setups.
>  * The metric-reporter examples section does not mention that reporter options in {{spec.flinkConfiguration}} must use the plain {{metrics.reporter.{*}{*}}} \{*}prefix (consumed by JM/TMs), while the operator JVM uses {{kubernetes.operator.metrics.reporter.}}{*}. This is a frequent source of misconfiguration.
>  * {{KubernetesOperatorMetricOptions}} only declares operator-specific toggles; it is not obvious from the code that every Flink {{metrics.*}} key is also honoured when written with the {{kubernetes.operator.}} prefix (the prefix is stripped at startup and the remainder is forwarded to Flink's metric registry). This routing is undocumented.
> h1. Proposed Change
> h2. Documentation
>  # *New {{Scope}} section*
>  ** Add a typed table of the three scopes (System / Namespace / Resource) with their configuration option and default scope format.
>  ** Document all scope variables ({{{}<host>{}}}, {{{}<namespace>{}}}, {{{}<name>{}}}, {{{}<resourcens>{}}}, {{{}<resourcename>{}}}, {{{}<resourcetype>{}}}).
>  ** Add a {{How Metric Identifiers Are Built}} subsection explaining the distinction between scope components (variable substitution in the scope format) and logical scope (operator metric-group chain: {{{}k8soperator{}}}, {{{}k8soperator.namespace{}}}, {{{}k8soperator.namespace.resource{}}}).
>  ** Explain how non-labeling reporters build the identifier from *scope components + metric* name, while labeling reporters build it from *logical scope + metric* name and expose scope variables as labels/tags. Include an info hint that labeling reporters drop scope-format literals.
>  ** Add a Concrete Example subsection with Prometheus and SLF4J/JMX renderings for System, Namespace and Resource scopes, using real metrics ({{{}Lifecycle.State.<State>.TimeSeconds{}}}, {{{}Lifecycle.State.<State>.Count{}}}, {{{}AutoScaler.<jobVertexID>.TRUE_PROCESSING_RATE.Current{}}}).
>  # *Rewritten {{Operator Custom Resource Metrics}} table*
>  ** Replace the previous flat markdown table ({{{}### Flink Resource Metrics{}}}) with a single Scope / Resource type / Metrics / Description / Type table, adopting the {{<table class="table table-bordered">}} markup used by the core Flink documentation (this is a new styling choice, not a retrofit from a pre-existing HTML table).
>  ** Group rows by scope (System / Namespace / Resource) and, within each scope, by resource type (FlinkBlueGreenDeployment, FlinkDeployment, FlinkDeployment, FlinkSessionJob, FlinkStateSnapshot).
>  ** Add previously undocumented rows:
>  *** {{{}FlinkBlueGreenDeployment.Failures{}}}, {{{}FlinkBlueGreenDeployment.JobStatus.<Status>.Count{}}}.
>  *** {{{}FlinkDeployment.FlinkMinorVersion.<FlinkMinorVersion>.Count{}}}, {{{}FlinkDeployment.JmDeploymentStatus.<Status>.Count{}}}, {{{}ResourceUsage.Cpu/Memory/StateSize{}}}.
>  *** All FlinkStateSnapshot checkpoint/savepoint gauges.
>  *** All autoscaler resource-scoped rows ({{{}AutoScaler.scalings{}}}, {{{}AutoScaler.errors, AutoScaler.balanced{}}}, {{{}AutoScaler.jobVertexID.<jobVertexID>.<ScalingMetric>.Current{}}}).
>  # *New per-topic subsections (narrative + tables), ordered for readability*
>  ** {{FlinkDeployment Version and Resource Usage}} (new): introductory paragraph on fleet-wide Flink version adoption, capacity / quota monitoring, plus bullet list of the involved gauges.
>  ** {{FlinkDeployment / FlinkSessionJob Lifecycle metrics}} (renamed from the old Lifecycle metrics; clarifies that it covers FlinkDeployment and FlinkSessionJob only).
>  ** {{FlinkBlueGreenDeployment Lifecycle metrics}} (kept, minor rewording).
>  ** FlinkDeployment / FlinkSessionJob JobStatus Tracking (new): narrative on how JmDeploymentStatus complements lifecycle metrics; placed before the blue-green JobStatus subsection.
>  ** {{FlinkBlueGreenDeployment JobStatus Tracking}} (kept, minor rewording).
>  ** {{FlinkStateSnapshot State Tracking}} (new): narrative on detecting stuck / failing snapshot pipelines; placed before Scaling metrics.
>  ** {{Scaling metrics}} (new): narrative on what the autoscaler counters and per-vertex gauges are useful for, plus a new alphabetically-ordered table listing every valid {{<ScalingMetric>}} value ({{{}CATCH_UP_DATA_RATE{}}}, E\{{XPECTED_PROCESSING_RATE}}, {{{}LAG{}}}, {{{}LOAD{}}}, {{{}MAX_PARALLELISM{}}}, {{{}NUM_SOURCE_PARTITIONS{}}}, {{{}PARALLELISM, RECOMMENDED_PARALLELISM{}}}, {{{}SCALE_DOWN_RATE_THRESHOLD{}}}, {{{}SCALE_UP_RATE_THRESHOLD{}}}, {{{}TARGET_DATA_RATE{}}}, {{{}TRUE_PROCESSING_RATE{}}}) with a short description and an {{.Average emitted?}} column.
>  # *Kubernetes Client Metrics tables*
>  ** Convert both existing markdown tables (default + {{{}http.response.code.groups.enabled{}}}) to the same {{<table class="table table-bordered">}} style now used by the new {{Operator Custom Resource Metrics}} table, keeping row content but sorting metric names alphabetically within each table for stable navigation.
>  # *JOSDK Metrics subsection*
>  ** Keep the current short paragraph but add a link to the upstream JOSDK metrics documentation as the authoritative reference, and note that JOSDK-owned metrics are subject to the same scope/reporter rules as the rest of the operator metrics.
>  # *Metric Reporters section*
>  ** Enumerate the reporter plugins actually bundled in the operator image: SLF4J, Prometheus, JMX, Graphite, InfluxDB, Datadog, StatsD, Dropwizard, OpenTelemetry. Mention that any other {{MetricReporterFactory}} can be added by dropping its plugin jar under \{{/opt/flink/plugins/<name>/ }}in a custom image.
>  ** New {{Operator-scoped Metric Configuration }}subsection: explain that the operator accepts the standard Flink {{metrics.*}} keys under the {{kubernetes.operator.metrics.*}} prefix, that the kubernetes.operator. prefix is stripped at startup and the remainder forwarded to the operator's Flink metric registry, and that reporter options therefore follow Flink's schema verbatim and are not re-declared on the Configuration page.
>  ** New {{Configuring Reporters on a FlinkDeployment}} example: clarify that reporters for a managed Flink cluster (spec.flinkConfiguration) use the plain {{metrics.reporter.*}} prefix, while {{kubernetes.operator.metrics.reporter.*}} is reserved for the operator JVM.
>  ** {{Rename How to Enable Prometheus}} (Example) → Prometheus, rename Set up Prometheus locally → Monitoring the Operator with Prometheus, and drop the metrics.reporter.prom.interval line from the Prometheus example (Prometheus is pull-based).
> h2. Image / Packaging
>  * Extend the {{maven-dependency-plugin}} {{artifactItems}} to copy *flink-metrics-dropwizard* and *flink-metrics-otel* under {{{}${plugins.tmp.dir}/{}}}, so both plugins end up in {{/opt/flink/plugins/}} of the operator image. This matches the updated {{Metric Reporters}} documentation.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)