You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by gy...@apache.org on 2022/03/16 12:02:22 UTC
[flink-kubernetes-operator] branch main updated: [FLINK-26662] Document logging & metrics configs for Operator
This is an automated email from the ASF dual-hosted git repository.
gyfora pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/flink-kubernetes-operator.git
The following commit(s) were added to refs/heads/main by this push:
new 4ba1213 [FLINK-26662] Document logging & metrics configs for Operator
4ba1213 is described below
commit 4ba12138a229dab1ef2685079821798feb22708d
Author: Matyas Orhidi <ma...@apple.com>
AuthorDate: Tue Mar 15 18:22:58 2022 +0100
[FLINK-26662] Document logging & metrics configs for Operator
---
README.md | 73 +-------------------
docs/content/docs/operations/metrics-logging.md | 90 ++++++++++++++++++++++++-
2 files changed, 90 insertions(+), 73 deletions(-)
diff --git a/README.md b/README.md
index 5e0a30f..9b2a092 100644
--- a/README.md
+++ b/README.md
@@ -96,75 +96,4 @@ Workflows are custom automated processes that you can set up in your repository
Considering the cost of running the builds, the stability, and the maintainability, flink-kubernetes-operator chose GitHub Actions and build the whole CI/CD solution on it.
All the unit tests, integration tests, and the end-to-end tests will be triggered for each PR.
-Note: Please make sure the CI passed before merging.
-
-## Operator Metrics
-
-The operator extends the [Flink Metric System](https://nightlies.apache.org/flink/flink-docs-master/docs/ops/metrics/) that allows gathering and exposing metrics to centralized monitoring solutions. The well known [Metric Reporters](https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/metric_reporters) are shipped in the operator image and are ready to use.
-
-### Slf4j
-The default metrics reporter in the operator is Slf4j. It does not require any external monitoring systems, and it is enabled in the operator [Helm chart](helm/flink-operator/templates/flink-operator.yaml) by default, mainly for demonstrating purposes.
-```properties
-metrics.reporter.slf4j.factory.class: org.apache.flink.metrics.slf4j.Slf4jReporterFactory
-metrics.reporter.slf4j.interval: 1 MINUTE
-```
-To use a more robust production grade monitoring solution the configuration needs to be changed.
-
-### Prometheus
-The following example shows how to enable the Prometheus metric reporter:
-```properties
-metrics.reporter.prom.class: org.apache.flink.metrics.prometheus.PrometheusReporter
-metrics.reporter.prom.port: 9999
-```
-Some metric reporters, including the Prometheus, needs a port to be exposed on the container. This can be achieved be defining a value for the otherwise empty `metrics.port` variable.
-Either in the [values.yaml](helm/flink-operator/values.yaml) file:
-```yaml
-metrics:
- port: 9999
-```
-or using the option `--set metrics.port=9999` in the command line.
-
-The Prometheus Operator among other options provides an elegant, declarative way to specify how group of pods should be monitored using custom resources.
-
-To install the Prometheus operator via Helm run:
-
-```shell
-helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
-helm install prometheus prometheus-community/kube-prometheus-stack
-```
-The Grafana dashboard can be accessed through port-forwarding:
-```shell
-kubectl port-forward deployment/prometheus-grafana 3000
-```
-To enable the operator metrics in Prometheus create a `pod-monitor.yaml` file with the following content:
-```yaml
-apiVersion: monitoring.coreos.com/v1
-kind: PodMonitor
-metadata:
- name: flink-operator
- labels:
- release: prometheus
-spec:
- selector:
- matchLabels:
- app.kubernetes.io/name: flink-operator
- podMetricsEndpoints:
- - port: metrics
-```
-and apply it on your Kubernetes environment:
-```shell
-kubectl create -f pod-monitor.yaml
-```
-Once the custom resource is created in the Kubernetes environment the operator metrics are ready to explore [http://localhost:3000/explore](http://localhost:3000/explore).
-
-# Savepoints
-Savepoints can be triggered manually by defining a random (nonce) value to the variable `savepointTriggerNonce` in the job specification:
-```yaml
- job:
- jarURI: local:///opt/flink/examples/streaming/StateMachineExample.jar
- parallelism: 2
- upgradeMode: savepoint
- state: running
- savepointTriggerNonce: 123
-```
-The operator will trigger a savepoint every time the modified CR is applied and the nonce is different from the previous value.
+Note: Please make sure the CI passed before merging.
\ No newline at end of file
diff --git a/docs/content/docs/operations/metrics-logging.md b/docs/content/docs/operations/metrics-logging.md
index c430f14..aa9b8d4 100644
--- a/docs/content/docs/operations/metrics-logging.md
+++ b/docs/content/docs/operations/metrics-logging.md
@@ -24,4 +24,92 @@ specific language governing permissions and limitations
under the License.
-->
-# Metrics and Logging
+# Metrics
+
+The Flink Kubernetes Operator (Operator) extends the [Flink Metric System](https://nightlies.apache.org/flink/flink-docs-master/docs/ops/metrics/) that allows gathering and exposing metrics to centralized monitoring solutions. The well known [Metric Reporters](https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/metric_reporters) are shipped in the operator image and are ready to use.
+
+## Slf4j
+The default metrics reporter in the operator is Slf4j. It does not require any external monitoring systems, and it is enabled in the `values.yaml` file by default, mainly for demonstrating purposes.
+```yaml
+operatorConfiguration:
+ create: true
+ append: true
+ flink-conf.yaml: |+
+ metrics.reporter.slf4j.factory.class: org.apache.flink.metrics.slf4j.Slf4jReporterFactory
+ metrics.reporter.slf4j.interval: 5 MINUTE
+```
+To use a more robust production grade monitoring solution the configuration needs to be changed.
+
+## How to Enable Prometheus (Example)
+The following example shows how to enable the Prometheus metric reporter:
+```yaml
+operatorConfiguration:
+ create: true
+ append: true
+ flink-conf.yaml: |+
+ metrics.reporter.prom.class: org.apache.flink.metrics.prometheus.PrometheusReporter
+ metrics.reporter.prom.port: 9999
+```
+Some metric reporters, including the Prometheus, needs a port to be exposed on the container. This can be achieved be defining a value for the otherwise empty `metrics.port` variable.
+Either in the `values.yaml` file:
+```yaml
+metrics:
+ port: 9999
+```
+or using the option `--set metrics.port=9999` in the command line.
+
+### Set up Prometheus locally
+The Prometheus Operator among other options provides an elegant, declarative way to specify how group of pods should be monitored using custom resources.
+
+To install the Prometheus operator via Helm run:
+
+```shell
+helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
+helm install prometheus prometheus-community/kube-prometheus-stack
+```
+The Grafana dashboard can be accessed through port-forwarding:
+```shell
+kubectl port-forward deployment/prometheus-grafana 3000
+```
+To enable the operator metrics in Prometheus create a `pod-monitor.yaml` file with the following content:
+```yaml
+apiVersion: monitoring.coreos.com/v1
+kind: PodMonitor
+metadata:
+ name: flink-operator
+ labels:
+ release: prometheus
+spec:
+ selector:
+ matchLabels:
+ app.kubernetes.io/name: flink-operator
+ podMetricsEndpoints:
+ - port: metrics
+```
+and apply it on your Kubernetes environment:
+```shell
+kubectl create -f pod-monitor.yaml
+```
+Once the custom resource is created in the Kubernetes environment the operator metrics are ready to explore [http://localhost:3000/explore](http://localhost:3000/explore).
+
+# Logging
+The Operator controls the logging behaviour for Flink applications and the Operator itself using configuration files mounted externally via ConfigMaps. [Configuration files](https://github.com/apache/flink-kubernetes-operator/tree/main/helm/flink-operator/conf) with default values are shipped in the Helm chart. It is recommended to review and adjust them if needed in the `values.yaml` file before deploying the Operator in production environments.
+
+To append/override the default log configuration properties for the Operator use:
+```yaml
+operatorConfiguration:
+ create: true
+ append: true
+ log4j2.properties: |+
+ rootLogger.level = DEBUG
+```
+
+To append/override the default log configuration properties for Flink applications use:
+```yaml
+flinkDefaultConfiguration:
+ create: true
+ append: true
+ log4j-console.properties: |+
+ rootLogger.level = DEBUG
+```
+To learn more about accessing the job logs or changing the log level dynamically check the corresponding [section](https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/resource-providers/native_kubernetes/#logging) of the core documentation.