You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by gy...@apache.org on 2022/07/19 11:44:06 UTC
[flink-kubernetes-operator] branch main updated: [doc] Doc cleanup and updates for upcoming 1.1.0
This is an automated email from the ASF dual-hosted git repository.
gyfora pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/flink-kubernetes-operator.git
The following commit(s) were added to refs/heads/main by this push:
new 1006c3c3 [doc] Doc cleanup and updates for upcoming 1.1.0
1006c3c3 is described below
commit 1006c3c35cb003f65eac085a1292a97c4e68b688
Author: Gyula Fora <g_...@apple.com>
AuthorDate: Tue Jul 19 11:02:23 2022 +0200
[doc] Doc cleanup and updates for upcoming 1.1.0
---
docs/content/docs/concepts/architecture.md | 51 +++++++-----------
docs/content/docs/concepts/overview.md | 63 +++++++++++++---------
.../content/docs/custom-resource/job-management.md | 32 ++++++++---
docs/content/docs/development/roadmap.md | 9 +---
docs/content/docs/operations/ingress.md | 2 +-
docs/content/docs/operations/listeners.md | 41 --------------
docs/content/docs/operations/metrics-logging.md | 57 ++++++++++----------
.../docs/operations/{validator.md => plugins.md} | 26 +++++++--
.../img/concepts/JM_deployment_state_machine.svg | 21 --------
docs/static/img/concepts/resource_lifecycle.svg | 22 ++++++++
10 files changed, 159 insertions(+), 165 deletions(-)
diff --git a/docs/content/docs/concepts/architecture.md b/docs/content/docs/concepts/architecture.md
index 1ca86312..962657dc 100644
--- a/docs/content/docs/concepts/architecture.md
+++ b/docs/content/docs/concepts/architecture.md
@@ -25,6 +25,7 @@ under the License.
-->
# Architecture
+
Flink Kubernetes Operator (Operator) acts as a control plane to manage the complete deployment lifecycle of Apache Flink applications. The Operator can be installed on a Kubernetes cluster using [Helm](https://helm.sh). In most production environments it is typically deployed in a designated namespace and controls Flink deployments in one or more managed namespaces. The custom resource definition (CRD) that describes the schema of a `FlinkDeployment` is a cluster wide resource. For a CRD [...]
{{< img src="/img/concepts/architecture.svg" alt="Flink Kubernetes Operator Architecture" >}}
@@ -35,44 +36,30 @@ The Operator follow the Kubernetes principles, notably the [control loop](https:
{{< img src="/img/concepts/control_loop.svg" alt="Control Loop" >}}
-Users can interact with the operator using the Kubernetes command-line tool, [kubectl](https://kubernetes.io/docs/tasks/tools/). The Operator continuously tracks cluster events relating to the `FlinkDeployment` custom resource. When the operator receives a new event, it will take action to adjust the Kubernetes cluster to the desired state as part of its reconciliation loop. The initial loop consists of the following high-level steps:
-1. User submits a `FlinkDeployment` custom resource(CR) using `kubectl`
-2. The operator launches the Flink cluster deployment and creates an ingress rule for UI access
-3. The `JobManager` creates `TaskManager` pods
-4. The `JobManager` submits the job
+Users can interact with the operator using the Kubernetes command-line tool, [kubectl](https://kubernetes.io/docs/tasks/tools/). The Operator continuously tracks cluster events relating to the `FlinkDeployment` and `FlinkSessionJob` custom resources. When the operator receives a new resource update, it will take action to adjust the Kubernetes cluster to the desired state as part of its reconciliation loop. The initial loop consists of the following high-level steps:
+
+1. User submits a `FlinkDeployment`/`FlinkSessionJob` custom resource(CR) using `kubectl`
+2. Operator observes the current status of the Flink resource (if previously deployed)
+3. Operator validates the submitted resource change
+4. Operator reconciles any required changes and executes upgrades
The CR can be (re)applied on the cluster any time. The Operator makes continuous adjustments to imitate the desired state until the current state becomes the desired state. All lifecycle management operations are realized using this very simple principle in the Operator.
The Operator is built with the [Java Operator SDK](https://github.com/java-operator-sdk/java-operator-sdk) and uses the [Native Kubernetes Integration](https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/resource-providers/native_kubernetes/) for launching Flink deployments and submitting jobs under the hood. The Java Operator SDK is a higher level framework and related tooling to support writing Kubernetes Operators in Java. Both the Java Operator SDK and Flink's native [...]
-## State Machine of JobManager Deployment
-The Operator manages the lifecycle of the JobManager Deployment. Its state machine is as follows:
-
-{{< img src="/img/concepts/JM_deployment_state_machine.svg" alt="State Machine of JobManager Deployment" >}}
-
-The possible transitions usually indicate that there are some underlying changes:
-
-1. `MISSING` -> `DEPLOYING`: A new JM deployment exists and is being created
-2. `DEPLOYING` -> `DEPLOYED_NOT_READY`: The JM deployment exists and passes checks of the availability of replicas and JM port connectivity. Now, it is waiting the REST service to be ready.
-3. `DEPLOYED_NOT_READY` -> `READY`: JM can serve requests.
-4. `READY` -> `READY`: JM works fine.
-5. `READY` -> `DEPLOYED_NOT_READY`: JM REST service becomes unavailable.
-6. `READY` -> `ERROR`: REST service is unavailable and JM deployment failed(e.g. in CrashLoopBackoff state).
-7. `READY` -> `MISSING`: JM deployment does not exist(e.g. deleted by kubectl or by `SUSPEND` action).
-8. `ERROR` -> `ERROR`: JM deployment failed.
-9. `DEPLOYING` -> `DEPLOYING`: The JM deployment exists and is still being created.
-10. `DEPLOYING` -> `ERROR`: JM deployment failed.
-11. `MISSING` -> `MISSING`: JM deployment does not exist.
-
-
-
-
-
-
-
-
-
+## Flink Resource Lifecycle
+The Operator manages the lifecycle of Flink resources. The following chart illustrates the different possible states and transitions:
+{{< img src="/img/concepts/resource_lifecycle.svg" alt="Flink Resource Lifecycle" >}}
+**We can distinguish the following states:**
+ - CREATED : The resource was created in Kubernetes but not yet handled by the operator
+ - SUSPENDED : The (job) resource has been suspended
+ - UPGRADING : The resource is suspended before upgrading to a new spec
+ - DEPLOYED : The resource is deployed/submitted to Kubernetes, but it's not yet considered to be stable and might be rolled back in the future
+ - STABLE : The resource deployment is considered to be stable and won't be rolled back
+ - ROLLING_BACK : The resource is being rolled back to the last stable spec
+ - ROLLED_BACK : The resource is deployed with the last stable spec
+ - FAILED : The job terminally failed
diff --git a/docs/content/docs/concepts/overview.md b/docs/content/docs/concepts/overview.md
index 442d9e73..7951ca07 100644
--- a/docs/content/docs/concepts/overview.md
+++ b/docs/content/docs/concepts/overview.md
@@ -29,56 +29,71 @@ Flink Kubernetes Operator acts as a control plane to manage the complete deploym
Flink Kubernetes Operator aims to capture the responsibilities of a human operator who is managing Flink deployments. Human operators have deep knowledge of how Flink deployments ought to behave, how to start clusters, how to deploy jobs, how to upgrade them and how to react if there are problems. The main goal of the operator is the automation of these activities, which cannot be achieved through the Flink native integration alone.
-# Features
-## Core
-- Fully-automated [Job Lifecycle Management](https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/custom-resource/job-management/)
+## Features
+### Core
+- Fully-automated [Job Lifecycle Management]({{< ref "docs/custom-resource/job-management" >}})
- Running, suspending and deleting applications
- Stateful and stateless application upgrades
- Triggering and managing savepoints
- Handling errors, rolling-back broken upgrades
- Multiple Flink version support: v1.13, v1.14, v1.15
-- [Deployment Modes](https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/custom-resource/overview/#application-deployments):
+- [Deployment Modes]({{< ref "docs/custom-resource/overview#application-deployments" >}}):
- Application cluster
- Session cluster
- Session job
- Built-in [High Availability](https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/ha/kubernetes_ha/)
-- Extensible validation framework
- - Webhook and Operator based validation
- - [Custom validators](https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/operations/validator/)
-- Advanced [Configuration](https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/operations/configuration/) management
+- Extensible framework
+ - [Custom validators]({{< ref "docs/operations/plugins#custom-flink-resource-validators" >}})
+ - [Custom resource listeners]({{< ref "docs/operations/plugins#custom-flink-resource-listeners" >}})
+- Advanced [Configuration]({{< ref "docs/operations/configuration" >}}) management
- Default configurations with dynamic updates
- Per job configuration
- Environment variables
-- POD augmentation via [Pod Templates](https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/custom-resource/pod-template/)
+- POD augmentation via [Pod Templates]({{< ref "docs/custom-resource/pod-template" >}})
- Native Kubernetes POD definitions
- Layering (Base/JobManager/TaskManager overrides)
-## Operations
-- Operator [Metrics](https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/operations/metrics-logging/#metrics)
+### Operations
+- Operator [Metrics]({{< ref "docs/operations/metrics-logging#metrics" >}})
- Utilizes the well-established [Flink Metric System](https://nightlies.apache.org/flink/flink-docs-master/docs/ops/metrics)
- Pluggable metrics reporters
-- Fully-customizable [Logging](https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/operations/metrics-logging/#logging)
+ - Detailed resources and kubernetes api access metrics
+- Fully-customizable [Logging]({{< ref "docs/operations/metrics-logging#logging" >}})
- Default log configuration
- Per job log configuration
- Sidecar based log forwarders
- Flink Web UI and REST Endpoint Access
- Fully supported Flink Native Kubernetes [service expose types](https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/resource-providers/native_kubernetes/#accessing-flinks-web-ui)
- - Dynamic [Ingress templates](https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/operations/ingress/)
-- [Helm based installation](https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/operations/helm/)
- - Automated [RBAC configuration](https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/operations/rbac/)
- - Advanced customization techniques
-- Up-to-date public repositories
- - Helm [https://downloads.apache.org/flink/flink-kubernetes-operator-1.0.0/](https://downloads.apache.org/flink/flink-kubernetes-operator-1.0.0)
- - Docker [ghcr.io/apache/flink-kubernetes-operator](http://ghcr.io/apache/flink-kubernetes-operator)
+ - Dynamic [Ingress templates]({{< ref "docs/operations/ingress" >}})
+- [Helm based installation]({{< ref "docs/operations/helm" >}})
+ - Automated [RBAC configuration]({{< ref "docs/operations/rbac" >}})
+ - Advanced customization techniques
+- Up-to-date public repositories
+ - GitHub Container Registry [ghcr.io/apache/flink-kubernetes-operator](http://ghcr.io/apache/flink-kubernetes-operator)
+ - DockerHub [https://hub.docker.com/r/apache/flink-kubernetes-operator](https://hub.docker.com/r/apache/flink-kubernetes-operator)
-# Known Issues & Limitations
+## Built-in Examples
-## JobManager High-availability
+The operator project comes with a wide variety of built in examples to show you how to use the operator functionality.
+The examples are maintained as part of the operator repo and can be found [here](https://github.com/apache/flink-kubernetes-operator/tree/main/examples).
+
+**What is covered:**
+
+ - Application, Session and SessionJob submission
+ - Checkpointing and HA configuration
+ - Java, SQL and Python Flink jobs
+ - Ingress, logging and metrics configuration
+ - Advanced operator deployment techniques using Kustomize
+ - And some more...
+
+## Known Issues & Limitations
+
+### JobManager High-availability
The Operator leverages [Kubernetes HA Services](https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/ha/kubernetes_ha/) for providing High-availability for Flink jobs. The HA solution can benefit form using additional [Standby replicas](https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/ha/overview/), it will result in a faster recovery time, but Flink jobs will still restart when the Leader JobManager goes down.
-## Standalone Kubernetes Support
-The Operator does not support [Standalone Kubernetes](https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/resource-providers/standalone/kubernetes/) deployments yet
+### Standalone Kubernetes Support
+The Operator does not support [Standalone Kubernetes](https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/resource-providers/standalone/kubernetes/) deployments yet. It is expected to be part of the `1.2.0` release.
-## JobResultStore Resource Leak
+### JobResultStore Resource Leak
To mitigate the impact of [FLINK-27569](https://issues.apache.org/jira/browse/FLINK-27569) the operator introduced a workaround [FLINK-27573](https://issues.apache.org/jira/browse/FLINK-27573) by setting `job-result-store.delete-on-commit=false` and a unique value for `job-result-store.storage-path` for every cluster launch. The storage path for older runs must be cleaned up manually, keeping the latest directory always:
```shell
ls -lth /tmp/flink/ha/job-result-store/basic-checkpoint-ha-example/
diff --git a/docs/content/docs/custom-resource/job-management.md b/docs/content/docs/custom-resource/job-management.md
index e5a2a7f2..c81c3f9a 100644
--- a/docs/content/docs/custom-resource/job-management.md
+++ b/docs/content/docs/custom-resource/job-management.md
@@ -32,7 +32,7 @@ What is covered:
- Running, suspending and deleting applications
- Stateful and stateless application upgrades
- Triggering and managing savepoints
- - Handling errors, rolling-back broken upgrades
+ - Handling errors, rolling-back broken upgrades
The behaviour is always controlled by the respective configuration fields of the `JobSpec` object as introduced in the [FlinkDeployment/FlinkSessionJob overview]({{< ref "docs/custom-resource/overview" >}}).
@@ -169,9 +169,13 @@ Restarts work exactly the same way as other application upgrades and follow the
## Savepoint management
-Savepoints are triggered automatically by the system during savepoint upgrades as we have seen in the previous questions.
+Savepoints are triggered automatically by the system during the upgrade process as we have seen in the previous sections.
-Savepoints can also be triggered manually by defining a new (different/random) value to the variable `savepointTriggerNonce` in the job specification:
+For backup, job forking and other purposes savepoints can be triggered manually or periodically by the operator, however generally speaking these will not be used during upgrades and are not required for the correct operation.
+
+### Manual Savepoint Triggering
+
+Users can trigger savepoints manually by defining a new (different/random) value to the variable `savepointTriggerNonce` in the job specification:
```yaml
job:
@@ -181,9 +185,21 @@ Savepoints can also be triggered manually by defining a new (different/random) v
Changing the nonce value will trigger a new savepoint. Information about pending and last savepoint is stored in the resource status.
+### Periodic Savepoint Triggering
+
+The operator also supports periodic savepoint triggering through the following config option which can be configured on a per job level:
+
+```yaml
+ flinkConfiguration:
+ ...
+ kubernetes.operator.periodic.savepoint.interval: 6h
+```
+
+There is no guarantee on the timely execution of the periodic savepoints as they might be delayed by unhealthy job status or other interfering user operation.
+
### Savepoint History
-The operator automatically keeps track of the savepoint history triggered by upgrade or manual savepoint operations.
+The operator automatically keeps track of the savepoint history triggered by upgrade or manual savepoint operations.
This is necessary so cleanup can be performed by the operator for old savepoints.
Users can control the cleanup behaviour by specifying a maximum age and maximum count for the savepoints in the history.
@@ -194,7 +210,7 @@ kubernetes.operator.savepoint.history.max.count: 5
```
{{< hint info >}}
-Savepoint cleanup happens lazily and only when the application is running.
+Savepoint cleanup happens lazily and only when the application is running.
It is therefore very likely that savepoints live beyond the max age configuration.
{{< /hint >}}
@@ -203,7 +219,7 @@ It is therefore very likely that savepoints live beyond the max age configuratio
When Kubernetes HA is enabled, the operator can recover the Flink cluster deployments in cases when it was accidentally deleted
by the user or some external process. Deployment recovery can be turned off in the configuration by setting `kubernetes.operator.jm-deployment-recovery.enabled` to `false`, however it is recommended to keep this setting on the default `true` value.
-This is not something that would usually happen during normal operation and can also indicate a deeper problem,
+This is not something that would usually happen during normal operation and can also indicate a deeper problem,
therefore an Error event is also triggered by the system when it detects a missing deployment.
One scenario which could lead to a loss of jobmanager deployment in Flink versions before 1.15 is the job entering a terminal state:
@@ -219,7 +235,7 @@ Please check the [manual Recovery section](#manual-recovery) to understand how t
## Application upgrade rollbacks (Experimental)
The operator supports upgrade rollbacks as an experimental feature.
-The rollback feature works based on the concept of stable deployments specs.
+The rollback feature works based on the concept of stable deployments specs.
When an application is upgraded, the new spec is initially considered unstable. Once the operator successfully observes the new job in a healthy running state, the spec is marked as stable.
If a new upgrade is not marked stable within a certain configurable time period (`kubernetes.operator.deployment.readiness.timeout`) then a rollback operation will be performed, rolling back to the last stable spec.
@@ -257,7 +273,7 @@ Fortunately almost any issue can be recovered by the user manually by using the
3. Check that you have the current savepoint, and that your `FlinkDeployment` is deleted completely
4. Modify your `FlinkDeployment` JobSpec and set the `initialSavepointPath` to your last checkpoint location
5. Recreate the deployment
-
+
These steps ensure that the operator will start completely fresh from the user defined savepoint path and can hopefully fully recover.
Keep an eye on your job to see what could have cause the problem in the first place.
diff --git a/docs/content/docs/development/roadmap.md b/docs/content/docs/development/roadmap.md
index c12d1cc9..97545385 100644
--- a/docs/content/docs/development/roadmap.md
+++ b/docs/content/docs/development/roadmap.md
@@ -31,11 +31,6 @@ It's not a comprehensive list and might be slightly outdated at any given time.
## What’s Next?
-Our intention is to advance further on the [Operator Maturity Model](https://operatorframework.io/operator-capabilities/)
-by adding more dynamic/automatic features
-
- Standalone deployment mode support [FLIP-225](https://cwiki.apache.org/confluence/display/FLINK/FLIP-225%3A+Implement+standalone+mode+support+in+the+kubernetes+operator)
-- Auto-scaling using Horizontal Pod AutoScaler
-- Dynamic change of watched namespaces
-- Pluggable Status and Event reporters (Making it easier to integrate with proprietary control planes)
-- Improved Savepoint management & periodic triggering
+- Improved scaling and autoscaling support
+- Improved rollback mechanism and stability conditions
diff --git a/docs/content/docs/operations/ingress.md b/docs/content/docs/operations/ingress.md
index 2b981d7a..03df1964 100644
--- a/docs/content/docs/operations/ingress.md
+++ b/docs/content/docs/operations/ingress.md
@@ -27,7 +27,7 @@ under the License.
# Accessing Flink’s Web UI
The Flink Kubernetes Operator, by default, does not change the way the native kubernetes integration [exposes](https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/resource-providers/native_kubernetes/#accessing-flinks-web-ui) the Flink Web UI.
-# Ingress
+## Ingress
Beyond the native options, the Operator also supports creating [Ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/) entries for external UI access.
Ingress generation can be turned on by defining the `ingress` field in the `FlinkDeployment`:
```yaml
diff --git a/docs/content/docs/operations/listeners.md b/docs/content/docs/operations/listeners.md
deleted file mode 100644
index 755a725d..00000000
--- a/docs/content/docs/operations/listeners.md
+++ /dev/null
@@ -1,41 +0,0 @@
----
-title: "Custom Listeners"
-weight: 5
-type: docs
-aliases:
-- /operations/listeners.html
----
-<!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
--->
-
-# Custom Flink Resource Listeners
-
-The Flink Kubernetes Operator allows users to listen to events and status updates triggered for the Flink Resources managed by the operator.
-This feature enables tighter integration with the user's own data platform.
-
-By implementing the `FlinkResourceListener` interface users can listen to both events and status updates per resource type (`FlinkDeployment` / `FlinkSessionJob`). These methods will be called after the respective events have been triggered by the system.
-Using the context provided on each listener method users can also get access to the related Flink resource and the `KubernetesClient` itself in order to trigger any further events etc on demand.
-
-Similar to [custom validator implementations]({{< ref "docs/operations/validator" >}}) resource listeners are loaded via the Flink [Plugins](https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/filesystems/plugins) mechanism.
-
-In order to enable your custom `FlinkResourceListener` you need to:
-
- 1. Implement the interface
- 2. Add your listener class to `org.apache.flink.kubernetes.operator.listener.FlinkResourceListener` in `META-INF/services`
- 3. Package your JAR and add it to the plugins directory of your operator image (`/opt/flink/plugins`)
diff --git a/docs/content/docs/operations/metrics-logging.md b/docs/content/docs/operations/metrics-logging.md
index 2467e31e..186cd419 100644
--- a/docs/content/docs/operations/metrics-logging.md
+++ b/docs/content/docs/operations/metrics-logging.md
@@ -24,33 +24,28 @@ specific language governing permissions and limitations
under the License.
-->
-# Metrics
+# Metrics and Logging
-The Flink Kubernetes Operator (Operator) extends the [Flink Metric System](https://nightlies.apache.org/flink/flink-docs-master/docs/ops/metrics/) that allows gathering and exposing metrics to centralized monitoring solutions.
+## Metrics
-## Flink Resource Metrics
-The Operator gathers aggregates metrics about managed resources.
+The Flink Kubernetes Operator (Operator) extends the [Flink Metric System](https://nightlies.apache.org/flink/flink-docs-master/docs/ops/metrics/) that allows gathering and exposing metrics to centralized monitoring solutions.
+
+Different operator metrics can be turned on/off individually using the configuration. For details check the [metrics config reference]({{< ref "docs/operations/configuration#system-metrics-configuration" >}}).
-| Scope | Metrics | Description | Type |
-|------------------|-------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------|
-| Namespace | FlinkDeployment/FlinkSessionJob.Count | Number of managed FlinkDeployment/SessionJob instances per namespace | Gauge |
-| Namespace | FlinkDeployment.JmDeploymentStatus.<Status>.Count | Number of managed FlinkDeployment resources per <Status> per namespace. <Status> can take values from: READY, DEPLOYED_NOT_READY, DEPLOYING, MISSING, ERROR | Gauge |
-| Namespace | FlinkDeployment/FlinkSessionJob.Lifecycle.State.<State>.Count | Number of managed resources currently in state <State> per namespace. <State> can take values from: CREATED, SUSPENDED, UPGRADING, DEPLOYED, STABLE, ROLLING_BACK, ROLLED_BACK, FAILED | Gauge |
-| System/Namespace | FlinkDeployment/FlinkSessionJob.Lifecycle.State.<State>.TimeSeconds | Time spent in state <State$gt for a given resource. <State> can take values from: CREATED, SUSPENDED, UPGRADING, DEPLOYED, STABLE, ROLLING_BACK, ROLLED_BACK, FAILED | Histogram |
-| System/Namespace | FlinkDeployment/FlinkSessionJob.Lifecycle.Transition.<Transition>.TimeSeconds | Time statistics for selected lifecycle state transitions. <Transition> can take values from: Resume, Upgrade, Suspend, Stabilization, Rollback, Submission | Histogram |
+### Flink Resource Metrics
+The Operator gathers aggregates metrics about managed resources.
-### Lifecycle metrics
+| Scope | Metrics | Description | Type |
+|--------------------|-------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------|
+| Namespace | FlinkDeployment/FlinkSessionJob.Count | Number of managed resources per namespace | Gauge |
+| Namespace | FlinkDeployment.JmDeploymentStatus.<Status>.Count | Number of managed FlinkDeployment resources per <Status> per namespace. <Status> can take values from: READY, DEPLOYED_NOT_READY, DEPLOYING, MISSING, ERROR | Gauge |
+| Namespace | FlinkDeployment/FlinkSessionJob.Lifecycle.State.<State>.Count | Number of managed resources currently in state <State> per namespace. <State> can take values from: CREATED, SUSPENDED, UPGRADING, DEPLOYED, STABLE, ROLLING_BACK, ROLLED_BACK, FAILED | Gauge |
+| System/Namespace | FlinkDeployment/FlinkSessionJob.Lifecycle.State.<State>.TimeSeconds | Time spent in state <State$gt for a given resource. <State> can take values from: CREATED, SUSPENDED, UPGRADING, DEPLOYED, STABLE, ROLLING_BACK, ROLLED_BACK, FAILED | Histogram |
+| System/Namespace | FlinkDeployment/FlinkSessionJob.Lifecycle.Transition.<Transition>.TimeSeconds | Time statistics for selected lifecycle state transitions. <Transition> can take values from: Resume, Upgrade, Suspend, Stabilization, Rollback, Submission | Histogram |
-Based on the resource status the operator tracks the following resource lifecycle states:
+#### Lifecycle metrics
- - CREATED : The resource was created in Kubernetes but not yet handled by the operator
- - SUSPENDED : The (job) resource has been suspended
- - UPGRADING : The resource is suspended before upgrading to a new spec
- - DEPLOYED : The resource is deployed/submitted to Kubernetes, but it's not yet considered to be stable and might be rolled back in the future
- - STABLE : The resource deployment is considered to be stable and won't be rolled back
- - ROLLING_BACK : The resource is being rolled back to the last stable spec
- - ROLLED_BACK : The resource is deployed with the last stable spec
- - FAILED : The job terminally failed
+Based on the resource status the operator monitors [resource lifecycle states]({{< ref "docs/concepts/architecture#flink-resource-lifecycle" >}}).
The number of resources and time spend in each of these states at any given time is tracked by the `Lifecycle.<STATE>.Count` and `Lifecycle.<STATE>.TimeSeconds` metrics.
@@ -63,9 +58,9 @@ In addition to the simple counts we further track a few selected state transitio
- Rollback : Time from deployed to rolled_back state if the resource was rolled back
- Submission: Flink resource submission time
-## Kubernetes Client Metrics
+### Kubernetes Client Metrics
-The Operator gathers various metrics related to Kubernetes API server access. The Kubernetes client metrics can be enabled by the configuration `kubernetes.operator.kubernetes.client.metrics.enabled` (default: `true`).
+The Operator gathers various metrics related to Kubernetes API server access.
| Scope | Metrics | Description | Type |
|--------|----------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------|
@@ -79,9 +74,15 @@ The Operator gathers various metrics related to Kubernetes API server access. Th
| System | KubeClient.HttpResponse.NumPerSecond | Number of HTTP responses received from the Kubernetes API Server per second | Meter |
| System | KubeClient.HttpResponse.TimeNanos | Latency statistics obtained from the HTTP responses received from the Kubernetes API Server | Histogram |
-## System Metrics
+### JVM Metrics
+
The Operator gathers metrics about the JVM process and exposes it similarly to core Flink [System metrics](https://nightlies.apache.org/flink/flink-docs-master/docs/ops/metrics/#system-metrics). The list of metrics are not repeated in this document.
+### JOSDK Metrics
+
+The Flink operator also forwards metrics created by the Java Operator SDK framework itself under the `JOSDK` metric name prefix.
+Some of these metrics are on system, namespace and resource level.
+
## Metric Reporters
The well known [Metric Reporters](https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/metric_reporters) are shipped in the operator image and are ready to use.
@@ -90,7 +91,7 @@ In order to specify metrics configuration for the operator, simply prefix them w
Let's look at a few examples.
-## Slf4j
+### Slf4j
The default metrics reporter in the operator is Slf4j. It does not require any external monitoring systems, and it is enabled in the `values.yaml` file by default, mainly for demonstrating purposes.
```yaml
defaultConfiguration:
@@ -102,7 +103,7 @@ defaultConfiguration:
```
To use a more robust production grade monitoring solution the configuration needs to be changed.
-## How to Enable Prometheus (Example)
+### How to Enable Prometheus (Example)
The following example shows how to enable the Prometheus metric reporter:
```yaml
defaultConfiguration:
@@ -120,7 +121,7 @@ metrics:
```
or using the option `--set metrics.port=9999` in the command line.
-### Set up Prometheus locally
+#### Set up Prometheus locally
The Prometheus Operator among other options provides an elegant, declarative way to specify how group of pods should be monitored using custom resources.
To install the Prometheus operator via Helm run:
@@ -154,7 +155,7 @@ kubectl create -f pod-monitor.yaml
```
Once the custom resource is created in the Kubernetes environment the operator metrics are ready to explore [http://localhost:3000/explore](http://localhost:3000/explore).
-# Logging
+## Logging
The Operator controls the logging behaviour for Flink applications and the Operator itself using configuration files mounted externally via ConfigMaps. [Configuration files](https://github.com/apache/flink-kubernetes-operator/tree/main/helm/flink-kubernetes-operator/conf) with default values are shipped in the Helm chart. It is recommended to review and adjust them if needed in the `values.yaml` file before deploying the Operator in production environments.
To append/override the default log configuration properties for the operator and Flink deployments define the `log4j-operator.properties` and `log4j-console.properties` keys respectively:
diff --git a/docs/content/docs/operations/validator.md b/docs/content/docs/operations/plugins.md
similarity index 72%
rename from docs/content/docs/operations/validator.md
rename to docs/content/docs/operations/plugins.md
index 316965e8..855b2deb 100644
--- a/docs/content/docs/operations/validator.md
+++ b/docs/content/docs/operations/plugins.md
@@ -1,9 +1,9 @@
---
-title: "Custom Resource Validators"
+title: "Custom Operator Plugins"
weight: 5
type: docs
aliases:
-- /operations/validator.html
+- /operations/plugins.html
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
@@ -24,7 +24,11 @@ specific language governing permissions and limitations
under the License.
-->
-# Custom `FlinkResourceValidator` implementation
+# Custom Operator Plugins
+
+The operator provides a customizable platform for Flink resource management. Users can develop plugins to tailor the operator behaviour to their own needs.
+
+## Custom Flink Resource Validators
`FlinkResourceValidator`, an interface for validating the resources of `FlinkDeployment` and `FlinkSessionJob`, is a pluggable component based on the [Plugins](https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/filesystems/plugins) mechanism. During development, we can customize the implementation of `FlinkResourceValidator` and make sure to retain the service definition in `META-INF/services`.
The following steps demonstrate how to develop and use a custom validator.
@@ -89,3 +93,19 @@ The following steps demonstrate how to develop and use a custom validator.
```text
2022-05-04 14:01:46,551 o.a.f.k.o.u.FlinkUtils [INFO ] Discovered resource validator from plugin directory[/opt/flink/plugins]: org.apache.flink.kubernetes.operator.validation.CustomValidator.
```
+
+## Custom Flink Resource Listeners
+
+The Flink Kubernetes Operator allows users to listen to events and status updates triggered for the Flink Resources managed by the operator.
+This feature enables tighter integration with the user's own data platform.
+
+By implementing the `FlinkResourceListener` interface users can listen to both events and status updates per resource type (`FlinkDeployment` / `FlinkSessionJob`). These methods will be called after the respective events have been triggered by the system.
+Using the context provided on each listener method users can also get access to the related Flink resource and the `KubernetesClient` itself in order to trigger any further events etc on demand.
+
+Similar to custom validator implementations, resource listeners are loaded via the Flink [Plugins](https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/filesystems/plugins) mechanism.
+
+In order to enable your custom `FlinkResourceListener` you need to:
+
+ 1. Implement the interface
+ 2. Add your listener class to `org.apache.flink.kubernetes.operator.listener.FlinkResourceListener` in `META-INF/services`
+ 3. Package your JAR and add it to the plugins directory of your operator image (`/opt/flink/plugins`)
diff --git a/docs/static/img/concepts/JM_deployment_state_machine.svg b/docs/static/img/concepts/JM_deployment_state_machine.svg
deleted file mode 100644
index bbfd1c35..00000000
--- a/docs/static/img/concepts/JM_deployment_state_machine.svg
+++ /dev/null
@@ -1,21 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
--->
-<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
-<svg version="1.1" viewBox="0.0 0.0 960.0 540.0" fill="none" stroke="none" stroke-linecap="square" stroke-miterlimit="10" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns="http://www.w3.org/2000/svg"><clipPath id="p.0"><path d="m0 0l960.0 0l0 540.0l-960.0 0l0 -540.0z" clip-rule="nonzero"/></clipPath><g clip-path="url(#p.0)"><path fill="#ffffff" d="m0 0l960.0 0l0 540.0l-960.0 0z" fill-rule="evenodd"/><path fill="#cfe2f3" d="m516.15485 97.97128l0 0c0 -4.3632812 3.5371704 -7.9004135 7.90045 [...]
\ No newline at end of file
diff --git a/docs/static/img/concepts/resource_lifecycle.svg b/docs/static/img/concepts/resource_lifecycle.svg
new file mode 100644
index 00000000..faaada65
--- /dev/null
+++ b/docs/static/img/concepts/resource_lifecycle.svg
@@ -0,0 +1,22 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+<!-- Do not edit this file with editors other than diagrams.net -->
+<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
+<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" width="381px" height="451px" viewBox="-0.5 -0.5 381 451" content="<mxfile host="app.diagrams.net" modified="2022-07-19T07:39:03.441Z" agent="5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36" etag="kRJMNE4bMRhcRguxJ2XL" version="20.1.1" type="google"><diagram id=&qu [...]
\ No newline at end of file