You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by po...@apache.org on 2020/11/15 02:51:43 UTC
[airflow] branch v1-10-test updated (fe515fb -> eeb9134)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a change to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git.
from fe515fb Install cattr on Python 3.7 - Fix docs build on RTD (#12045)
new a1077dc Fix gitSync user in the helm Chart (#11127)
new 4d99508 Adds Kubernetes Service Account for the webserver (#11131)
new 96f2b6b Enables Kerberos sidecar support (#11130)
new 3fbf6df Kubernetes executor can adopt tasks from other schedulers (#10996)
new ae681ba Single/Multi-Namespace mode for helm chart (#11034)
new 4c2cb60 Add CeleryKubernetesExecutor to helm chart (#11288)
new 5ce3da6 Add capability of adding service account annotations to Helm Chart (#11387)
new 6598002 Mutual SSL added in PGBouncer configuration in the Chart (#11384)
new f86d534 Adds missing schema for kerberos sidecar configuration (#11413)
new f3d5fe2 Mount volumes and volumemounts into scheduler and workers (#11426)
new 2458461 Allow multiple schedulers in helm chart (#11330)
new e85cc77 Add missing values entries to Parameters in chart/README.md (#11477)
new 3f74b17 Create job for airflow migrations (#11533)
new 5ee07b9 Improvements for pod template file with git sync container (#11511)
new 788f1c0 Pod template file uses custom custom env variable (#11480)
new 6d8ec84 fix pod launcher rolebinding in helm chart (#11675)
new 0df5e99 Consistent use images in Helm Chart (#11701)
new f9ba264 Remove unused value in Helm Chart - podMutation (#11703)
new 6a12758 Validate airflow chart values.yaml & values.schema.json (#11990)
new a01bd32 Add Flower Authentication to Helm Chart (#11836)
new 7a72fb7 fix helm chart worker deployment without kerberos (#11681)
new 21f792f All k8s object must comply with JSON Schema (#12003)
new ffcbf6f fix helm scheduler deployment / scheduler logs (#11685)
new a70cdd1 Add Kubernetes cleanup-pods CLI command for Helm Chart (#11802)
new a31ccb7 Use PyUpgrade to use Python 3.6 features (#11447)
new 265d155 Enable Black - Python Auto Formmatter (#9550)
new f750045 Fix default values for Helm Chart (#12153)
new 44b5077 Move metrics configuration to new section - metrics (#12165)
new 13442db Fix spelling (#12253)
new fda5f7ad Fix indentation for affinities in helm chart (#12288)
new 1bd745e Mount airflow.cfg to pod_template_file (#12311)
new eeb9134 Fix helm unit test for pod_template_file (#12345)
The 32 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.pre-commit-config.yaml | 4 +-
breeze | 2 +-
chart/README.md | 51 +-
chart/files/pod-template-file.kubernetes-helm-yaml | 38 +-
chart/templates/NOTES.txt | 16 +-
chart/templates/_helpers.yaml | 57 +-
chart/templates/cleanup/cleanup-cronjob.yaml | 2 +-
chart/templates/configmap.yaml | 6 +-
chart/templates/create-user-job.yaml | 1 +
chart/templates/flower/flower-deployment.yaml | 31 +-
.../flower-ingress.yaml} | 51 +-
chart/templates/flower/flower-service.yaml | 2 +-
...ate-user-job.yaml => migrate-database-job.yaml} | 41 +-
.../templates/pgbouncer/pgbouncer-deployment.yaml | 27 +
chart/templates/rbac/pod-launcher-role.yaml | 8 +
chart/templates/rbac/pod-launcher-rolebinding.yaml | 15 +-
chart/templates/redis/redis-networkpolicy.yaml | 2 +-
chart/templates/redis/redis-service.yaml | 2 +-
chart/templates/redis/redis-statefulset.yaml | 12 +-
.../templates/scheduler/scheduler-deployment.yaml | 19 +-
.../scheduler/scheduler-serviceaccount.yaml | 12 +-
...uncer-config-secret.yaml => flower-secret.yaml} | 9 +-
...ret.yaml => pgbouncer-certificates-secret.yaml} | 16 +-
chart/templates/secrets/redis-secrets.yaml | 3 +
chart/templates/statsd/statsd-deployment.yaml | 6 +-
.../templates/webserver/webserver-deployment.yaml | 1 +
chart/templates/webserver/webserver-ingress.yaml | 63 +
chart/templates/webserver/webserver-service.yaml | 4 +
.../webserver-serviceaccount.yaml} | 20 +-
chart/templates/workers/worker-deployment.yaml | 72 +-
chart/templates/workers/worker-kedaautoscaler.yaml | 2 +-
chart/templates/workers/worker-networkpolicy.yaml | 2 +-
chart/templates/workers/worker-service.yaml | 2 +-
chart/templates/workers/worker-serviceaccount.yaml | 10 +-
chart/tests/helm_template_generator.py | 37 +
chart/tests/test_basic_helm_chart.py | 57 +-
chart/tests/test_celery_kubernetes_executor.py | 11 +-
.../test_celery_kubernetes_pod_launcher_role.py | 1 +
.../tests/test_chart_quality.py | 39 +-
chart/tests/test_dags_persistent_volume_claim.py | 1 +
chart/tests/test_flower_authorization.py | 69 ++
chart/tests/test_git_sync_scheduler.py | 1 +
chart/tests/test_git_sync_webserver.py | 5 +-
chart/tests/test_git_sync_worker.py | 13 +-
...st_migrate_database_job.py => test_kerberos.py} | 22 +-
chart/tests/test_migrate_database_job.py | 1 +
chart/tests/test_pod_template_file.py | 75 +-
chart/tests/test_scheduler.py | 1 +
chart/tests/test_worker.py | 1 +
chart/values.schema.json | 1263 ++++++++++++++++++++
chart/values.yaml | 266 +++--
51 files changed, 2195 insertions(+), 277 deletions(-)
copy chart/templates/{statsd/statsd-service.yaml => flower/flower-ingress.yaml} (55%)
copy chart/templates/{create-user-job.yaml => migrate-database-job.yaml} (69%)
copy chart/templates/secrets/{pgbouncer-config-secret.yaml => flower-secret.yaml} (81%)
copy chart/templates/secrets/{pgbouncer-stats-secret.yaml => pgbouncer-certificates-secret.yaml} (69%)
create mode 100644 chart/templates/webserver/webserver-ingress.yaml
copy chart/templates/{scheduler/scheduler-serviceaccount.yaml => webserver/webserver-serviceaccount.yaml} (74%)
copy airflow/utils/configuration.py => chart/tests/test_chart_quality.py (52%)
create mode 100644 chart/tests/test_flower_authorization.py
copy chart/tests/{test_migrate_database_job.py => test_kerberos.py} (61%)
create mode 100644 chart/values.schema.json
[airflow] 05/32: Single/Multi-Namespace mode for helm chart (#11034)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit ae681ba4b0c2c83c4d4533f500b539ee64ba0af2
Author: Daniel Imberman <da...@gmail.com>
AuthorDate: Mon Oct 5 10:38:52 2020 -0700
Single/Multi-Namespace mode for helm chart (#11034)
* Multi-Namespace mode for helm chart
Users should not REQUIRE a ClusterRole/ClusterRolebinding
to run airflow via helm. This change will allow "single" and "multi"
namespace modes so users can add airflow to managed kubernetes clusters
* add namespace to role
* add rolebinding too
* add docs
* add values.schema.json change
(cherry picked from commit 93475e9f4dc92d5cb5efea3310e7ddc468c0b991)
---
chart/README.md | 2 +-
chart/templates/rbac/pod-launcher-role.yaml | 7 +
chart/templates/rbac/pod-launcher-rolebinding.yaml | 7 +
chart/values.schema.json | 1059 ++++++++++++++++++++
chart/values.yaml | 145 ++-
5 files changed, 1169 insertions(+), 51 deletions(-)
diff --git a/chart/README.md b/chart/README.md
index 8372bb4..ef1caa3 100644
--- a/chart/README.md
+++ b/chart/README.md
@@ -206,7 +206,7 @@ The following tables lists the configurable parameters of the Airflow chart and
| `webserver.defaultUser` | Optional default airflow user information | `{}` |
| `dags.persistence.*` | Dag persistence configuration | Please refer to `values.yaml` |
| `dags.gitSync.*` | Git sync configuration | Please refer to `values.yaml` |
-
+| `multiNamespaceMode` | Whether the KubernetesExecutor can launch pods in multiple namespaces | `False` |
Specify each parameter using the `--set key=value[,key=value]` argument to `helm install`. For example,
diff --git a/chart/templates/rbac/pod-launcher-role.yaml b/chart/templates/rbac/pod-launcher-role.yaml
index 7e10122..dc56c92 100644
--- a/chart/templates/rbac/pod-launcher-role.yaml
+++ b/chart/templates/rbac/pod-launcher-role.yaml
@@ -19,10 +19,17 @@
## Airflow Pod Launcher Role
#################################
{{- if and .Values.rbacEnabled .Values.allowPodLaunching }}
+{{- if .Values.multiNamespaceMode }}
kind: ClusterRole
+{{- else }}
+kind: Role
+{{- end }}
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: {{ .Release.Name }}-pod-launcher-role
+{{- if not .Values.multiNamespaceMode }}
+ namespace: {{ .Release.Namespace }}
+{{- end }}
labels:
tier: airflow
release: {{ .Release.Name }}
diff --git a/chart/templates/rbac/pod-launcher-rolebinding.yaml b/chart/templates/rbac/pod-launcher-rolebinding.yaml
index 4dba494..6a1a5be 100644
--- a/chart/templates/rbac/pod-launcher-rolebinding.yaml
+++ b/chart/templates/rbac/pod-launcher-rolebinding.yaml
@@ -21,9 +21,16 @@
{{- if and .Values.rbacEnabled .Values.allowPodLaunching }}
{{- $grantScheduler := or (eq .Values.executor "LocalExecutor") (eq .Values.executor "SequentialExecutor") (eq .Values.executor "KubernetesExecutor") }}
{{- $grantWorker := or (eq .Values.executor "CeleryExecutor") (eq .Values.executor "KubernetesExecutor") }}
+{{- if .Values.multiNamespaceMode }}
kind: ClusterRoleBinding
+{{- else }}
+kind: RoleBinding
+{{- end }}
apiVersion: rbac.authorization.k8s.io/v1
metadata:
+{{- if not .Values.multiNamespaceMode }}
+ namespace: {{ .Release.Namespace }}
+{{- end }}
name: {{ .Release.Name }}-pod-launcher-rolebinding
labels:
tier: airflow
diff --git a/chart/values.schema.json b/chart/values.schema.json
new file mode 100644
index 0000000..9776116
--- /dev/null
+++ b/chart/values.schema.json
@@ -0,0 +1,1059 @@
+{
+ "$schema": "http://json-schema.org/draft-07/schema",
+ "description": "Default values for airflow. Declare variables to be passed into your templates.",
+ "type": "object",
+ "properties": {
+ "uid": {
+ "description": "User of airflow user.",
+ "type": "integer"
+ },
+ "gid": {
+ "description": "Group of airflow user.",
+ "type": "integer"
+ },
+ "airflowHome": {
+ "description": "Airflow home directory. Used for mount paths.",
+ "type": "string"
+ },
+ "defaultAirflowRepository": {
+ "description": "Default airflow repository. Overrides all the specific images below.",
+ "type": "string"
+ },
+ "defaultAirflowTag": {
+ "description": "Default airflow tag to deploy.",
+ "type": "string"
+ },
+ "multi_namespaceMode": {
+ "description": "Whether the KubernetesExecutor can launch workers in multiple namespaces",
+ "type": "boolean"
+ },
+ "nodeSelector": {
+ "description": "Select certain nodes for airflow pods.",
+ "type": "object",
+ "additionalProperties": {
+ "type": "string"
+ }
+ },
+ "affinity": {
+ "description": "Select certain nodes for airflow pods.",
+ "type": "object"
+ },
+ "tolerations": {
+ "description": "Select certain nodes for airflow pods.",
+ "type": "array"
+ },
+ "labels": {
+ "description": "Add common labels to all objects and pods defined in this chart.",
+ "type": "object",
+ "additionalProperties": {
+ "type": "string"
+ }
+ },
+ "ingress": {
+ "description": "Ingress configuration.",
+ "type": "object",
+ "properties": {
+ "enabled": {
+ "description": "Enable ingress resource.",
+ "type": "boolean"
+ },
+ "web": {
+ "description": "Configuration for the Ingress of the web Service.",
+ "type": "object",
+ "properties": {
+ "annotations": {
+ "description": "Annotations for the web Ingress.",
+ "type": "object"
+ },
+ "path": {
+ "description": "The path for the web Ingress.",
+ "type": "string"
+ },
+ "host": {
+ "description": "The hostname for the web Ingress.",
+ "type": "string"
+ },
+ "tls": {
+ "description": "Configuration for web Ingress TLS.",
+ "type": "object",
+ "properties": {
+ "enabled": {
+ "description": "Enable TLS termination for the web Ingress.",
+ "type": "boolean"
+ },
+ "secretName": {
+ "description": "The name of a pre-created Secret containing a TLS private key and certificate.",
+ "type": "string"
+ }
+ }
+ },
+ "precedingPaths": {
+ "description": "HTTP paths to add to the web Ingress before the default path.",
+ "type": "array"
+ },
+ "succeedingPaths": {
+ "description": "HTTP paths to add to the web Ingress after the default path.",
+ "type": "array"
+ }
+ }
+ },
+ "flower": {
+ "description": "Configuration for the Ingress of the flower Service.",
+ "type": "object",
+ "properties": {
+ "annotations": {
+ "description": "Annotations for the flower Ingress.",
+ "type": "object"
+ },
+ "path": {
+ "description": "The path for the flower Ingress.",
+ "type": "string"
+ },
+ "host": {
+ "description": "The hostname for the flower Ingress.",
+ "type": "string"
+ },
+ "tls": {
+ "description": "Configuration for flower Ingress TLS.",
+ "type": "object",
+ "properties": {
+ "enabled": {
+ "description": "Enable TLS termination for the flower Ingress.",
+ "type": "boolean"
+ },
+ "secretName": {
+ "description": "The name of a pre-created Secret containing a TLS private key and certificate.",
+ "type": "string"
+ }
+ }
+ },
+ "precedingPaths": {
+ "description": "HTTP paths to add to the flower Ingress before the default path.",
+ "type": "array"
+ },
+ "succeedingPaths": {
+ "description": "HTTP paths to add to the flower Ingress after the default path.",
+ "type": "array"
+ }
+ }
+ }
+ }
+ },
+ "networkPolicies": {
+ "description": "Network policy configuration.",
+ "type": "object",
+ "properties": {
+ "enabled": {
+ "description": "Enabled network policies.",
+ "type": "boolean"
+ }
+ }
+ },
+ "airflowPodAnnotations": {
+ "description": "Extra annotations to apply to all Airflow pods.",
+ "type": "object"
+ },
+ "rbacEnabled": {
+ "description": "Enable RBAC (default on most clusters these days).",
+ "type": "boolean"
+ },
+ "executor": {
+ "description": "Airflow executor.",
+ "type": "string"
+ },
+ "allowPodLaunching": {
+ "description": "If this is true and using LocalExecutor/SequentialExecutor/KubernetesExecutor, the scheduler's service account will have access to communicate with the api-server and launch pods. If this is true and using the CeleryExecutor, the workers will be able to launch pods.",
+ "type": "boolean"
+ },
+ "images": {
+ "description": "Images.",
+ "type": "object",
+ "properties": {
+ "airflow": {
+ "description": "Configuration of the airflow image.",
+ "type": "object",
+ "properties": {
+ "repository": {
+ "description": "The airflow image repository.",
+ "type": [
+ "string",
+ "null"
+ ]
+ },
+ "tag": {
+ "description": "The airflow image tag.",
+ "type": [
+ "string",
+ "null"
+ ]
+ },
+ "pullPolicy": {
+ "description": "The airflow image pull policy.",
+ "type": "string"
+ }
+ }
+ },
+ "flower": {
+ "description": "Configuration of the flower image.",
+ "type": "object",
+ "properties": {
+ "repository": {
+ "description": "The flower image repository.",
+ "type": [
+ "string",
+ "null"
+ ]
+ },
+ "tag": {
+ "description": "The flower image tag.",
+ "type": [
+ "string",
+ "null"
+ ]
+ },
+ "pullPolicy": {
+ "description": "The flower image pull policy.",
+ "type": "string"
+ }
+ }
+ },
+ "statsd": {
+ "description": "Configuration of the statsd image.",
+ "type": "object",
+ "properties": {
+ "repository": {
+ "description": "The statsd image repository.",
+ "type": "string"
+ },
+ "tag": {
+ "description": "The statsd image tag.",
+ "type": "string"
+ },
+ "pullPolicy": {
+ "description": "The statsd image pull policy.",
+ "type": "string"
+ }
+ }
+ },
+ "redis": {
+ "description": "Configuration of the redis image.",
+ "type": "object",
+ "properties": {
+ "repository": {
+ "description": "The redis image repository.",
+ "type": "string"
+ },
+ "tag": {
+ "description": "The redis image tag.",
+ "type": "string"
+ },
+ "pullPolicy": {
+ "description": "The redis image pull policy.",
+ "type": "string"
+ }
+ }
+ },
+ "pgbouncer": {
+ "description": "Configuration of the pgbouncer image.",
+ "type": "object",
+ "properties": {
+ "repository": {
+ "description": "The pgbouncer image repository.",
+ "type": "string"
+ },
+ "tag": {
+ "description": "The pgbouncer image tag.",
+ "type": "string"
+ },
+ "pullPolicy": {
+ "description": "The pgbouncer image pull policy.",
+ "type": "string"
+ }
+ }
+ },
+ "pgbouncerExporter": {
+ "description": "Configuration of the pgbouncerExporter image.",
+ "type": "object",
+ "properties": {
+ "repository": {
+ "description": "The pgbouncerExporter image repository.",
+ "type": "string"
+ },
+ "tag": {
+ "description": "The pgbouncerExporter image tag.",
+ "type": "string"
+ },
+ "pullPolicy": {
+ "description": "The pgbouncerExporter image pull policy.",
+ "type": "string"
+ }
+ }
+ }
+ }
+ },
+ "env": {
+ "description": "Environment variables for all airflow containers.",
+ "type": "array"
+ },
+ "secret": {
+ "description": "Secrets for all airflow containers.",
+ "type": "array"
+ },
+ "data": {
+ "description": "Airflow database configuration.",
+ "type": "object",
+ "properties": {
+ "metadataSecretName": {
+ "description": "Metadata connection string secret.",
+ "type": [
+ "string",
+ "null"
+ ]
+ },
+ "resultBackendSecretName": {
+ "description": "Result backend connection string secret.",
+ "type": [
+ "string",
+ "null"
+ ]
+ },
+ "metadataConnection": {
+ "description": "Metadata connection configuration.",
+ "type": "object",
+ "properties": {
+ "user": {
+ "description": "The database user.",
+ "type": "string"
+ },
+ "pass": {
+ "description": "The user's password.",
+ "type": "string"
+ },
+ "host": {
+ "description": "The database host.",
+ "type": [
+ "string",
+ "null"
+ ]
+ },
+ "port": {
+ "description": "The database port.",
+ "type": "integer"
+ },
+ "db": {
+ "description": "The name of the database.",
+ "type": "string"
+ },
+ "sslmode": {
+ "description": "The database SSL parameter.",
+ "type": "string"
+ }
+ }
+ },
+ "resultBackendConnection": {
+ "description": "Result backend connection configuration.",
+ "type": "object",
+ "properties": {
+ "user": {
+ "description": "The database user.",
+ "type": "string"
+ },
+ "pass": {
+ "description": "The database password.",
+ "type": "string"
+ },
+ "host": {
+ "description": "The database host.",
+ "type": [
+ "string",
+ "null"
+ ]
+ },
+ "port": {
+ "description": "The database port.",
+ "type": "integer"
+ },
+ "db": {
+ "description": "The name of the database.",
+ "type": "string"
+ },
+ "sslmode": {
+ "description": "The database SSL parameter.",
+ "type": "string"
+ }
+ }
+ }
+ }
+ },
+ "fernetKey": {
+ "description": "The Fernet key used to encrypt passwords.",
+ "type": [
+ "string",
+ "null"
+ ]
+ },
+ "fernetKeySecretName": {
+ "description": "The Fernet key secret name.",
+ "type": [
+ "string",
+ "null"
+ ]
+ },
+ "workers": {
+ "description": "Airflow Worker configuration.",
+ "type": "object",
+ "properties": {
+ "replicas": {
+ "description": "Number of airflow celery workers in StatefulSet.",
+ "type": "integer"
+ },
+ "keda": {
+ "description": "KEDA configuration.",
+ "type": "object",
+ "properties": {
+ "enabled": {
+ "description": "Allow KEDA autoscaling. `Persistence.enabled` must be set to false to use KEDA.",
+ "type": "boolean"
+ },
+ "namespaceLabels": {
+ "type": "object"
+ },
+ "pollingInterval": {
+ "description": "How often KEDA polls the airflow DB to report new scale requests to the HPA.",
+ "type": "integer"
+ },
+ "cooldownPeriod": {
+ "description": "How many seconds KEDA will wait before scaling to zero.",
+ "type": "integer"
+ },
+ "maxReplicaCount": {
+ "description": "Maximum number of workers created by KEDA.",
+ "type": "integer"
+ }
+ }
+ },
+ "persistence": {
+ "description": "Persistence configuration.",
+ "type": "object",
+ "properties": {
+ "enabled": {
+ "description": "Enable persistent volumes.",
+ "type": "boolean"
+ },
+ "size": {
+ "description": "Volume size for worker StatefulSet.",
+ "type": "string"
+ },
+ "storageClassName": {
+ "description": "If using a custom storageClass, pass name ref to all StatefulSets here.",
+ "type": [
+ "string",
+ "null"
+ ]
+ },
+ "fixPermissions": {
+ "description": "Execute init container to chown log directory. This is currently only needed in KinD, due to usage of local-path provisioner.",
+ "type": "boolean"
+ }
+ }
+ },
+ "resources": {
+ "type": "object"
+ },
+ "terminationGracePeriodSeconds": {
+ "description": "Grace period for tasks to finish after SIGTERM is sent from Kubernetes.",
+ "type": "integer"
+ },
+ "safeToEvict": {
+ "description": "This setting tells Kubernetes that it's ok to evict when it wants to scale a node down.",
+ "type": "boolean"
+ }
+ }
+ },
+ "scheduler": {
+ "description": "Airflow scheduler settings.",
+ "type": "object",
+ "properties": {
+ "podDisruptionBudget": {
+ "description": "Scheduler pod disruption budget.",
+ "type": "object",
+ "properties": {
+ "enabled": {
+ "description": "Enable pod disruption budget.",
+ "type": "boolean"
+ },
+ "config": {
+ "description": "Disruption budget configuration.",
+ "type": "object",
+ "properties": {
+ "maxUnavailable": {
+ "description": "Max unavailable pods for scheduler.",
+ "type": "integer"
+ }
+ }
+ }
+ }
+ },
+ "resources": {
+ "type": "object"
+ },
+ "airflowLocalSettings": {
+ "description": "This setting can overwrite podMutation setting.",
+ "type": [
+ "string",
+ "null"
+ ]
+ },
+ "safeToEvict": {
+ "description": "This setting tells Kubernetes that its ok to evict when it wants to scale a node down.",
+ "type": "boolean"
+ }
+ }
+ },
+ "webserver": {
+ "description": "Airflow webserver settings.",
+ "type": "object",
+ "properties": {
+ "livenessProbe": {
+ "description": "Liveness probe configuration.",
+ "type": "object",
+ "properties": {
+ "initialDelaySeconds": {
+ "description": "Webserver Liveness probe initial delay.",
+ "type": "integer"
+ },
+ "timeoutSeconds": {
+ "description": "Webserver Liveness probe timeout seconds.",
+ "type": "integer"
+ },
+ "failureThreshold": {
+ "description": "Webserver Liveness probe failure threshold.",
+ "type": "integer"
+ },
+ "periodSeconds": {
+ "description": "Webserver Liveness probe period seconds.",
+ "type": "integer"
+ }
+ }
+ },
+ "readinessProbe": {
+ "description": "Readiness probe configuration.",
+ "type": "object",
+ "properties": {
+ "initialDelaySeconds": {
+ "description": "Webserver Readiness probe initial delay.",
+ "type": "integer"
+ },
+ "timeoutSeconds": {
+ "description": "Webserver Readiness probe timeout seconds.",
+ "type": "integer"
+ },
+ "failureThreshold": {
+ "description": "Webserver Readiness probe failure threshold.",
+ "type": "integer"
+ },
+ "periodSeconds": {
+ "description": "Webserver Readiness probe period seconds.",
+ "type": "integer"
+ }
+ }
+ },
+ "replicas": {
+ "description": "How many Airflow webserver replicas should run.",
+ "type": "integer"
+ },
+ "extraNetworkPolicies": {
+ "description": "Additional network policies as needed.",
+ "type": "array"
+ },
+ "resources": {
+ "type": "object"
+ },
+ "defaultUser": {
+ "description": "Optional default airflow user information",
+ "type": "object",
+ "properties": {
+ "enabled": {
+ "description": "Enable default user creation.",
+ "type": "boolean"
+ },
+ "role": {
+ "description": "Default user role.",
+ "type": "string"
+ },
+ "username": {
+ "description": "Default user username.",
+ "type": "string"
+ },
+ "email": {
+ "description": "Default user email address.",
+ "type": "string"
+ },
+ "firstName": {
+ "description": "Default user firstname.",
+ "type": "string"
+ },
+ "lastName": {
+ "description": "Default user lastname.",
+ "type": "string"
+ },
+ "password": {
+ "description": "Default user password.",
+ "type": "string"
+ }
+ }
+ },
+ "extraVolumes": {
+ "description": "Mount additional volumes into webserver.",
+ "type": "array"
+ },
+ "extraVolumeMounts": {
+ "description": "Mount additional volumes into webserver.",
+ "type": "array"
+ },
+ "webserverConfig": {
+ "description": "This will be mounted into the Airflow Webserver as a custom `webserver_config.py`. You can bake a `webserver_config.py` in to your image instead.",
+ "type": [
+ "string",
+ "null"
+ ]
+ },
+ "service": {
+ "description": "Webserver service configuration.",
+ "type": "object",
+ "properties": {
+ "type": {
+ "description": "Webserver service type.",
+ "type": "string"
+ },
+ "annotations": {
+ "description": "Annotations for the webserver service.",
+ "type": "object"
+ }
+ }
+ }
+ }
+ },
+ "flower": {
+ "description": "Flower settings.",
+ "type": "object",
+ "properties": {
+ "extraNetworkPolicies": {
+ "description": "Additional network policies as needed.",
+ "type": "array"
+ },
+ "resources": {
+ "type": "object"
+ },
+ "service": {
+ "description": "Flower service configuration.",
+ "type": "object",
+ "properties": {
+ "type": {
+ "description": "Flower service type.",
+ "type": "string"
+ }
+ }
+ }
+ }
+ },
+ "statsd": {
+ "description": "Statsd settings.",
+ "type": "object",
+ "properties": {
+ "enabled": {
+ "description": "Enable statsd.",
+ "type": "boolean"
+ },
+ "extraNetworkPolicies": {
+ "description": "Additional network policies as needed.",
+ "type": "array"
+ },
+ "resources": {
+ "type": "object"
+ },
+ "service": {
+ "description": "Statsd service configuration.",
+ "type": "object",
+ "properties": {
+ "extraAnnotations": {
+ "description": "Extra annotations for the statsd service.",
+ "type": "object"
+ }
+ }
+ }
+ }
+ },
+ "pgbouncer": {
+ "description": "Pgbouncer settings.",
+ "type": "object",
+ "properties": {
+ "enabled": {
+ "description": "Enable pgbouncer.",
+ "type": "boolean"
+ },
+ "extraNetworkPolicies": {
+ "description": "Additional network policies as needed.",
+ "type": "array"
+ },
+ "metadataPoolSize": {
+ "description": "Metadata pool size.",
+ "type": "integer"
+ },
+ "resultBackendPoolSize": {
+ "description": "Result backend pool size.",
+ "type": "integer"
+ },
+ "maxClientConn": {
+ "description": "Maximum clients that can connect to pgbouncer (higher = more file descriptors).",
+ "type": "integer"
+ },
+ "podDisruptionBudget": {
+ "description": "Pgbouner pod disruption budget.",
+ "type": "object",
+ "properties": {
+ "enabled": {
+ "description": "Enabled pod distribution budget.",
+ "type": "boolean"
+ },
+ "config": {
+ "description": "Pod distribution configuration.",
+ "type": "object",
+ "properties": {
+ "maxUnavailable": {
+ "description": "Max unavailable pods for pgbouncer.",
+ "type": "integer"
+ }
+ }
+ }
+ }
+ },
+ "resources": {
+ "type": "object"
+ },
+ "service": {
+ "description": "Pgbouncer service configuration.",
+ "type": "object",
+ "properties": {
+ "extraAnnotations": {
+ "description": "Extra annotations for the pgbouncer service.",
+ "type": "object"
+ }
+ }
+ },
+ "verbose": {
+ "description": "Increase pgbouncer verbosity.",
+ "type": "integer"
+ },
+ "logDisconnections": {
+ "description": "Log disconnections with reasons.",
+ "type": "integer"
+ },
+ "logConnections": {
+ "description": "Log successful logins.",
+ "type": "integer"
+ }
+ }
+ },
+ "redis": {
+ "description": "",
+ "type": "object",
+ "properties": {
+ "terminationGracePeriodSeconds": {
+ "description": "Grace period for tasks to finish after SIGTERM is sent from Kubernetes.",
+ "type": "integer"
+ },
+ "persistence": {
+ "description": "Persistence configuration.",
+ "type": "object",
+ "properties": {
+ "enabled": {
+ "description": "Enable persistent volumes.",
+ "type": "boolean"
+ },
+ "size": {
+ "description": "Volume size for worker StatefulSet.",
+ "type": "string"
+ },
+ "storageClassName": {
+ "description": "If using a custom storageClass, pass name ref to all StatefulSets here.",
+ "type": [
+ "string",
+ "null"
+ ]
+ }
+ }
+ },
+ "resources": {
+ "type": "object"
+ },
+ "passwordSecretName": {
+ "description": "Redis password secret.",
+ "type": [
+ "string",
+ "null"
+ ]
+ },
+ "brokerURLSecretName": {
+ "description": "Redis broker URL secret.",
+ "type": [
+ "string",
+ "null"
+ ]
+ },
+ "password": {
+ "description": "If password is set, create secret with it, else generate a new one on install.",
+ "type": [
+ "string",
+ "null"
+ ]
+ },
+ "safeToEvict": {
+ "description": "This setting tells Kubernetes that its ok to evict when it wants to scale a node down.",
+ "type": "boolean"
+ }
+ }
+ },
+ "registry": {
+ "description": "Auth secret for a private registry. This is used if pulling airflow images from a private registry.",
+ "type": "object",
+ "properties": {
+ "secretName": {
+ "description": "Registry connection string secret.",
+ "type": [
+ "string",
+ "null"
+ ]
+ },
+ "connection": {
+ "description": "Registry connection configuration.",
+ "type": "object"
+ }
+ }
+ },
+ "elasticsearch": {
+ "description": "Elasticsearch logging configuration.",
+ "type": "object",
+ "properties": {
+ "enabled": {
+ "description": "Enable elasticsearch task logging.",
+ "type": "boolean"
+ },
+ "secretName": {
+ "description": "A secret containing the connection string.",
+ "type": [
+ "string",
+ "null"
+ ]
+ },
+ "connection": {
+ "description": "Elasticsearch connection configuration.",
+ "type": "object"
+ }
+ }
+ },
+ "ports": {
+ "description": "All ports used by chart.",
+ "type": "object",
+ "properties": {
+ "flowerUI": {
+ "description": "Flower UI port.",
+ "type": "integer"
+ },
+ "airflowUI": {
+ "description": "Airflow UI port.",
+ "type": "integer"
+ },
+ "workerLogs": {
+ "description": "Worker logs port.",
+ "type": "integer"
+ },
+ "redisDB": {
+ "description": "Redis port.",
+ "type": "integer"
+ },
+ "statsdIngest": {
+ "description": "Statsd ingest port.",
+ "type": "integer"
+ },
+ "statsdScrape": {
+ "description": "Statsd scrape port.",
+ "type": "integer"
+ },
+ "pgbouncer": {
+ "description": "Pgbouncer port.",
+ "type": "integer"
+ },
+ "pgbouncerScrape": {
+ "description": "Pgbouncer scrape port.",
+ "type": "integer"
+ }
+ }
+ },
+ "quotas": {
+ "description": "Define any ResourceQuotas for namespace.",
+ "type": "object"
+ },
+ "limits": {
+ "description": "Define default/max/min values for pods and containers in namespace.",
+ "type": "array"
+ },
+ "podMutation": {
+ "description": "Settings for pod_mutation_hook",
+ "type": "object",
+ "properties": {
+ "tolerations": {
+ "description": "Tolerations provided here would be applied using pod_mutation_hook. So any pods spun up using KubernetesExecutor or KubernetesPodOperator will contain these tolerations.",
+ "type": "array"
+ },
+ "affinity": {
+ "description": "Pods spun up would land in the node that matches the affinity.",
+ "type": "object"
+ }
+ }
+ },
+ "cleanup": {
+ "description": "This runs as a CronJob to cleanup old pods.",
+ "type": "object",
+ "properties": {
+ "enabled": {
+ "description": "Enable cleanup.",
+ "type": "boolean"
+ },
+ "schedule": {
+ "description": "Cleanup schedule.",
+ "type": "string"
+ }
+ }
+ },
+ "postgresql": {
+ "description": "Configuration for postgresql subchart.",
+ "type": "object",
+ "properties": {
+ "enabled": {
+ "description": "Enable postgresql subchart.",
+ "type": "boolean"
+ },
+ "postgresqlPassword": {
+ "description": "Postgresql password.",
+ "type": "string"
+ },
+ "postgresqlUsername": {
+ "description": "Postgresql username.",
+ "type": "string"
+ }
+ }
+ },
+ "config": {
+ "description": "Settings to go into the mounted airflow.cfg",
+ "type": "object",
+ "additionalProperties": {
+ "type": "object",
+ "additionalProperties": {
+ "type": [
+ "boolean",
+ "integer",
+ "number",
+ "string"
+ ]
+ }
+ }
+ },
+ "dags": {
+ "description": "DAGs settings.",
+ "type": "object",
+ "properties": {
+ "persistence": {
+ "description": "Persistence configuration.",
+ "type": "object",
+ "properties": {
+ "enabled": {
+ "description": "Enable persistent volume for storing dags.",
+ "type": "boolean"
+ },
+ "size": {
+ "description": "Volume size for dags.",
+ "type": "string"
+ },
+ "storageClassName": {
+ "description": "If using a custom storageClass, pass name here.",
+ "type": [
+ "string",
+ "null"
+ ]
+ },
+ "accessMode": {
+ "description": "Access mode of the persistent volume.",
+ "type": "string"
+ },
+ "existingClaim": {
+ "description": "The name of an existing PVC to use.",
+ "type": [
+ "string",
+ "null"
+ ]
+ }
+ }
+ },
+ "gitSync": {
+ "description": "Git sync settings.",
+ "type": "object",
+ "properties": {
+ "enabled": {
+ "description": "Enable Git sync.",
+ "type": "boolean"
+ },
+ "repo": {
+ "description": "Git repository.",
+ "type": "string"
+ },
+ "branch": {
+ "description": "Git branch",
+ "type": "string"
+ },
+ "rev": {
+ "description": "Git revision.",
+ "type": "string"
+ },
+ "root": {
+ "description": "Root directory.",
+ "type": "string"
+ },
+ "dest": {
+ "description": "Destination folder.",
+ "type": "string"
+ },
+ "depth": {
+ "description": "Repository depth.",
+ "type": "integer"
+ },
+ "maxFailures": {
+ "description": "The number of consecutive failures allowed before aborting.",
+ "type": "integer"
+ },
+ "subPath": {
+ "description": "Subpath within the repo where dags are located.",
+ "type": "string"
+ },
+ "wait": {
+ "description": "Interval between git sync attempts in seconds.",
+ "type": "integer"
+ },
+ "containerRepository": {
+ "description": "Git sync image repository.",
+ "type": "string"
+ },
+ "containerTag": {
+ "description": "Git sync image tag.",
+ "type": "string"
+ },
+ "containerName": {
+ "description": "Git sync container name.",
+ "type": "string"
+ }
+ }
+ }
+ }
+ }
+ }
+}
diff --git a/chart/values.yaml b/chart/values.yaml
index 513dc47..f7c5e67 100644
--- a/chart/values.yaml
+++ b/chart/values.yaml
@@ -31,7 +31,7 @@ airflowHome: "/opt/airflow"
defaultAirflowRepository: apache/airflow
# Default airflow tag to deploy
-defaultAirflowTag: 1.10.10.1-alpha2-python3.6
+defaultAirflowTag: 1.10.12
# Select certain nodes for airflow pods.
@@ -42,6 +42,59 @@ tolerations: []
# Add common labels to all objects and pods defined in this chart.
labels: {}
+# Ingress configuration
+ingress:
+ # Enable ingress resource
+ enabled: false
+
+ # Configs for the Ingress of the web Service
+ web:
+ # Annotations for the web Ingress
+ annotations: {}
+
+ # The path for the web Ingress
+ path: ""
+
+ # The hostname for the web Ingress
+ host: ""
+
+ # configs for web Ingress TLS
+ tls:
+ # Enable TLS termination for the web Ingress
+ enabled: false
+ # the name of a pre-created Secret containing a TLS private key and certificate
+ secretName: ""
+
+ # HTTP paths to add to the web Ingress before the default path
+ precedingPaths: []
+
+ # Http paths to add to the web Ingress after the default path
+ succeedingPaths: []
+
+ # Configs for the Ingress of the flower Service
+ flower:
+ # Annotations for the flower Ingress
+ annotations: {}
+
+ # The path for the flower Ingress
+ path: ""
+
+ # The hostname for the flower Ingress
+ host: ""
+
+ # configs for web Ingress TLS
+ tls:
+ # Enable TLS termination for the flower Ingress
+ enabled: false
+ # the name of a pre-created Secret containing a TLS private key and certificate
+ secretName: ""
+
+ # HTTP paths to add to the flower Ingress before the default path
+ precedingPaths: []
+
+ # Http paths to add to the flower Ingress after the default path
+ succeedingPaths: []
+
# Network policy configuration
networkPolicies:
# Enabled network policies
@@ -196,7 +249,7 @@ workers:
pollingInterval: 5
# How many seconds KEDA will wait before scaling to zero.
- # Note that HPA has a seperate cooldwon period for scale-downs
+ # Note that HPA has a separate cooldown period for scale-downs
cooldownPeriod: 30
# Maximum number of workers created by keda
@@ -280,12 +333,12 @@ webserver:
extraNetworkPolicies: []
resources: {}
- # limits:
- # cpu: 100m
- # memory: 128Mi
- # requests:
- # cpu: 100m
- # memory: 128Mi
+ # limits:
+ # cpu: 100m
+ # memory: 128Mi
+ # requests:
+ # cpu: 100m
+ # memory: 128Mi
# Create initial user.
defaultUser:
@@ -316,18 +369,20 @@ webserver:
service:
type: ClusterIP
+ ## service annotations
+ annotations: {}
# Flower settings
flower:
# Additional network policies as needed
extraNetworkPolicies: []
resources: {}
- # limits:
- # cpu: 100m
- # memory: 128Mi
- # requests:
- # cpu: 100m
- # memory: 128Mi
+ # limits:
+ # cpu: 100m
+ # memory: 128Mi
+ # requests:
+ # cpu: 100m
+ # memory: 128Mi
service:
type: ClusterIP
@@ -338,12 +393,12 @@ statsd:
# Additional network policies as needed
extraNetworkPolicies: []
resources: {}
- # limits:
- # cpu: 100m
- # memory: 128Mi
- # requests:
- # cpu: 100m
- # memory: 128Mi
+ # limits:
+ # cpu: 100m
+ # memory: 128Mi
+ # requests:
+ # cpu: 100m
+ # memory: 128Mi
service:
extraAnnotations: {}
@@ -370,21 +425,13 @@ pgbouncer:
config:
maxUnavailable: 1
- # Limit the resources to pgbouncerExported.
- # When you specify the resource request the scheduler uses this information to decide which node to place
- # the Pod on. When you specify a resource limit for a Container, the kubelet enforces those limits so
- # that the running container is not allowed to use more of that resource than the limit you set.
- # See: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/
- # Example:
- #
- # resource:
- # limits:
- # cpu: 100m
- # memory: 128Mi
- # requests:
- # cpu: 100m
- # memory: 128Mi
resources: {}
+ # limits:
+ # cpu: 100m
+ # memory: 128Mi
+ # requests:
+ # cpu: 100m
+ # memory: 128Mi
service:
extraAnnotations: {}
@@ -430,13 +477,11 @@ redis:
registry:
secretName: ~
- # Example:
- # connection:
- # user: ~
- # pass: ~
- # host: ~
- # email: ~
connection: {}
+ # user: ~
+ # pass: ~
+ # host: ~
+ # email: ~
# Elasticsearch logging configuration
elasticsearch:
@@ -445,14 +490,11 @@ elasticsearch:
# A secret containing the connection
secretName: ~
# Or an object representing the connection
- # Example:
- # connection:
- # user: ~
- # pass: ~
- # host: ~
- # port: ~
connection: {}
-
+ # user: ~
+ # pass: ~
+ # host: ~
+ # port: ~
# All ports used by chart
ports:
@@ -562,6 +604,7 @@ config:
namespace: '{{ .Release.Namespace }}'
airflow_configmap: '{{ include "airflow_config" . }}'
airflow_local_settings_configmap: '{{ include "airflow_config" . }}'
+ pod_template_file: '{{ include "airflow_pod_template_file" . }}/pod_template_file.yaml'
worker_container_repository: '{{ .Values.images.airflow.repository | default .Values.defaultAirflowRepository }}'
worker_container_tag: '{{ .Values.images.airflow.tag | default .Values.defaultAirflowTag }}'
worker_container_image_pull_policy: '{{ .Values.images.airflow.pullPolicy }}'
@@ -569,6 +612,7 @@ config:
image_pull_secrets: '{{ template "registry_secret" . }}'
dags_in_image: '{{ ternary "False" "True" (or .Values.dags.gitSync.enabled .Values.dags.persistence.enabled) }}'
delete_worker_pods: 'True'
+ multi_namespace_mode: '{{ .Values.multiNamespaceMode }}'
delete_worker_pods_on_failure: 'False'
run_as_user: '{{ .Values.uid }}'
fs_group: '{{ .Values.gid }}'
@@ -594,6 +638,10 @@ config:
AIRFLOW__CORE__FERNET_KEY: '{{ printf "%s=fernet-key" (include "fernet_key_secret" .) }}'
# yamllint enable rule:line-length
+multiNamespaceMode: 'False'
+
+podTemplate: ~
+
# Git sync
dags:
persistence:
@@ -635,7 +683,6 @@ dags:
# GIT_SYNC_USERNAME: <base64_encoded_git_username>
# GIT_SYNC_PASSWORD: <base64_encoded_git_password>
# and specify the name of the secret below
- #
# credentialsSecret: git-credentials
#
#
@@ -651,10 +698,8 @@ dags:
# gitSshKey: <base64_encoded_data>
# and specify the name of the secret below
# sshKeySecret: airflow-ssh-secret
- #
# If you are using an ssh private key, you can additionally
# specify the content of your known_hosts file, example:
- #
# knownHosts: |
# <host1>,<ip1> <key1>
# <host2>,<ip2> <key2>
[airflow] 02/32: Adds Kubernetes Service Account for the webserver
(#11131)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 4d9950827b60d0bfc78195832b855c453a33459c
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Sun Sep 27 23:39:14 2020 +0200
Adds Kubernetes Service Account for the webserver (#11131)
Webserver did not have a Kubernetes Service Account defined and
while we do not strictly need to use the service account for
anything now, having the Service Account defined allows to
define various capabilities for the webserver.
For example when you are in the GCP environment, you can map
the Kubernetes service account into a GCP one, using
Workload Identity without the need to define any secrets
and performing additional authentication.
Then you can have that GCP service account get
the permissions to write logs to GCS bucket. Similar mechanisms
exist in AWS and it also opens up on-premises configuration.
See more at
https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity
Co-authored-by: Jacob Ferriero <jf...@google.com>
Co-authored-by: Jacob Ferriero <jf...@google.com>
(cherry picked from commit 0ea3e611d38f0829b3a2a06a9ba7af6423b2921d)
---
.../templates/webserver/webserver-deployment.yaml | 1 +
.../webserver/webserver-serviceaccount.yaml | 32 ++++++++++++++++++++++
2 files changed, 33 insertions(+)
diff --git a/chart/templates/webserver/webserver-deployment.yaml b/chart/templates/webserver/webserver-deployment.yaml
index 9ea2bc1..f6f781f 100644
--- a/chart/templates/webserver/webserver-deployment.yaml
+++ b/chart/templates/webserver/webserver-deployment.yaml
@@ -59,6 +59,7 @@ spec:
{{- toYaml .Values.airflowPodAnnotations | nindent 8 }}
{{- end }}
spec:
+ serviceAccountName: {{ .Release.Name }}-webserver
nodeSelector:
{{ toYaml .Values.nodeSelector | indent 8 }}
affinity:
diff --git a/chart/templates/webserver/webserver-serviceaccount.yaml b/chart/templates/webserver/webserver-serviceaccount.yaml
new file mode 100644
index 0000000..ea86713
--- /dev/null
+++ b/chart/templates/webserver/webserver-serviceaccount.yaml
@@ -0,0 +1,32 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+######################################
+## Airflow Webserver ServiceAccount
+######################################
+kind: ServiceAccount
+apiVersion: v1
+metadata:
+ name: {{ .Release.Name }}-webserver
+ labels:
+ tier: airflow
+ release: {{ .Release.Name }}
+ chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
+ heritage: {{ .Release.Service }}
+{{- with .Values.labels }}
+{{ toYaml . | indent 4 }}
+{{- end }}
[airflow] 26/32: Enable Black - Python Auto Formmatter (#9550)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 265d155858052916d2325023815ef93a21316948
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Sun Nov 15 03:42:47 2020 +0100
Enable Black - Python Auto Formmatter (#9550)
(cherry-picked from commit 4e8f9cc8d02b29c325b8a5a76b4837671bdf5f68)
---
chart/tests/test_celery_kubernetes_executor.py | 1 +
chart/tests/test_celery_kubernetes_pod_launcher_role.py | 1 +
chart/tests/test_chart_quality.py | 3 +--
chart/tests/test_dags_persistent_volume_claim.py | 1 +
chart/tests/test_flower_authorization.py | 1 +
chart/tests/test_git_sync_scheduler.py | 1 +
chart/tests/test_git_sync_webserver.py | 1 +
chart/tests/test_git_sync_worker.py | 1 +
chart/tests/test_migrate_database_job.py | 1 +
chart/tests/test_pod_template_file.py | 3 ++-
chart/tests/test_scheduler.py | 1 +
chart/tests/test_worker.py | 1 +
12 files changed, 13 insertions(+), 3 deletions(-)
diff --git a/chart/tests/test_celery_kubernetes_executor.py b/chart/tests/test_celery_kubernetes_executor.py
index 57c3980..fb21929 100644
--- a/chart/tests/test_celery_kubernetes_executor.py
+++ b/chart/tests/test_celery_kubernetes_executor.py
@@ -18,6 +18,7 @@
import unittest
import jmespath
+
from tests.helm_template_generator import render_chart
diff --git a/chart/tests/test_celery_kubernetes_pod_launcher_role.py b/chart/tests/test_celery_kubernetes_pod_launcher_role.py
index 535be11..952bc39 100644
--- a/chart/tests/test_celery_kubernetes_pod_launcher_role.py
+++ b/chart/tests/test_celery_kubernetes_pod_launcher_role.py
@@ -18,6 +18,7 @@
import unittest
import jmespath
+
from tests.helm_template_generator import render_chart
diff --git a/chart/tests/test_chart_quality.py b/chart/tests/test_chart_quality.py
index 32237cb..5f17165 100644
--- a/chart/tests/test_chart_quality.py
+++ b/chart/tests/test_chart_quality.py
@@ -18,11 +18,10 @@
import json
import os
import unittest
-import yaml
+import yaml
from jsonschema import validate
-
CHART_FOLDER = os.path.dirname(os.path.dirname(__file__))
diff --git a/chart/tests/test_dags_persistent_volume_claim.py b/chart/tests/test_dags_persistent_volume_claim.py
index 069a0cd..946c40f 100644
--- a/chart/tests/test_dags_persistent_volume_claim.py
+++ b/chart/tests/test_dags_persistent_volume_claim.py
@@ -18,6 +18,7 @@
import unittest
import jmespath
+
from tests.helm_template_generator import render_chart
diff --git a/chart/tests/test_flower_authorization.py b/chart/tests/test_flower_authorization.py
index f0cc5b0..0520ddd 100644
--- a/chart/tests/test_flower_authorization.py
+++ b/chart/tests/test_flower_authorization.py
@@ -18,6 +18,7 @@
import unittest
import jmespath
+
from tests.helm_template_generator import render_chart
diff --git a/chart/tests/test_git_sync_scheduler.py b/chart/tests/test_git_sync_scheduler.py
index a01c0f2..068f36c 100644
--- a/chart/tests/test_git_sync_scheduler.py
+++ b/chart/tests/test_git_sync_scheduler.py
@@ -18,6 +18,7 @@
import unittest
import jmespath
+
from tests.helm_template_generator import render_chart
diff --git a/chart/tests/test_git_sync_webserver.py b/chart/tests/test_git_sync_webserver.py
index 30c3f33..75ec51b 100644
--- a/chart/tests/test_git_sync_webserver.py
+++ b/chart/tests/test_git_sync_webserver.py
@@ -18,6 +18,7 @@
import unittest
import jmespath
+
from tests.helm_template_generator import render_chart
diff --git a/chart/tests/test_git_sync_worker.py b/chart/tests/test_git_sync_worker.py
index a70d311..e5036d7 100644
--- a/chart/tests/test_git_sync_worker.py
+++ b/chart/tests/test_git_sync_worker.py
@@ -18,6 +18,7 @@
import unittest
import jmespath
+
from tests.helm_template_generator import render_chart
diff --git a/chart/tests/test_migrate_database_job.py b/chart/tests/test_migrate_database_job.py
index 0524315..4b92aca 100644
--- a/chart/tests/test_migrate_database_job.py
+++ b/chart/tests/test_migrate_database_job.py
@@ -18,6 +18,7 @@
import unittest
import jmespath
+
from tests.helm_template_generator import render_chart
diff --git a/chart/tests/test_pod_template_file.py b/chart/tests/test_pod_template_file.py
index 0673e08..d9334de 100644
--- a/chart/tests/test_pod_template_file.py
+++ b/chart/tests/test_pod_template_file.py
@@ -17,10 +17,11 @@
import unittest
from os import remove
-from os.path import realpath, dirname
+from os.path import dirname, realpath
from shutil import copyfile
import jmespath
+
from tests.helm_template_generator import render_chart
ROOT_FOLDER = realpath(dirname(realpath(__file__)) + "/..")
diff --git a/chart/tests/test_scheduler.py b/chart/tests/test_scheduler.py
index 9769848..eb5225e 100644
--- a/chart/tests/test_scheduler.py
+++ b/chart/tests/test_scheduler.py
@@ -18,6 +18,7 @@
import unittest
import jmespath
+
from tests.helm_template_generator import render_chart
diff --git a/chart/tests/test_worker.py b/chart/tests/test_worker.py
index 2fc6d17..9b3515e 100644
--- a/chart/tests/test_worker.py
+++ b/chart/tests/test_worker.py
@@ -18,6 +18,7 @@
import unittest
import jmespath
+
from tests.helm_template_generator import render_chart
[airflow] 27/32: Fix default values for Helm Chart (#12153)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit f7500455ccc8d84e091bdfc405aaeb6f2c38eb60
Author: Kaxil Naik <ka...@gmail.com>
AuthorDate: Sat Nov 7 11:14:58 2020 +0000
Fix default values for Helm Chart (#12153)
(cherry picked from commit 874e497ff8b81121308018d02137361f66ae3c36)
---
chart/README.md | 12 ++++++------
1 file changed, 6 insertions(+), 6 deletions(-)
diff --git a/chart/README.md b/chart/README.md
index df8ea5d..db59db7 100644
--- a/chart/README.md
+++ b/chart/README.md
@@ -142,17 +142,17 @@ The following tables lists the configurable parameters of the Airflow chart and
| `images.flower.repository` | Docker repository to pull image from. Update this to deploy a custom image | `~` |
| `images.flower.tag` | Docker image tag to pull image from. Update this to deploy a new custom image tag | `~` |
| `images.flower.pullPolicy` | PullPolicy for flower image | `IfNotPresent` |
-| `images.statsd.repository` | Docker repository to pull image from. Update this to deploy a custom image | `astronomerinc/ap-statsd-exporter` |
-| `images.statsd.tag` | Docker image tag to pull image from. Update this to deploy a new custom image tag | `~` |
+| `images.statsd.repository` | Docker repository to pull image from. Update this to deploy a custom image | `apache/airflow` |
+| `images.statsd.tag` | Docker image tag to pull image from. Update this to deploy a new custom image tag | `airflow-statsd-exporter-2020.09.05-v0.17.0` |
| `images.statsd.pullPolicy` | PullPolicy for statsd-exporter image | `IfNotPresent` |
| `images.redis.repository` | Docker repository to pull image from. Update this to deploy a custom image | `redis` |
| `images.redis.tag` | Docker image tag to pull image from. Update this to deploy a new custom image tag | `6-buster` |
| `images.redis.pullPolicy` | PullPolicy for redis image | `IfNotPresent` |
-| `images.pgbouncer.repository` | Docker repository to pull image from. Update this to deploy a custom image | `astronomerinc/ap-pgbouncer` |
-| `images.pgbouncer.tag` | Docker image tag to pull image from. Update this to deploy a new custom image tag | `~` |
+| `images.pgbouncer.repository` | Docker repository to pull image from. Update this to deploy a custom image | `apache/airflow` |
+| `images.pgbouncer.tag` | Docker image tag to pull image from. Update this to deploy a new custom image tag | `airflow-pgbouncer-2020.09.05-1.14.0` |
| `images.pgbouncer.pullPolicy` | PullPolicy for pgbouncer image | `IfNotPresent` |
-| `images.pgbouncerExporter.repository` | Docker repository to pull image from. Update this to deploy a custom image | `astronomerinc/ap-pgbouncer-exporter` |
-| `images.pgbouncerExporter.tag` | Docker image tag to pull image from. Update this to deploy a new custom image tag | `~` |
+| `images.pgbouncerExporter.repository` | Docker repository to pull image from. Update this to deploy a custom image | `apache/airflow` |
+| `images.pgbouncerExporter.tag` | Docker image tag to pull image from. Update this to deploy a new custom image tag | `airflow-pgbouncer-exporter-2020.09.25-0.5.0` |
| `images.pgbouncerExporter.pullPolicy` | PullPolicy for pgbouncer-exporter image | `IfNotPresent` |
| `env` | Environment variables key/values to mount into Airflow pods | `[]` |
| `secret` | Secret name/key pairs to mount into Airflow pods | `[]` |
[airflow] 08/32: Mutual SSL added in PGBouncer configuration in the
Chart (#11384)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 6598002cfb724685e4375c67a41e0fd77b5d9d83
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Sun Oct 11 22:38:30 2020 +0200
Mutual SSL added in PGBouncer configuration in the Chart (#11384)
Adds SSL configuration for PGBouncer in the Helm Chart. PGBouncer
is crucial to handle the big number of connections that airflow
opens for the database, but often the database is outside of the
Kubernetes Cluster or generally the environment where Airflow is
installed and PGBouncer needs to connect securely to such database.
This PR adds capability of seting CA/Certificate and Private Key
in the PGBouncer configuration that allows for mTLS authentication
(both client and server are authenticated) and secure connection
even over public network.
(cherry picked from commit 9a01ce04076b20cba7c4f31f2665602512c78cdf)
---
chart/templates/_helpers.yaml | 18 +++++++++
.../templates/pgbouncer/pgbouncer-deployment.yaml | 22 +++++++++++
.../secrets/pgbouncer-certificates-secret.yaml | 44 ++++++++++++++++++++++
chart/values.schema.json | 27 +++++++++++++
chart/values.yaml | 8 ++++
5 files changed, 119 insertions(+)
diff --git a/chart/templates/_helpers.yaml b/chart/templates/_helpers.yaml
index 02b255a..4f3e0a0 100644
--- a/chart/templates/_helpers.yaml
+++ b/chart/templates/_helpers.yaml
@@ -223,6 +223,10 @@
{{ .Release.Name }}-pgbouncer-config
{{- end }}
+{{ define "pgbouncer_certificates_secret" -}}
+{{ .Release.Name }}-pgbouncer-certificates
+{{- end }}
+
{{ define "pgbouncer_stats_secret" -}}
{{ .Release.Name }}-pgbouncer-stats
{{- end }}
@@ -262,6 +266,20 @@ max_client_conn = {{ .Values.pgbouncer.maxClientConn }}
verbose = {{ .Values.pgbouncer.verbose }}
log_disconnections = {{ .Values.pgbouncer.logDisconnections }}
log_connections = {{ .Values.pgbouncer.logConnections }}
+
+server_tls_sslmode = {{ .Values.pgbouncer.sslmode }}
+server_tls_ciphers = {{ .Values.pgbouncer.ciphers }}
+
+{{- if .Values.pgbouncer.ssl.ca }}
+server_tls_ca_file = /etc/pgbouncer/root.crt
+{{- end }}
+{{- if .Values.pgbouncer.ssl.cert }}
+server_tls_cert_file = /etc/pgbouncer/server.crt
+{{- end }}
+{{- if .Values.pgbouncer.ssl.key }}
+server_tls_key_file = /etc/pgbouncer/server.key
+{{- end }}
+
{{- end }}
{{ define "pgbouncer_users" }}
diff --git a/chart/templates/pgbouncer/pgbouncer-deployment.yaml b/chart/templates/pgbouncer/pgbouncer-deployment.yaml
index 8a79e65..9237e6c 100644
--- a/chart/templates/pgbouncer/pgbouncer-deployment.yaml
+++ b/chart/templates/pgbouncer/pgbouncer-deployment.yaml
@@ -54,6 +54,7 @@ spec:
{{- end }}
annotations:
checksum/pgbouncer-config-secret: {{ include (print $.Template.BasePath "/secrets/pgbouncer-config-secret.yaml") . | sha256sum }}
+ checksum/pgbouncer-certificates-secret: {{ include (print $.Template.BasePath "/secrets/pgbouncer-certificates-secret.yaml") . | sha256sum }}
spec:
nodeSelector:
{{ toYaml .Values.nodeSelector | indent 8 }}
@@ -91,6 +92,24 @@ spec:
subPath: users.txt
mountPath: /etc/pgbouncer/users.txt
readOnly: true
+{{- if .Values.pgbouncer.ssl.ca }}
+ - name: pgbouncer-certificates
+ subPath: root.crt
+ mountPath: /etc/pgbouncer/root.crt
+ readOnly: true
+{{- end }}
+{{- if .Values.pgbouncer.ssl.cert }}
+ - name: pgbouncer-certificates
+ subPath: server.crt
+ mountPath: /etc/pgbouncer/server.crt
+ readOnly: true
+{{- end }}
+{{- if .Values.pgbouncer.ssl.key }}
+ - name: pgbouncer-certificates
+ subPath: server.key
+ mountPath: /etc/pgbouncer/server.key
+ readOnly: true
+{{- end }}
lifecycle:
preStop:
exec:
@@ -125,4 +144,7 @@ spec:
- name: pgbouncer-config
secret:
secretName: {{ template "pgbouncer_config_secret" . }}
+ - name: pgbouncer-certificates
+ secret:
+ secretName: {{ template "pgbouncer_certificates_secret" . }}
{{- end }}
diff --git a/chart/templates/secrets/pgbouncer-certificates-secret.yaml b/chart/templates/secrets/pgbouncer-certificates-secret.yaml
new file mode 100644
index 0000000..d9f1f2d
--- /dev/null
+++ b/chart/templates/secrets/pgbouncer-certificates-secret.yaml
@@ -0,0 +1,44 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+################################
+## Pgbouncer Certificate Secret
+#################################
+{{- if or .Values.pgbouncer.ssl.ca .Values.pgbouncer.ssl.cert .Values.pgbouncer.ssl.key }}
+kind: Secret
+apiVersion: v1
+metadata:
+ name: {{ template "pgbouncer_certificates_secret" . }}
+ labels:
+ release: {{ .Release.Name }}
+ chart: {{ .Chart.Name }}
+ heritage: {{ .Release.Service }}
+{{- with .Values.labels }}
+{{ toYaml . | indent 4 }}
+{{- end }}
+type: Opaque
+data:
+{{- if .Values.pgbouncer.ssl.ca }}
+ root.crt: {{ .Values.pgbouncer.ssl.ca | b64enc }}
+{{- end }}
+{{- if .Values.pgbouncer.ssl.cert }}
+ server.crt: {{ .Values.pgbouncer.ssl.cert | b64enc }}
+{{- end }}
+{{- if .Values.pgbouncer.ssl.key }}
+ server.key: {{ .Values.pgbouncer.ssl.key | b64enc }}
+{{- end }}
+{{- end }}
diff --git a/chart/values.schema.json b/chart/values.schema.json
index a5eda19..6c2c41a 100644
--- a/chart/values.schema.json
+++ b/chart/values.schema.json
@@ -764,6 +764,33 @@
"logConnections": {
"description": "Log successful logins.",
"type": "integer"
+ },
+ "sslmode": {
+ "description": "SSL mode for pgbouncer.",
+ "type": "string",
+ "enum": ["disable", "allow", "prefer", "require", "verify-ca", "verify-full"]
+ },
+ "ciphers": {
+ "description": "The allowed ciphers, might be 'fast', 'normal' or list ciphers separated with ':'.",
+ "type": "string"
+ },
+ "ssl": {
+ "description": "SSL certificates for pgbouncer connection.",
+ "type": "object",
+ "properties": {
+ "ca": {
+ "description": "Certificate Authority for server side",
+ "type": ["string", "null"]
+ },
+ "cert": {
+ "description": "Server Certificate for server side",
+ "type": ["string", "null"]
+ },
+ "key": {
+ "description": "Private key used to authenticate with the server",
+ "type": ["string", "null"]
+ }
+ }
}
}
},
diff --git a/chart/values.yaml b/chart/values.yaml
index 800f58a..707a34b 100644
--- a/chart/values.yaml
+++ b/chart/values.yaml
@@ -448,6 +448,14 @@ pgbouncer:
logDisconnections: 0
logConnections: 0
+ sslmode: "prefer"
+ ciphers: "normal"
+
+ ssl:
+ ca: ~
+ cert: ~
+ key: ~
+
redis:
terminationGracePeriodSeconds: 600
[airflow] 15/32: Pod template file uses custom custom env variable
(#11480)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 788f1c0019ce83eaa2cfdd698595190bc05f8905
Author: Prakshal Jain <32...@users.noreply.github.com>
AuthorDate: Mon Oct 19 14:22:19 2020 -0700
Pod template file uses custom custom env variable (#11480)
(cherry picked from commit 3970bfad4c878d99adce80a4bfd824a15132a161)
---
chart/files/pod-template-file.kubernetes-helm-yaml | 1 +
1 file changed, 1 insertion(+)
diff --git a/chart/files/pod-template-file.kubernetes-helm-yaml b/chart/files/pod-template-file.kubernetes-helm-yaml
index 2206c57..ed21d68 100644
--- a/chart/files/pod-template-file.kubernetes-helm-yaml
+++ b/chart/files/pod-template-file.kubernetes-helm-yaml
@@ -31,6 +31,7 @@ spec:
- name: AIRFLOW__CORE__EXECUTOR
value: LocalExecutor
{{- include "standard_airflow_environment" . | indent 4 }}
+{{- include "custom_airflow_environment" . | indent 4 }}
envFrom: []
image: {{ template "pod_template_image" . }}
imagePullPolicy: {{ .Values.images.airflow.pullPolicy }}
[airflow] 30/32: Fix indentation for affinities in helm chart
(#12288)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit fda5f7ad5f366aa8469f944b02a60c0a00ab8a8a
Author: Daniel Imberman <da...@gmail.com>
AuthorDate: Wed Nov 11 09:23:20 2020 -0800
Fix indentation for affinities in helm chart (#12288)
This PR fixes a bug in the helm chart where custom affinities in
the pod_template_file cause the yaml to fail due to invalid spacing
(cherry picked from commit cbe4ef2c5e143c87c9edd1c54a4949bbdd7a8edd)
---
chart/files/pod-template-file.kubernetes-helm-yaml | 9 ++--
chart/tests/test_pod_template_file.py | 51 ++++++++++++++++++++++
2 files changed, 54 insertions(+), 6 deletions(-)
diff --git a/chart/files/pod-template-file.kubernetes-helm-yaml b/chart/files/pod-template-file.kubernetes-helm-yaml
index 8647060..b4ec9a5 100644
--- a/chart/files/pod-template-file.kubernetes-helm-yaml
+++ b/chart/files/pod-template-file.kubernetes-helm-yaml
@@ -66,12 +66,9 @@ spec:
restartPolicy: Never
securityContext:
runAsUser: {{ .Values.uid }}
- nodeSelector:
- {{ toYaml .Values.nodeSelector | indent 8 }}
- affinity:
- {{ toYaml .Values.affinity | indent 8 }}
- tolerations:
- {{ toYaml .Values.tolerations | indent 8 }}
+ nodeSelector: {{ toYaml .Values.nodeSelector | nindent 4 }}
+ affinity: {{ toYaml .Values.affinity | nindent 4 }}
+ tolerations: {{ toYaml .Values.tolerations | nindent 4 }}
serviceAccountName: '{{ .Release.Name }}-worker'
volumes:
{{- if .Values.dags.persistence.enabled }}
diff --git a/chart/tests/test_pod_template_file.py b/chart/tests/test_pod_template_file.py
index d9334de..3c61733 100644
--- a/chart/tests/test_pod_template_file.py
+++ b/chart/tests/test_pod_template_file.py
@@ -184,3 +184,54 @@ class PodTemplateFileTest(unittest.TestCase):
self.assertRegex(docs[0]["kind"], "Pod")
self.assertEqual("dummy_image:latest", jmespath.search("spec.containers[0].image", docs[0]))
self.assertEqual("base", jmespath.search("spec.containers[0].name", docs[0]))
+
+ def test_should_create_valid_affinity_and_node_selector(self):
+ docs = render_chart(
+ values={
+ "affinity": {
+ "nodeAffinity": {
+ "requiredDuringSchedulingIgnoredDuringExecution": {
+ "nodeSelectorTerms": [
+ {
+ "matchExpressions": [
+ {"key": "foo", "operator": "In", "values": ["true"]},
+ ]
+ }
+ ]
+ }
+ }
+ },
+ "tolerations": [
+ {"key": "dynamic-pods", "operator": "Equal", "value": "true", "effect": "NoSchedule"}
+ ],
+ "nodeSelector": {"diskType": "ssd"},
+ },
+ show_only=["templates/pod-template-file.yaml"],
+ )
+
+ self.assertRegex(docs[0]["kind"], "Pod")
+ self.assertEqual(
+ "foo",
+ jmespath.search(
+ "spec.affinity.nodeAffinity."
+ "requiredDuringSchedulingIgnoredDuringExecution."
+ "nodeSelectorTerms[0]."
+ "matchExpressions[0]."
+ "key",
+ docs[0],
+ ),
+ )
+ self.assertEqual(
+ "ssd",
+ jmespath.search(
+ "spec.nodeSelector.diskType",
+ docs[0],
+ ),
+ )
+ self.assertEqual(
+ "dynamic-pods",
+ jmespath.search(
+ "spec.tolerations[0].key",
+ docs[0],
+ ),
+ )
[airflow] 25/32: Use PyUpgrade to use Python 3.6 features (#11447)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit a31ccb780d2d867c641ac2aa3c705f39c907a03f
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Sun Nov 15 03:41:17 2020 +0100
Use PyUpgrade to use Python 3.6 features (#11447)
Use features like `f-strings` instead of format across the code-base.
More details: https://github.com/asottile/pyupgrade
(cherry-picked from commit 8c42cf1b00c90f0d7f11b8a3a455381de8e003c5)
---
chart/tests/test_chart_quality.py | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/chart/tests/test_chart_quality.py b/chart/tests/test_chart_quality.py
index 3898941..32237cb 100644
--- a/chart/tests/test_chart_quality.py
+++ b/chart/tests/test_chart_quality.py
@@ -28,9 +28,9 @@ CHART_FOLDER = os.path.dirname(os.path.dirname(__file__))
class ChartQualityTest(unittest.TestCase):
def test_values_validate_schema(self):
- with open(os.path.join(CHART_FOLDER, "values.yaml"), "r") as f:
+ with open(os.path.join(CHART_FOLDER, "values.yaml")) as f:
values = yaml.safe_load(f)
- with open(os.path.join(CHART_FOLDER, "values.schema.json"), "r") as f:
+ with open(os.path.join(CHART_FOLDER, "values.schema.json")) as f:
schema = json.load(f)
# Add extra restrictions just for the tests to make sure
[airflow] 09/32: Adds missing schema for kerberos sidecar
configuration (#11413)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit f86d5345723476bb49c4980ae4587eb16e50788a
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Mon Oct 12 13:09:47 2020 +0200
Adds missing schema for kerberos sidecar configuration (#11413)
* Adds missing schema for kerberos sidecar configuration
The kerberos support added in #11130 did not have schema added
to the values.yml. This PR fixes it.
Co-authored-by: Jacob Ferriero <jf...@google.com>
* Update chart/values.schema.json
Co-authored-by: Jacob Ferriero <jf...@google.com>
(cherry picked from commit 9142eed715e87a6b04884dc8950c3af79aec0425)
---
chart/values.schema.json | 48 ++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 48 insertions(+)
diff --git a/chart/values.schema.json b/chart/values.schema.json
index 6c2c41a..2fc01a9 100644
--- a/chart/values.schema.json
+++ b/chart/values.schema.json
@@ -823,6 +823,16 @@
}
}
},
+ "kerberosSidecar": {
+ "description": "Run a side car in each worker pod to refresh Kerberos ccache with `airflow kerberos` according to the airflow security configuration",
+ "type": "object",
+ "properties": {
+ "enabled": {
+ "description": "Enable Kerberos side car on worker pods.",
+ "type": "boolean"
+ }
+ }
+ },
"resources": {
"type": "object"
},
@@ -1093,6 +1103,44 @@
}
}
}
+ },
+ "kerberos": {
+ "description": "Kerberos configurations for airflow",
+ "type": "object",
+ "properties": {
+ "enabled": {
+ "description": "Enable kerberos.",
+ "type": "boolean"
+ },
+ "ccacheMountPath": {
+ "description": "Path to mount shared volume for kerberos credentials cache.",
+ "type": "string"
+ },
+ "ccacheFileName": {
+ "description": "Name for kerberos credentials cache file.",
+ "type": "string"
+ },
+ "configPath":{
+ "description": "Path to mount krb5.conf kerberos configuration file.",
+ "type": "string"
+ },
+ "keytabPath":{
+ "description": "Path to mount the keytab for refreshing credentials in the kerberos sidecar.",
+ "type": "string"
+ },
+ "principal":{
+ "description": "Principal to use when refreshing kerberos credentials.",
+ "type": "string"
+ },
+ "reinitFrequency": {
+ "description": "How often (in seconds) airflow kerberos will reinitialize the credentials cache.",
+ "type": "integer"
+ },
+ "config": {
+ "description": "Contents of krb5.conf.",
+ "type": "string"
+ }
+ }
}
}
}
[airflow] 04/32: Kubernetes executor can adopt tasks from other
schedulers (#10996)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 3fbf6dfb9bcbfd2cce1d119206fd73546cfe46d4
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Sun Nov 15 03:09:27 2020 +0100
Kubernetes executor can adopt tasks from other schedulers (#10996)
(cherry picked from commit 3ca11eb9b02a2c2591292fd6b76e0e98b8f22656)
Co-authored-by: Daniel Imberman <da...@astronomer.io>
---
chart/templates/rbac/pod-launcher-role.yaml | 1 +
1 file changed, 1 insertion(+)
diff --git a/chart/templates/rbac/pod-launcher-role.yaml b/chart/templates/rbac/pod-launcher-role.yaml
index 132c839..7e10122 100644
--- a/chart/templates/rbac/pod-launcher-role.yaml
+++ b/chart/templates/rbac/pod-launcher-role.yaml
@@ -40,6 +40,7 @@ rules:
- "create"
- "list"
- "get"
+ - "patch"
- "watch"
- "delete"
- apiGroups:
[airflow] 10/32: Mount volumes and volumemounts into scheduler and
workers (#11426)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit f3d5fe2b76958ce780a644be58c278fd0d6c8798
Author: Daniel Imberman <da...@gmail.com>
AuthorDate: Mon Oct 12 11:32:05 2020 -0700
Mount volumes and volumemounts into scheduler and workers (#11426)
* Mount arbitrary volumes and volumeMounts to scheduler and worker
Allows users to mount volumes to scheduler and workers
* tested
(cherry picked from commit 045d68da7fccccdf5b8209a4fd3bf138b9ca9f96)
---
.../templates/scheduler/scheduler-deployment.yaml | 6 ++++
chart/templates/workers/worker-deployment.yaml | 6 ++++
chart/tests/scheduler_test.yaml | 38 ++++++++++++++++++++++
chart/tests/worker_test.yaml | 38 ++++++++++++++++++++++
chart/values.schema.json | 20 ++++++++++--
chart/values.yaml | 7 ++++
6 files changed, 113 insertions(+), 2 deletions(-)
diff --git a/chart/templates/scheduler/scheduler-deployment.yaml b/chart/templates/scheduler/scheduler-deployment.yaml
index 794a473..b6878a7 100644
--- a/chart/templates/scheduler/scheduler-deployment.yaml
+++ b/chart/templates/scheduler/scheduler-deployment.yaml
@@ -152,6 +152,9 @@ spec:
mountPath: {{ template "airflow_dags_mount_path" . }}
{{- include "git_sync_container" . | indent 8 }}
{{- end }}
+{{- if .Values.scheduler.extraVolumeMounts }}
+{{ toYaml .Values.scheduler.extraVolumeMounts | indent 12 }}
+{{- end }}
# Always start the garbage collector sidecar.
- name: scheduler-gc
image: {{ template "airflow_image" . }}
@@ -195,6 +198,9 @@ spec:
{{- if and .Values.dags.gitSync.enabled .Values.dags.gitSync.sshKeySecret }}
{{- include "git_sync_ssh_key_volume" . | indent 8 }}
{{- end }}
+{{- if .Values.scheduler.extraVolumes }}
+{{ toYaml .Values.scheduler.extraVolumes | indent 8 }}
+{{- end }}
{{- if not $stateful }}
- name: logs
emptyDir: {}
diff --git a/chart/templates/workers/worker-deployment.yaml b/chart/templates/workers/worker-deployment.yaml
index 38b3a9a..47aa18e 100644
--- a/chart/templates/workers/worker-deployment.yaml
+++ b/chart/templates/workers/worker-deployment.yaml
@@ -118,6 +118,9 @@ spec:
- name: worker-logs
containerPort: {{ .Values.ports.workerLogs }}
volumeMounts:
+{{- if .Values.workers.extraVolumeMounts }}
+{{ toYaml .Values.workers.extraVolumeMounts | indent 12 }}
+{{- end }}
- name: logs
mountPath: {{ template "airflow_logs" . }}
- name: config
@@ -200,6 +203,9 @@ spec:
{{- include "standard_airflow_environment" . | indent 10 }}
{{- end }}
volumes:
+{{- if .Values.workers.extraVolumes }}
+{{ toYaml .Values.workers.extraVolumes | indent 8 }}
+{{- end }}
- name: kerberos-keytab
secret:
secretName: {{ include "kerberos_keytab_secret" . | quote }}
diff --git a/chart/tests/scheduler_test.yaml b/chart/tests/scheduler_test.yaml
new file mode 100644
index 0000000..327b6e3
--- /dev/null
+++ b/chart/tests/scheduler_test.yaml
@@ -0,0 +1,38 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+---
+templates:
+ - scheduler/scheduler-deployment.yaml
+tests:
+ - it: should add extraVolume and extraVolumeMount
+ set:
+ executor: CeleryExecutor
+ scheduler:
+ extraVolumes:
+ - name: test-volume
+ emptyDir: {}
+ extraVolumeMounts:
+ - name: test-volume
+ mountPath: /opt/test
+ asserts:
+ - equal:
+ path: spec.template.spec.volumes[1].name
+ value: test-volume
+ - equal:
+ path: spec.template.spec.containers[0].volumeMounts[3].name
+ value: test-volume
diff --git a/chart/tests/worker_test.yaml b/chart/tests/worker_test.yaml
new file mode 100644
index 0000000..29c4b97
--- /dev/null
+++ b/chart/tests/worker_test.yaml
@@ -0,0 +1,38 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+---
+templates:
+ - workers/worker-deployment.yaml
+tests:
+ - it: should add extraVolume and extraVolumeMount
+ set:
+ executor: CeleryExecutor
+ workers:
+ extraVolumes:
+ - name: test-volume
+ emptyDir: {}
+ extraVolumeMounts:
+ - name: test-volume
+ mountPath: /opt/test
+ asserts:
+ - equal:
+ path: spec.template.spec.volumes[0].name
+ value: test-volume
+ - equal:
+ path: spec.template.spec.containers[0].volumeMounts[0].name
+ value: test-volume
diff --git a/chart/values.schema.json b/chart/values.schema.json
index 2fc01a9..9370c80 100644
--- a/chart/values.schema.json
+++ b/chart/values.schema.json
@@ -472,7 +472,15 @@
"description": "Annotations to add to the worker kubernetes service account.",
"type": "object"
}
- }
+ },
+ "extraVolumes": {
+ "description": "Mount additional volumes into workers.",
+ "type": "array"
+ },
+ "extraVolumeMounts": {
+ "description": "Mount additional volumes into workers.",
+ "type": "array"
+ }
},
"scheduler": {
"description": "Airflow scheduler settings.",
@@ -516,7 +524,15 @@
"description": "Annotations to add to the scheduler kubernetes service account.",
"type": "object"
}
- }
+ },
+ "extraVolumes": {
+ "description": "Mount additional volumes into scheduler.",
+ "type": "array"
+ },
+ "extraVolumeMounts": {
+ "description": "Mount additional volumes into scheduler.",
+ "type": "array"
+ }
},
"webserver": {
"description": "Airflow webserver settings.",
diff --git a/chart/values.yaml b/chart/values.yaml
index 707a34b..1370088 100644
--- a/chart/values.yaml
+++ b/chart/values.yaml
@@ -286,6 +286,9 @@ workers:
safeToEvict: true
# Annotations to add to worker kubernetes service account.
serviceAccountAnnotations: {}
+ # Mount additional volumes into worker.
+ extraVolumes: []
+ extraVolumeMounts: []
# Airflow scheduler settings
scheduler:
@@ -316,6 +319,10 @@ scheduler:
# Annotations to add to scheduler kubernetes service account.
serviceAccountAnnotations: {}
+ # Mount additional volumes into scheduler.
+ extraVolumes: []
+ extraVolumeMounts: []
+
# Airflow webserver settings
webserver:
livenessProbe:
[airflow] 16/32: fix pod launcher rolebinding in helm chart (#11675)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 6d8ec84a2e854189de9df167b726bb45127e1e46
Author: FloChehab <fl...@gmail.com>
AuthorDate: Tue Oct 20 23:31:37 2020 +0200
fix pod launcher rolebinding in helm chart (#11675)
* Followup to #11034
* Was not referencing the correct kind of resources if multiNamespaceMode = False
(cherry picked from commit 3391c90247c7f34687ba33a459aba8ea79517604)
---
chart/templates/rbac/pod-launcher-rolebinding.yaml | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/chart/templates/rbac/pod-launcher-rolebinding.yaml b/chart/templates/rbac/pod-launcher-rolebinding.yaml
index a3eab66..f258de7 100644
--- a/chart/templates/rbac/pod-launcher-rolebinding.yaml
+++ b/chart/templates/rbac/pod-launcher-rolebinding.yaml
@@ -42,7 +42,11 @@ metadata:
{{- end }}
roleRef:
apiGroup: rbac.authorization.k8s.io
+{{- if .Values.multiNamespaceMode }}
kind: ClusterRole
+{{- else }}
+ kind: Role
+{{- end }}
name: {{ .Release.Name }}-pod-launcher-role
subjects:
{{- if $grantScheduler }}
[airflow] 22/32: All k8s object must comply with JSON Schema
(#12003)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 21f792ffb2c7ae041d766853d1aa5fe7191d4fe4
Author: Kamil BreguĊa <mi...@users.noreply.github.com>
AuthorDate: Sat Oct 31 18:51:36 2020 +0100
All k8s object must comply with JSON Schema (#12003)
* All k8s resources should have global labels
* All k8s object must comply with JSON Schema
(cherry picked from commit 3c85c2c16e845fa7eafaadf7a7598428e4022c6f)
---
.pre-commit-config.yaml | 4 +-
chart/files/pod-template-file.kubernetes-helm-yaml | 9 +---
chart/templates/redis/redis-statefulset.yaml | 4 +-
.../scheduler/scheduler-serviceaccount.yaml | 6 +--
chart/templates/secrets/redis-secrets.yaml | 3 ++
chart/templates/webserver/webserver-ingress.yaml | 63 ++++++++++++++++++++++
chart/templates/webserver/webserver-service.yaml | 4 ++
.../webserver/webserver-serviceaccount.yaml | 10 ++--
chart/templates/workers/worker-serviceaccount.yaml | 12 ++---
chart/tests/helm_template_generator.py | 37 +++++++++++++
chart/tests/test_basic_helm_chart.py | 22 +++++++-
11 files changed, 146 insertions(+), 28 deletions(-)
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index 4dbf84c..2e27e50 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -272,14 +272,14 @@ repos:
^\s*def\s*\S*\([^:#)]*:.*| # Matches function param with Python3 type
^\sdef\s*\S*\(.*\):\s*\-\>\s*\S*.* # Matches -> return value syntax from Python3
)$
- exclude: ^dev|^scripts|^docs
+ exclude: ^dev|^scripts|^docs|^chart
pass_filenames: true
- id: python2-compile
name: Compile code using python2
language: system
entry: python2.7 -m py_compile
files: \.py$
- exclude: ^dev|^scripts|^docs
+ exclude: ^dev|^scripts|^docs|^chart
pass_filenames: true
require_serial: true
- id: incorrect-use-of-LoggingMixin
diff --git a/chart/files/pod-template-file.kubernetes-helm-yaml b/chart/files/pod-template-file.kubernetes-helm-yaml
index ed21d68..8647060 100644
--- a/chart/files/pod-template-file.kubernetes-helm-yaml
+++ b/chart/files/pod-template-file.kubernetes-helm-yaml
@@ -40,12 +40,7 @@ spec:
volumeMounts:
- mountPath: {{ template "airflow_logs" . }}
name: airflow-logs
-{{- if or .Values.dags.persistence.enabled .Values.dags.gitSync.enabled }}
- - mountPath: {{ template "airflow_dags_mount_path" . }}
- name: airflow-dags
- readOnly: false
-{{- end }}
-{{- if .Values.dags.gitSync.sshKeySecret }}
+{{- if .Values.dags.gitSync.knownHosts }}
- mountPath: /etc/git-secret/known_hosts
name: {{ .Values.dags.gitSync.knownHosts }}
subPath: known_hosts
@@ -77,7 +72,7 @@ spec:
{{ toYaml .Values.affinity | indent 8 }}
tolerations:
{{ toYaml .Values.tolerations | indent 8 }}
- serviceAccountName: '{{ .Release.Name }}-worker-serviceaccount'
+ serviceAccountName: '{{ .Release.Name }}-worker'
volumes:
{{- if .Values.dags.persistence.enabled }}
- name: dags
diff --git a/chart/templates/redis/redis-statefulset.yaml b/chart/templates/redis/redis-statefulset.yaml
index 6df78b4..ca47d97 100644
--- a/chart/templates/redis/redis-statefulset.yaml
+++ b/chart/templates/redis/redis-statefulset.yaml
@@ -48,10 +48,10 @@ spec:
{{- with .Values.labels }}
{{ toYaml . | indent 8 }}
{{- end }}
+ {{- if .Values.redis.safeToEvict }}
annotations:
- {{- if .Values.redis.safeToEvict }}
cluster-autoscaler.kubernetes.io/safe-to-evict: "true"
- {{- end }}
+ {{- end }}
spec:
nodeSelector:
{{ toYaml .Values.nodeSelector | indent 8 }}
diff --git a/chart/templates/scheduler/scheduler-serviceaccount.yaml b/chart/templates/scheduler/scheduler-serviceaccount.yaml
index 5dfa1dc..f12991d 100644
--- a/chart/templates/scheduler/scheduler-serviceaccount.yaml
+++ b/chart/templates/scheduler/scheduler-serviceaccount.yaml
@@ -28,13 +28,13 @@ metadata:
release: {{ .Release.Name }}
chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
heritage: {{ .Release.Service }}
+ {{- with .Values.labels }}
+ {{ toYaml . | nindent 4 }}
+ {{- end }}
{{- with .Values.scheduler.serviceAccountAnnotations }}
annotations:
{{- range $key, $value := . }}
{{- printf "%s: %s" $key (tpl $value $ | quote) | nindent 4 }}
{{- end }}
{{- end }}
-{{- with .Values.labels }}
-{{ toYaml . | indent 4 }}
-{{- end }}
{{- end }}
diff --git a/chart/templates/secrets/redis-secrets.yaml b/chart/templates/secrets/redis-secrets.yaml
index 7c9fe26..958474d 100644
--- a/chart/templates/secrets/redis-secrets.yaml
+++ b/chart/templates/secrets/redis-secrets.yaml
@@ -51,6 +51,9 @@ metadata:
release: {{ .Release.Name }}
chart: {{ .Chart.Name }}
heritage: {{ .Release.Service }}
+ {{- with .Values.labels }}
+ {{ toYaml . | nindent 4 }}
+ {{- end }}
annotations:
"helm.sh/hook": "pre-install"
"helm.sh/hook-delete-policy": "before-hook-creation"
diff --git a/chart/templates/webserver/webserver-ingress.yaml b/chart/templates/webserver/webserver-ingress.yaml
new file mode 100644
index 0000000..c249cb2
--- /dev/null
+++ b/chart/templates/webserver/webserver-ingress.yaml
@@ -0,0 +1,63 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+################################
+## Airflow Webserver Ingress
+#################################
+{{- if .Values.ingress.enabled }}
+apiVersion: networking.k8s.io/v1beta1
+kind: Ingress
+metadata:
+ name: {{ .Release.Name }}-airflow-ingress
+ labels:
+ tier: airflow
+ component: airflow-ingress
+ release: {{ .Release.Name }}
+ chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
+ heritage: {{ .Release.Service }}
+ {{- with .Values.ingress.web.annotations }}
+ annotations:
+ {{ toYaml . | indent 4 }}
+ {{- end }}
+spec:
+ {{- if .Values.ingress.web.tls.enabled }}
+ tls:
+ - hosts:
+ - {{ .Values.ingress.web.host }}
+ secretName: {{ .Values.ingress.web.tls.secretName }}
+ {{- end }}
+ rules:
+ - http:
+ paths:
+ {{- range .Values.ingress.web.precedingPaths }}
+ - path: {{ .path }}
+ backend:
+ serviceName: {{ .serviceName }}
+ servicePort: {{ .servicePort }}
+ {{- end }}
+ - path: {{ .Values.ingress.web.path }}
+ backend:
+ serviceName: {{ .Release.Name }}-webserver
+ servicePort: airflow-ui
+ {{- range .Values.ingress.web.succeedingPaths }}
+ - path: {{ .path }}
+ backend:
+ serviceName: {{ .serviceName }}
+ servicePort: {{ .servicePort }}
+ {{- end }}
+ host: {{ .Values.ingress.web.host }}
+{{- end }}
diff --git a/chart/templates/webserver/webserver-service.yaml b/chart/templates/webserver/webserver-service.yaml
index 77f5995..feae23d 100644
--- a/chart/templates/webserver/webserver-service.yaml
+++ b/chart/templates/webserver/webserver-service.yaml
@@ -31,6 +31,10 @@ metadata:
{{- with .Values.labels }}
{{ toYaml . | indent 4 }}
{{- end }}
+{{- with .Values.webserver.service.annotations }}
+ annotations:
+{{- toYaml . | nindent 4 }}
+{{- end }}
spec:
type: {{ .Values.webserver.service.type }}
selector:
diff --git a/chart/templates/webserver/webserver-serviceaccount.yaml b/chart/templates/webserver/webserver-serviceaccount.yaml
index ba99cea..e42d767 100644
--- a/chart/templates/webserver/webserver-serviceaccount.yaml
+++ b/chart/templates/webserver/webserver-serviceaccount.yaml
@@ -27,12 +27,10 @@ metadata:
release: {{ .Release.Name }}
chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
heritage: {{ .Release.Service }}
+ {{- with .Values.labels }}
+ {{ toYaml . | nindent 4 }}
+ {{- end }}
{{- with .Values.webserver.serviceAccountAnnotations }}
annotations:
- {{- range $key, $value := . }}
- {{- printf "%s: %s" $key (tpl $value $ | quote) | nindent 4 }}
- {{- end }}
+ {{ toYaml . | nindent 4 }}
{{- end }}
-{{- with .Values.labels }}
-{{ toYaml . | indent 4 }}
-{{- end }}
diff --git a/chart/templates/workers/worker-serviceaccount.yaml b/chart/templates/workers/worker-serviceaccount.yaml
index 4a7542e..f87751b 100644
--- a/chart/templates/workers/worker-serviceaccount.yaml
+++ b/chart/templates/workers/worker-serviceaccount.yaml
@@ -28,13 +28,11 @@ metadata:
release: {{ .Release.Name }}
chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
heritage: {{ .Release.Service }}
- {{- with .Values.workers.serviceAccountAnnotations }}
+ {{- with .Values.labels }}
+ {{ toYaml . | nindent 4 }}
+ {{- end }}
+ {{- with .Values.workers.serviceAccountAnnotations}}
annotations:
- {{- range $key, $value := . }}
- {{- printf "%s: %s" $key (tpl $value $ | quote) | nindent 4 }}
- {{- end }}
+ {{ toYaml . | nindent 4 }}
{{- end }}
- {{- with .Values.labels }}
-{{ toYaml . | indent 4 }}
-{{- end }}
{{- end }}
diff --git a/chart/tests/helm_template_generator.py b/chart/tests/helm_template_generator.py
index dccbb1e..ba870ed 100644
--- a/chart/tests/helm_template_generator.py
+++ b/chart/tests/helm_template_generator.py
@@ -17,13 +17,48 @@
import subprocess
import sys
+from functools import lru_cache
from tempfile import NamedTemporaryFile
+import jmespath
+import jsonschema
+import requests
import yaml
from kubernetes.client.api_client import ApiClient
api_client = ApiClient()
+BASE_URL_SPEC = "https://raw.githubusercontent.com/instrumenta/kubernetes-json-schema/master/v1.12.9"
+
+
+@lru_cache(maxsize=None)
+def create_validator(api_version, kind):
+ api_version = api_version.lower()
+ kind = kind.lower()
+
+ if '/' in api_version:
+ ext, _, api_version = api_version.partition("/")
+ ext = ext.split(".")[0]
+ url = f'{BASE_URL_SPEC}/{kind}-{ext}-{api_version}.json'
+ else:
+ url = f'{BASE_URL_SPEC}/{kind}-{api_version}.json'
+ request = requests.get(url)
+ request.raise_for_status()
+ schema = request.json()
+ jsonschema.Draft7Validator.check_schema(schema)
+ validator = jsonschema.Draft7Validator(schema)
+ return validator
+
+
+def validate_k8s_object(instance):
+ # Skip PostgresSQL chart
+ chart = jmespath.search("metadata.labels.chart", instance)
+ if chart and 'postgresql' in chart:
+ return
+
+ validate = create_validator(instance.get("apiVersion"), instance.get("kind"))
+ validate.validate(instance)
+
def render_chart(name="RELEASE-NAME", values=None, show_only=None):
"""
@@ -41,6 +76,8 @@ def render_chart(name="RELEASE-NAME", values=None, show_only=None):
templates = subprocess.check_output(command)
k8s_objects = yaml.load_all(templates)
k8s_objects = [k8s_object for k8s_object in k8s_objects if k8s_object] # type: ignore
+ for k8s_object in k8s_objects:
+ validate_k8s_object(k8s_object)
return k8s_objects
diff --git a/chart/tests/test_basic_helm_chart.py b/chart/tests/test_basic_helm_chart.py
index e535ac9..767a073 100644
--- a/chart/tests/test_basic_helm_chart.py
+++ b/chart/tests/test_basic_helm_chart.py
@@ -17,6 +17,8 @@
import unittest
+import jmespath
+
from tests.helm_template_generator import render_chart
OBJECT_COUNT_IN_BASIC_DEPLOYMENT = 22
@@ -24,7 +26,15 @@ OBJECT_COUNT_IN_BASIC_DEPLOYMENT = 22
class TestBaseChartTest(unittest.TestCase):
def test_basic_deployments(self):
- k8s_objects = render_chart("TEST-BASIC", {"chart": {'metadata': 'AA'}})
+ k8s_objects = render_chart(
+ "TEST-BASIC",
+ values={
+ "chart": {
+ 'metadata': 'AA',
+ },
+ 'labels': {"TEST-LABEL": "TEST-VALUE"},
+ },
+ )
list_of_kind_names_tuples = [
(k8s_object['kind'], k8s_object['metadata']['name']) for k8s_object in k8s_objects
]
@@ -56,6 +66,16 @@ class TestBaseChartTest(unittest.TestCase):
],
)
self.assertEqual(OBJECT_COUNT_IN_BASIC_DEPLOYMENT, len(k8s_objects))
+ for k8s_object in k8s_objects:
+ labels = jmespath.search('metadata.labels', k8s_object) or {}
+ if 'postgresql' in labels.get('chart'):
+ continue
+ k8s_name = k8s_object['kind'] + ":" + k8s_object['metadata']['name']
+ self.assertEqual(
+ 'TEST-VALUE',
+ labels.get("TEST-LABEL"),
+ f"Missing label TEST-LABEL on {k8s_name}. Current labels: {labels}",
+ )
def test_basic_deployment_without_default_users(self):
k8s_objects = render_chart("TEST-BASIC", {"webserver": {'defaultUser': {'enabled': False}}})
[airflow] 11/32: Allow multiple schedulers in helm chart (#11330)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 2458461314d97b7759db9a887e4fd1528bf6d94f
Author: Daniel Imberman <da...@gmail.com>
AuthorDate: Mon Oct 12 13:48:59 2020 -0700
Allow multiple schedulers in helm chart (#11330)
* Allow multiple schedulers in helm chart
* schema
* add docs
* add to readme
Co-authored-by: Daniel Imberman <da...@astronomer.io>
(cherry picked from commit 7b3a00d4f97ada5e09bfc33c0e48b3b1833bfe56)
---
chart/README.md | 1 +
chart/templates/create-user-job.yaml | 8 ++++++++
chart/templates/scheduler/scheduler-deployment.yaml | 8 ++++----
chart/values.schema.json | 4 ++++
chart/values.yaml | 3 +++
5 files changed, 20 insertions(+), 4 deletions(-)
diff --git a/chart/README.md b/chart/README.md
index 266ef58..41a53ad 100644
--- a/chart/README.md
+++ b/chart/README.md
@@ -187,6 +187,7 @@ The following tables lists the configurable parameters of the Airflow chart and
| `workers.safeToEvict` | Allow Kubernetes to evict worker pods if needed (node downscaling) | `true` |
| `scheduler.podDisruptionBudget.enabled` | Enable PDB on Airflow scheduler | `false` |
| `scheduler.podDisruptionBudget.config.maxUnavailable` | MaxUnavailable pods for scheduler | `1` |
+| `scheduler.replicas` | # of parallel schedulers (Airflow 2.0 using Mysql 8+ or Postgres only) | `1` |
| `scheduler.resources.limits.cpu` | CPU Limit of scheduler | `~` |
| `scheduler.resources.limits.memory` | Memory Limit of scheduler | `~` |
| `scheduler.resources.requests.cpu` | CPU Request of scheduler | `~` |
diff --git a/chart/templates/create-user-job.yaml b/chart/templates/create-user-job.yaml
index f21d4b4..642e36f 100644
--- a/chart/templates/create-user-job.yaml
+++ b/chart/templates/create-user-job.yaml
@@ -51,6 +51,14 @@ spec:
tolerations:
{{ toYaml .Values.tolerations | indent 8 }}
containers:
+ - name: run-airflow-migrations
+ image: {{ template "airflow_image" . }}
+ imagePullPolicy: {{ .Values.images.airflow.pullPolicy }}
+ # Support running against 1.10.x and 2.0.0dev/master
+ args: ["bash", "-c", "airflow upgradedb || airflow db upgrade"]
+ env:
+ {{- include "custom_airflow_environment" . | indent 10 }}
+ {{- include "standard_airflow_environment" . | indent 10 }}
- name: create-user
image: {{ template "airflow_image" . }}
imagePullPolicy: {{ .Values.images.airflow.pullPolicy }}
diff --git a/chart/templates/scheduler/scheduler-deployment.yaml b/chart/templates/scheduler/scheduler-deployment.yaml
index b6878a7..2cf3dce 100644
--- a/chart/templates/scheduler/scheduler-deployment.yaml
+++ b/chart/templates/scheduler/scheduler-deployment.yaml
@@ -46,7 +46,7 @@ spec:
{{- if $stateful }}
serviceName: {{ .Release.Name }}-scheduler
{{- end }}
- replicas: 1
+ replicas: {{ .Values.scheduler.replicas }}
selector:
matchLabels:
tier: airflow
@@ -90,11 +90,11 @@ spec:
- name: {{ template "registry_secret" . }}
{{- end }}
initContainers:
- - name: run-airflow-migrations
+ - name: wait-for-airflow-migrations
image: {{ template "airflow_image" . }}
imagePullPolicy: {{ .Values.images.airflow.pullPolicy }}
- # Support running against 1.10.x and 2.0.0dev/master
- args: ["bash", "-c", "airflow upgradedb || airflow db upgrade"]
+ args:
+ {{- include "wait-for-migrations-command" . | indent 10 }}
env:
{{- include "custom_airflow_environment" . | indent 10 }}
{{- include "standard_airflow_environment" . | indent 10 }}
diff --git a/chart/values.schema.json b/chart/values.schema.json
index 9370c80..b2f7288 100644
--- a/chart/values.schema.json
+++ b/chart/values.schema.json
@@ -486,6 +486,10 @@
"description": "Airflow scheduler settings.",
"type": "object",
"properties": {
+ "replicas": {
+ "description": "Airflow 2.0 allows users to run multiple schedulers. This feature is only recommended for Mysql 8+ and postgres",
+ "type": "integer"
+ },
"podDisruptionBudget": {
"description": "Scheduler pod disruption budget.",
"type": "object",
diff --git a/chart/values.yaml b/chart/values.yaml
index 1370088..7b609a6 100644
--- a/chart/values.yaml
+++ b/chart/values.yaml
@@ -292,6 +292,9 @@ workers:
# Airflow scheduler settings
scheduler:
+ # Airflow 2.0 allows users to run multiple schedulers,
+ # However this feature is only recommended for MySQL 8+ and Postgres
+ replicas: 1
# Scheduler pod disruption budget
podDisruptionBudget:
enabled: false
[airflow] 18/32: Remove unused value in Helm Chart - podMutation
(#11703)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit f9ba264dea869be2161d5cb1f60fd2e3b69406e7
Author: Kamil BreguĊa <mi...@users.noreply.github.com>
AuthorDate: Wed Oct 21 17:40:02 2020 +0200
Remove unused value in Helm Chart - podMutation (#11703)
(cherry picked from commit b946b4487086f6e1ed5e2ddf45fa258315d77a50)
---
chart/values.schema.json | 14 --------------
chart/values.yaml | 21 ---------------------
2 files changed, 35 deletions(-)
diff --git a/chart/values.schema.json b/chart/values.schema.json
index 7289374..4395372 100644
--- a/chart/values.schema.json
+++ b/chart/values.schema.json
@@ -985,20 +985,6 @@
"description": "Define default/max/min values for pods and containers in namespace.",
"type": "array"
},
- "podMutation": {
- "description": "Settings for pod_mutation_hook",
- "type": "object",
- "properties": {
- "tolerations": {
- "description": "Tolerations provided here would be applied using pod_mutation_hook. So any pods spun up using KubernetesExecutor or KubernetesPodOperator will contain these tolerations.",
- "type": "array"
- },
- "affinity": {
- "description": "Pods spun up would land in the node that matches the affinity.",
- "type": "object"
- }
- }
- },
"cleanup": {
"description": "This runs as a CronJob to cleanup old pods.",
"type": "object",
diff --git a/chart/values.yaml b/chart/values.yaml
index b8bd0f8..54a78d6 100644
--- a/chart/values.yaml
+++ b/chart/values.yaml
@@ -546,27 +546,6 @@ quotas: {}
# Define default/max/min values for pods and containers in namespace
limits: []
-# Config Settings for pod_mutation_hook
-podMutation:
- # Tolerations provided here would be applied using pod_mutation_hook
- # So any pods spun up using KubernetesExecutor or KubernetesPodOperator will contain these tolerations.
- tolerations: []
- # - key: "dynamic-pods"
- # operator: "Equal"
- # value: "true"
- # effect: "NoSchedule"
-
- # Pods spun up would land in the node that matches the affinity
- affinity: {}
- # nodeAffinity:
- # requiredDuringSchedulingIgnoredDuringExecution:
- # nodeSelectorTerms:
- # - matchExpressions:
- # - key: "company.io/dynamic-pods"
- # operator: In
- # values:
- # - "true"
-
# This runs as a CronJob to cleanup old pods.
cleanup:
enabled: false
[airflow] 03/32: Enables Kerberos sidecar support (#11130)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 96f2b6bb11fa2bfb72c51335235d4e0834d2fc03
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Mon Sep 28 00:13:36 2020 +0200
Enables Kerberos sidecar support (#11130)
Some of the users of Airflow are using Kerberos to authenticate
their worker workflows. Airflow has a basic support for Kerberos
for some of the operators and it has support to refresh the
temporary Kerberos tokens via `airflow kerberos` command.
This change adds support for the Kerberos side-car that connects
to the Kerberos Key Distribution Center and retrieves the
token using Keytab that should be deployed as Kubernetes Secret.
It uses shared volume to share the temporary token. The nice
thing about setting it up as a sidecar is that the Keytab
is never shared with the workers - the secret is only mounted
by the sidecar and the workers have only access to the temporary
token.
Depends on #11129
(cherry picked from commit 4d2a7870704385db492081b41119c12a51445897)
---
breeze | 2 +-
chart/README.md | 22 ++++++---
chart/templates/_helpers.yaml | 12 +++++
chart/templates/configmap.yaml | 4 ++
chart/templates/workers/worker-deployment.yaml | 61 +++++++++++++++++++++++++
chart/values.yaml | 63 ++++++++++++++++++++++++++
6 files changed, 156 insertions(+), 8 deletions(-)
diff --git a/breeze b/breeze
index 175a4ab..ff5d7cb 100755
--- a/breeze
+++ b/breeze
@@ -3073,7 +3073,7 @@ function breeze::run_breeze_command() {
# 3. last used version stored in ./build/PYTHON_MAJOR_MINOR_VERSION
# 4. DEFAULT_PYTHON_MAJOR_MINOR_VERSION from scripts/ci/libraries/_initialization.sh
#
-# Here points 2. and 3. are realised. If result is empty string , the 4. will be set in
+# Here points 2. and 3. are realized. If result is empty string , the 4. will be set in
# the next step (sanity_checks::basic_sanity_checks() is called and the version is still not set by then)
# finally, if --python flag is specified, it will override whatever is set above.
#
diff --git a/chart/README.md b/chart/README.md
index 11dc632..8372bb4 100644
--- a/chart/README.md
+++ b/chart/README.md
@@ -74,8 +74,7 @@ helm upgrade airflow . \
--set images.airflow.tag=8a0da78
```
-For local development purppose you can also u
-You can also build the image locally and use it via deployment method described by Breeze.
+For local development purpose you can also build the image locally and use it via deployment method described by Breeze.
## Mounting DAGS using Git-Sync side car with Persistence enabled
@@ -129,7 +128,7 @@ The following tables lists the configurable parameters of the Airflow chart and
| `privateRegistry.repository` | Repository where base image lives (eg: quay.io) | `~` |
| `networkPolicies.enabled` | Enable Network Policies to restrict traffic | `true` |
| `airflowHome` | Location of airflow home directory | `/opt/airflow` |
-| `rbacEnabled` | Deploy pods with Kubernets RBAC enabled | `true` |
+| `rbacEnabled` | Deploy pods with Kubernetes RBAC enabled | `true` |
| `executor` | Airflow executor (eg SequentialExecutor, LocalExecutor, CeleryExecutor, KubernetesExecutor) | `KubernetesExecutor` |
| `allowPodLaunching` | Allow airflow pods to talk to Kubernetes API to launch more pods | `true` |
| `defaultAirflowRepository` | Fallback docker repository to pull airflow image from | `apache/airflow` |
@@ -158,13 +157,22 @@ The following tables lists the configurable parameters of the Airflow chart and
| `data.resultBackendSecretName` | Secret name to mount Celery result backend connection string from | `~` |
| `data.metadataConection` | Field separated connection data (alternative to secret name) | `{}` |
| `data.resultBackendConnection` | Field separated connection data (alternative to secret name) | `{}` |
-| `fernetKey` | String representing an Airflow fernet key | `~` |
-| `fernetKeySecretName` | Secret name for Airlow fernet key | `~` |
+| `fernetKey` | String representing an Airflow Fernet key | `~` |
+| `fernetKeySecretName` | Secret name for Airflow Fernet key | `~` |
+| `kerberos.enabled` | Enable kerberos support for workers | `false` |
+| `kerberos.ccacheMountPath` | Location of the ccache volume | `/var/kerberos-ccache` |
+| `kerberos.ccacheFileName` | Name of the ccache file | `ccache` |
+| `kerberos.configPath` | Path for the Kerberos config file | `/etc/krb5.conf` |
+| `kerberos.keytabPath` | Path for the Kerberos keytab file | `/etc/airflow.keytab` |
+| `kerberos.principal` | Name of the Kerberos principal | `airflow` |
+| `kerberos.reinitFrequency` | Frequency of reinitialization of the Kerberos token | `3600` |
+| `kerberos.confg` | Content of the configuration file for kerberos (might be templated using Helm templates) | `<see values.yaml>` |
| `workers.replicas` | Replica count for Celery workers (if applicable) | `1` |
| `workers.keda.enabled` | Enable KEDA autoscaling features | `false` |
| `workers.keda.pollingInverval` | How often KEDA should poll the backend database for metrics in seconds | `5` |
| `workers.keda.cooldownPeriod` | How often KEDA should wait before scaling down in seconds | `30` |
| `workers.keda.maxReplicaCount` | Maximum number of Celery workers KEDA can scale to | `10` |
+| `workers.kerberosSideCar.enabled` | Enable Kerberos sidecar for the worker | `false` |
| `workers.persistence.enabled` | Enable log persistence in workers via StatefulSet | `false` |
| `workers.persistence.size` | Size of worker volumes if enabled | `100Gi` |
| `workers.persistence.storageClassName` | StorageClass worker volumes should use if enabled | `default` |
@@ -196,8 +204,8 @@ The following tables lists the configurable parameters of the Airflow chart and
| `webserver.resources.requests.cpu` | CPU Request of webserver | `~` |
| `webserver.resources.requests.memory` | Memory Request of webserver | `~` |
| `webserver.defaultUser` | Optional default airflow user information | `{}` |
-| `dags.persistence.*` | Dag persistence configutation | Please refer to `values.yaml` |
-| `dags.gitSync.*` | Git sync configuration | Please refer to `values.yaml` |
+| `dags.persistence.*` | Dag persistence configuration | Please refer to `values.yaml` |
+| `dags.gitSync.*` | Git sync configuration | Please refer to `values.yaml` |
Specify each parameter using the `--set key=value[,key=value]` argument to `helm install`. For example,
diff --git a/chart/templates/_helpers.yaml b/chart/templates/_helpers.yaml
index 195d484..5d3ae73 100644
--- a/chart/templates/_helpers.yaml
+++ b/chart/templates/_helpers.yaml
@@ -235,6 +235,14 @@
{{ default (printf "%s-elasticsearch" .Release.Name) .Values.elasticsearch.secretName }}
{{- end }}
+{{ define "kerberos_keytab_secret" -}}
+{{ .Release.Name }}-kerberos-keytab
+{{- end }}
+
+{{ define "kerberos_ccache_path" -}}
+{{ printf "%s/%s" .Values.kerberos.ccacheMountPath .Values.kerberos.ccacheFileName }}
+{{- end }}
+
{{ define "pgbouncer_config" }}
{{- $pgMetadataHost := .Values.data.metadataConnection.host | default (printf "%s-%s.%s.svc.cluster.local" .Release.Name "postgresql" .Release.Namespace) }}
{{- $pgResultBackendHost := .Values.data.resultBackendConnection.host | default (printf "%s-%s.%s.svc.cluster.local" .Release.Name "postgresql" .Release.Namespace) }}
@@ -265,6 +273,10 @@ log_connections = {{ .Values.pgbouncer.logConnections }}
{{ (printf "%s/logs" .Values.airflowHome) | quote }}
{{- end }}
+{{ define "airflow_logs_no_quote" -}}
+{{ (printf "%s/logs" .Values.airflowHome) }}
+{{- end }}
+
{{ define "airflow_dags" -}}
{{- if .Values.dags.gitSync.enabled -}}
{{ (printf "%s/dags/%s/%s" .Values.airflowHome .Values.dags.gitSync.dest .Values.dags.gitSync.subPath ) }}
diff --git a/chart/templates/configmap.yaml b/chart/templates/configmap.yaml
index d5b4b08..b5bc656 100644
--- a/chart/templates/configmap.yaml
+++ b/chart/templates/configmap.yaml
@@ -62,4 +62,8 @@ data:
{{- else }}
{{ tpl (.Files.Get "files/pod-template-file.kubernetes-helm-yaml") . | nindent 4 }}
{{- end }}
+{{- if .Values.kerberos.enabled }}
+ krb5.conf: |
+ {{ tpl .Values.kerberos.config . | nindent 4 }}
+{{- end }}
{{- end }}
diff --git a/chart/templates/workers/worker-deployment.yaml b/chart/templates/workers/worker-deployment.yaml
index 23d2255..fe07e20 100644
--- a/chart/templates/workers/worker-deployment.yaml
+++ b/chart/templates/workers/worker-deployment.yaml
@@ -124,6 +124,15 @@ spec:
mountPath: {{ template "airflow_config_path" . }}
subPath: airflow.cfg
readOnly: true
+ {{- if .Values.workers.kerberosSidecar.enabled }}
+ - name: config
+ mountPath: {{ .Values.kerberos.configPath | quote }}
+ subPath: krb5.conf
+ readOnly: true
+ - name: kerberos-ccache
+ mountPath: {{ .Values.kerberos.ccacheMountPath | quote }}
+ readOnly: true
+ {{- end }}
{{- if .Values.scheduler.airflowLocalSettings }}
- name: config
mountPath: {{ template "airflow_local_setting_path" . }}
@@ -145,10 +154,62 @@ spec:
- name: logs
mountPath: {{ template "airflow_logs" . }}
{{- end }}
+ {{- if .Values.workers.kerberosSidecar.enabled }}
+ - name: KRB5_CONFIG
+ value: {{ .Values.kerberos.configPath | quote }}
+ - name: KRB5CCNAME
+ value: {{ include "kerberos_ccache_path" . | quote }}
+ {{- end }}
+ {{- if .Values.workers.kerberosSidecar.enabled }}
+ - name: worker-kerberos
+ image: {{ template "airflow_image" . }}
+ imagePullPolicy: {{ .Values.images.airflow.pullPolicy }}
+ args: ["kerberos"]
+ resources:
+ {{ toYaml .Values.workers.resources | indent 12 }}
+ volumeMounts:
+ - name: logs
+ mountPath: {{ template "airflow_logs" . }}
+ - name: config
+ mountPath: {{ template "airflow_config_path" . }}
+ subPath: airflow.cfg
+ readOnly: true
+ - name: config
+ mountPath: {{ .Values.kerberos.configPath | quote }}
+ subPath: krb5.conf
+ readOnly: true
+ {{- if .Values.scheduler.airflowLocalSettings }}
+ - name: config
+ mountPath: {{ template "airflow_local_setting_path" . }}
+ subPath: airflow_local_settings.py
+ readOnly: true
+ {{- end }}
+ - name: kerberos-keytab
+ subPath: "kerberos.keytab"
+ mountPath: {{ .Values.kerberos.keytabPath | quote }}
+ readOnly: true
+ - name: kerberos-ccache
+ mountPath: {{ .Values.kerberos.ccacheMountPath | quote }}
+ readOnly: false
+ env:
+ - name: KRB5_CONFIG
+ value: {{ .Values.kerberos.configPath | quote }}
+ - name: KRB5CCNAME
+ value: {{ include "kerberos_ccache_path" . | quote }}
+ {{- include "custom_airflow_environment" . | indent 10 }}
+ {{- include "standard_airflow_environment" . | indent 10 }}
+ {{- end }}
volumes:
+ - name: kerberos-keytab
+ secret:
+ secretName: {{ include "kerberos_keytab_secret" . | quote }}
- name: config
configMap:
name: {{ template "airflow_config" . }}
+ {{- if .Values.kerberos.enabled }}
+ - name: kerberos-ccache
+ emptyDir: {}
+ {{- end }}
{{- if .Values.dags.persistence.enabled }}
- name: dags
persistentVolumeClaim:
diff --git a/chart/values.yaml b/chart/values.yaml
index c0b9ff5..513dc47 100644
--- a/chart/values.yaml
+++ b/chart/values.yaml
@@ -128,6 +128,59 @@ data:
fernetKey: ~
fernetKeySecretName: ~
+
+# In order to use kerberos you need to create secret containing the keytab file
+# The secret name should follow naming convention of the application where resources are
+# name {{ .Release-name }}-<POSTFIX>. In case of the keytab file, the postfix is "kerberos-keytab"
+# So if your release is named "my-release" the name of the secret should be "my-release-kerberos-keytab"
+#
+# The Keytab content should be available in the "kerberos.keytab" key of the secret.
+#
+# apiVersion: v1
+# kind: Secret
+# data:
+# kerberos.keytab: <base64_encoded keytab file content>
+# type: Opaque
+#
+#
+# If you have such keytab file you can do it with similar
+#
+# kubectl create secret generic {{ .Release.name }}-kerberos-keytab --from-file=kerberos.keytab
+#
+kerberos:
+ enabled: false
+ ccacheMountPath: '/var/kerberos-ccache'
+ ccacheFileName: 'cache'
+ configPath: '/etc/krb5.conf'
+ keytabPath: '/etc/airflow.keytab'
+ principal: 'airflow@FOO.COM'
+ reinitFrequency: 3600
+ config: |
+ # This is an example config showing how you can use templating and how "example" config
+ # might look like. It works with the test kerberos server that we are using during integration
+ # testing at Apache Airflow (see `scripts/ci/docker-compose/integration-kerberos.yml` but in
+ # order to make it production-ready you must replace it with your own configuration that
+ # Matches your kerberos deployment. Administrators of your Kerberos instance should
+ # provide the right configuration.
+
+ [logging]
+ default = "FILE:{{ template "airflow_logs_no_quote" . }}/kerberos_libs.log"
+ kdc = "FILE:{{ template "airflow_logs_no_quote" . }}/kerberos_kdc.log"
+ admin_server = "FILE:{{ template "airflow_logs_no_quote" . }}/kadmind.log"
+
+ [libdefaults]
+ default_realm = FOO.COM
+ ticket_lifetime = 10h
+ renew_lifetime = 7d
+ forwardable = true
+
+ [realms]
+ FOO.COM = {
+ kdc = kdc-server.foo.com
+ admin_server = admin_server.foo.com
+ }
+
+
# Airflow Worker Config
workers:
# Number of airflow celery workers in StatefulSet
@@ -161,6 +214,10 @@ workers:
# of local-path provisioner.
fixPermissions: false
+ kerberosSidecar:
+ # Enable kerberos sidecar
+ enabled: false
+
resources: {}
# limits:
# cpu: 100m
@@ -495,6 +552,12 @@ config:
timeout: 30
retry_timeout: 'True'
+ kerberos:
+ keytab: '{{ .Values.kerberos.keytabPath }}'
+ reinit_frequency: '{{ .Values.kerberos.reinitFrequency }}'
+ principal: '{{ .Values.kerberos.principal }}'
+ ccache: '{{ .Values.kerberos.ccacheMountPath }}/{{ .Values.kerberos.ccacheFileName }}'
+
kubernetes:
namespace: '{{ .Release.Namespace }}'
airflow_configmap: '{{ include "airflow_config" . }}'
[airflow] 13/32: Create job for airflow migrations (#11533)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 3f74b178b4c6a9607198ca55cac20e069e0af899
Author: Daniel Imberman <da...@gmail.com>
AuthorDate: Thu Oct 15 11:10:14 2020 -0700
Create job for airflow migrations (#11533)
Creating airflow migrations should run seperately from the user creation
job, as many users might not want to create users on deployment.
(cherry picked from commit f7e01ab29339080c77431a1fc5e4403ba2d80326)
---
chart/templates/create-user-job.yaml | 9 +----
...ate-user-job.yaml => migrate-database-job.yaml} | 43 ++++++----------------
chart/tests/migrate-database-job_test.yaml | 27 ++++++++++++++
3 files changed, 40 insertions(+), 39 deletions(-)
diff --git a/chart/templates/create-user-job.yaml b/chart/templates/create-user-job.yaml
index 642e36f..0f76252 100644
--- a/chart/templates/create-user-job.yaml
+++ b/chart/templates/create-user-job.yaml
@@ -34,6 +34,7 @@ metadata:
{{- end }}
annotations:
"helm.sh/hook": post-install
+ "helm.sh/hook-weight": "2"
"helm.sh/hook-delete-policy": hook-succeeded
spec:
template:
@@ -51,14 +52,6 @@ spec:
tolerations:
{{ toYaml .Values.tolerations | indent 8 }}
containers:
- - name: run-airflow-migrations
- image: {{ template "airflow_image" . }}
- imagePullPolicy: {{ .Values.images.airflow.pullPolicy }}
- # Support running against 1.10.x and 2.0.0dev/master
- args: ["bash", "-c", "airflow upgradedb || airflow db upgrade"]
- env:
- {{- include "custom_airflow_environment" . | indent 10 }}
- {{- include "standard_airflow_environment" . | indent 10 }}
- name: create-user
image: {{ template "airflow_image" . }}
imagePullPolicy: {{ .Values.images.airflow.pullPolicy }}
diff --git a/chart/templates/create-user-job.yaml b/chart/templates/migrate-database-job.yaml
similarity index 65%
copy from chart/templates/create-user-job.yaml
copy to chart/templates/migrate-database-job.yaml
index 642e36f..37a9b2d 100644
--- a/chart/templates/create-user-job.yaml
+++ b/chart/templates/migrate-database-job.yaml
@@ -16,16 +16,15 @@
# under the License.
################################
-## Airflow Create User Job
+## Airflow Run Migrations
#################################
-{{- if .Values.webserver.defaultUser.enabled }}
apiVersion: batch/v1
kind: Job
metadata:
- name: {{ .Release.Name }}-create-user
+ name: {{ .Release.Name }}-run-airflow-migrations
labels:
tier: airflow
- component: create-user-job
+ component: run-airflow-migrations
release: {{ .Release.Name }}
chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
heritage: {{ .Release.Service }}
@@ -33,16 +32,19 @@ metadata:
{{ toYaml . | indent 4 }}
{{- end }}
annotations:
- "helm.sh/hook": post-install
+ "helm.sh/hook": post-install,post-upgrade
+ "helm.sh/hook-weight": "1"
"helm.sh/hook-delete-policy": hook-succeeded
spec:
template:
metadata:
labels:
tier: airflow
- component: create-user-job
+ component: run-airflow-migrations
release: {{ .Release.Name }}
spec:
+ securityContext:
+ runAsUser: {{ .Values.uid }}
restartPolicy: OnFailure
nodeSelector:
{{ toYaml .Values.nodeSelector | indent 8 }}
@@ -50,6 +52,10 @@ spec:
{{ toYaml .Values.affinity | indent 8 }}
tolerations:
{{ toYaml .Values.tolerations | indent 8 }}
+ {{- if or .Values.registry.secretName .Values.registry.connection }}
+ imagePullSecrets:
+ - name: {{ template "registry_secret" . }}
+ {{- end }}
containers:
- name: run-airflow-migrations
image: {{ template "airflow_image" . }}
@@ -59,30 +65,6 @@ spec:
env:
{{- include "custom_airflow_environment" . | indent 10 }}
{{- include "standard_airflow_environment" . | indent 10 }}
- - name: create-user
- image: {{ template "airflow_image" . }}
- imagePullPolicy: {{ .Values.images.airflow.pullPolicy }}
- args:
- - "bash"
- - "-c"
- # Support running against 1.10.x and 2.0.0dev/master
- - 'airflow create_user "$@" || airflow users create "$@"'
- - --
- - "-r"
- - {{ .Values.webserver.defaultUser.role }}
- - "-u"
- - {{ .Values.webserver.defaultUser.username }}
- - "-e"
- - {{ .Values.webserver.defaultUser.email }}
- - "-f"
- - {{ .Values.webserver.defaultUser.firstName }}
- - "-l"
- - {{ .Values.webserver.defaultUser.lastName }}
- - "-p"
- - {{ .Values.webserver.defaultUser.password }}
- env:
- {{- include "custom_airflow_environment" . | indent 10 }}
- {{- include "standard_airflow_environment" . | indent 10 }}
volumeMounts:
- name: config
mountPath: {{ template "airflow_config_path" . }}
@@ -92,4 +74,3 @@ spec:
- name: config
configMap:
name: {{ template "airflow_config" . }}
-{{- end }}
diff --git a/chart/tests/migrate-database-job_test.yaml b/chart/tests/migrate-database-job_test.yaml
new file mode 100644
index 0000000..117b7a7
--- /dev/null
+++ b/chart/tests/migrate-database-job_test.yaml
@@ -0,0 +1,27 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+templates:
+ - migrate-database-job.yaml
+tests:
+ - it: should run by default
+ asserts:
+ - isKind:
+ of: Job
+ - equal:
+ path: spec.template.spec.containers[0].name
+ value: run-airflow-migrations
[airflow] 12/32: Add missing values entries to Parameters in
chart/README.md (#11477)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit e85cc77e3ac88604d2ae3ecc7e8d1fca1ddcd07b
Author: Kaxil Naik <ka...@gmail.com>
AuthorDate: Tue Oct 13 14:21:47 2020 +0100
Add missing values entries to Parameters in chart/README.md (#11477)
(cherry picked from commit cb4cd91837c4b974b210289113a92074d26a01c5)
---
chart/README.md | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/chart/README.md b/chart/README.md
index 41a53ad..7bbe63c 100644
--- a/chart/README.md
+++ b/chart/README.md
@@ -185,6 +185,9 @@ The following tables lists the configurable parameters of the Airflow chart and
| `workers.resources.requests.memory` | Memory Request of workers | `~` |
| `workers.terminationGracePeriodSeconds` | How long Kubernetes should wait for Celery workers to gracefully drain before force killing | `600` |
| `workers.safeToEvict` | Allow Kubernetes to evict worker pods if needed (node downscaling) | `true` |
+| `workers.serviceAccountAnnotations` | Annotations to add to worker kubernetes service account | `{}` |
+| `workers.extraVolumes` | Mount additional volumes into worker | `[]` |
+| `workers.extraVolumeMounts` | Mount additional volumes into worker | `[]` |
| `scheduler.podDisruptionBudget.enabled` | Enable PDB on Airflow scheduler | `false` |
| `scheduler.podDisruptionBudget.config.maxUnavailable` | MaxUnavailable pods for scheduler | `1` |
| `scheduler.replicas` | # of parallel schedulers (Airflow 2.0 using Mysql 8+ or Postgres only) | `1` |
@@ -194,6 +197,9 @@ The following tables lists the configurable parameters of the Airflow chart and
| `scheduler.resources.requests.memory` | Memory Request of scheduler | `~` |
| `scheduler.airflowLocalSettings` | Custom Airflow local settings python file | `~` |
| `scheduler.safeToEvict` | Allow Kubernetes to evict scheduler pods if needed (node downscaling) | `true` |
+| `scheduler.serviceAccountAnnotations` | Annotations to add to scheduler kubernetes service account | `{}` |
+| `scheduler.extraVolumes` | Mount additional volumes into scheduler | `[]` |
+| `scheduler.extraVolumeMounts` | Mount additional volumes into scheduler | `[]` |
| `webserver.livenessProbe.initialDelaySeconds` | Webserver LivenessProbe initial delay | `15` |
| `webserver.livenessProbe.timeoutSeconds` | Webserver LivenessProbe timeout seconds | `30` |
| `webserver.livenessProbe.failureThreshold` | Webserver LivenessProbe failure threshold | `20` |
[airflow] 19/32: Validate airflow chart values.yaml &
values.schema.json (#11990)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 6a12758b0e9503977da1b71fde1ebc2a4691622a
Author: Florent Chehab <fc...@meilleursagents.com>
AuthorDate: Sat Oct 31 18:19:32 2020 +0100
Validate airflow chart values.yaml & values.schema.json (#11990)
* Correct type for multiNamespaceMode chart value
* Updated values.schema.json to reflect the latest change and to be stricter
* Fixed current test
* Added a test to validate the values file against the schema
(cherry picked from commit e238b882a8568532829be80e96e54856d7a0018d)
---
chart/README.md | 2 +-
chart/tests/test_basic_helm_chart.py | 4 +-
chart/tests/test_chart_quality.py | 42 +++++++
chart/values.schema.json | 220 ++++++++++++++++++++++++-----------
chart/values.yaml | 79 ++++++-------
5 files changed, 229 insertions(+), 118 deletions(-)
diff --git a/chart/README.md b/chart/README.md
index 7bbe63c..df8ea5d 100644
--- a/chart/README.md
+++ b/chart/README.md
@@ -169,7 +169,7 @@ The following tables lists the configurable parameters of the Airflow chart and
| `kerberos.keytabPath` | Path for the Kerberos keytab file | `/etc/airflow.keytab` |
| `kerberos.principal` | Name of the Kerberos principal | `airflow` |
| `kerberos.reinitFrequency` | Frequency of reinitialization of the Kerberos token | `3600` |
-| `kerberos.confg` | Content of the configuration file for kerberos (might be templated using Helm templates) | `<see values.yaml>` |
+| `kerberos.config` | Content of the configuration file for kerberos (might be templated using Helm templates) | `<see values.yaml>` |
| `workers.replicas` | Replica count for Celery workers (if applicable) | `1` |
| `workers.keda.enabled` | Enable KEDA autoscaling features | `false` |
| `workers.keda.pollingInverval` | How often KEDA should poll the backend database for metrics in seconds | `5` |
diff --git a/chart/tests/test_basic_helm_chart.py b/chart/tests/test_basic_helm_chart.py
index dcf20cf..e535ac9 100644
--- a/chart/tests/test_basic_helm_chart.py
+++ b/chart/tests/test_basic_helm_chart.py
@@ -38,8 +38,8 @@ class TestBaseChartTest(unittest.TestCase):
('Secret', 'TEST-BASIC-airflow-metadata'),
('Secret', 'TEST-BASIC-airflow-result-backend'),
('ConfigMap', 'TEST-BASIC-airflow-config'),
- ('ClusterRole', 'TEST-BASIC-pod-launcher-role'),
- ('ClusterRoleBinding', 'TEST-BASIC-pod-launcher-rolebinding'),
+ ('Role', 'TEST-BASIC-pod-launcher-role'),
+ ('RoleBinding', 'TEST-BASIC-pod-launcher-rolebinding'),
('Service', 'TEST-BASIC-postgresql-headless'),
('Service', 'TEST-BASIC-postgresql'),
('Service', 'TEST-BASIC-statsd'),
diff --git a/chart/tests/test_chart_quality.py b/chart/tests/test_chart_quality.py
new file mode 100644
index 0000000..3898941
--- /dev/null
+++ b/chart/tests/test_chart_quality.py
@@ -0,0 +1,42 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import json
+import os
+import unittest
+import yaml
+
+from jsonschema import validate
+
+
+CHART_FOLDER = os.path.dirname(os.path.dirname(__file__))
+
+
+class ChartQualityTest(unittest.TestCase):
+ def test_values_validate_schema(self):
+ with open(os.path.join(CHART_FOLDER, "values.yaml"), "r") as f:
+ values = yaml.safe_load(f)
+ with open(os.path.join(CHART_FOLDER, "values.schema.json"), "r") as f:
+ schema = json.load(f)
+
+ # Add extra restrictions just for the tests to make sure
+ # we don't forget to update the schema if we add a new property
+ schema["additionalProperties"] = False
+ schema["minProperties"] = len(schema["properties"].keys())
+
+ # shouldn't raise
+ validate(instance=values, schema=schema)
diff --git a/chart/values.schema.json b/chart/values.schema.json
index 4395372..2f25892 100644
--- a/chart/values.schema.json
+++ b/chart/values.schema.json
@@ -23,10 +23,6 @@
"description": "Default airflow tag to deploy.",
"type": "string"
},
- "multi_namespaceMode": {
- "description": "Whether the KubernetesExecutor can launch workers in multiple namespaces",
- "type": "boolean"
- },
"nodeSelector": {
"description": "Select certain nodes for airflow pods.",
"type": "object",
@@ -60,6 +56,7 @@
"web": {
"description": "Configuration for the Ingress of the web Service.",
"type": "object",
+ "additionalProperties": false,
"properties": {
"annotations": {
"description": "Annotations for the web Ingress.",
@@ -76,6 +73,7 @@
"tls": {
"description": "Configuration for web Ingress TLS.",
"type": "object",
+ "additionalProperties": false,
"properties": {
"enabled": {
"description": "Enable TLS termination for the web Ingress.",
@@ -100,6 +98,7 @@
"flower": {
"description": "Configuration for the Ingress of the flower Service.",
"type": "object",
+ "additionalProperties": false,
"properties": {
"annotations": {
"description": "Annotations for the flower Ingress.",
@@ -116,6 +115,7 @@
"tls": {
"description": "Configuration for flower Ingress TLS.",
"type": "object",
+ "additionalProperties": false,
"properties": {
"enabled": {
"description": "Enable TLS termination for the flower Ingress.",
@@ -142,6 +142,7 @@
"networkPolicies": {
"description": "Network policy configuration.",
"type": "object",
+ "additionalProperties": false,
"properties": {
"enabled": {
"description": "Enabled network policies.",
@@ -168,6 +169,7 @@
"images": {
"description": "Images.",
"type": "object",
+ "additionalProperties": false,
"properties": {
"airflow": {
"description": "Configuration of the airflow image.",
@@ -193,6 +195,30 @@
}
}
},
+ "pod_template": {
+ "description": "Configuration of the pod_template image.",
+ "type": "object",
+ "properties": {
+ "repository": {
+ "description": "The pod_template image repository.",
+ "type": [
+ "string",
+ "null"
+ ]
+ },
+ "tag": {
+ "description": "The pod_template image tag.",
+ "type": [
+ "string",
+ "null"
+ ]
+ },
+ "pullPolicy": {
+ "description": "The pod_template image pull policy.",
+ "type": "string"
+ }
+ }
+ },
"flower": {
"description": "Configuration of the flower image.",
"type": "object",
@@ -320,6 +346,7 @@
"data": {
"description": "Airflow database configuration.",
"type": "object",
+ "additionalProperties": false,
"properties": {
"metadataSecretName": {
"description": "Metadata connection string secret.",
@@ -338,6 +365,7 @@
"metadataConnection": {
"description": "Metadata connection configuration.",
"type": "object",
+ "additionalProperties": false,
"properties": {
"user": {
"description": "The database user.",
@@ -371,6 +399,7 @@
"resultBackendConnection": {
"description": "Result backend connection configuration.",
"type": "object",
+ "additionalProperties": false,
"properties": {
"user": {
"description": "The database user.",
@@ -417,9 +446,48 @@
"null"
]
},
+ "kerberos": {
+ "description": "Kerberos configurations for airflow",
+ "type": "object",
+ "properties": {
+ "enabled": {
+ "description": "Enable kerberos.",
+ "type": "boolean"
+ },
+ "ccacheMountPath": {
+ "description": "Path to mount shared volume for kerberos credentials cache.",
+ "type": "string"
+ },
+ "ccacheFileName": {
+ "description": "Name for kerberos credentials cache file.",
+ "type": "string"
+ },
+ "configPath":{
+ "description": "Path to mount krb5.conf kerberos configuration file.",
+ "type": "string"
+ },
+ "keytabPath":{
+ "description": "Path to mount the keytab for refreshing credentials in the kerberos sidecar.",
+ "type": "string"
+ },
+ "principal":{
+ "description": "Principal to use when refreshing kerberos credentials.",
+ "type": "string"
+ },
+ "reinitFrequency": {
+ "description": "How often (in seconds) airflow kerberos will reinitialize the credentials cache.",
+ "type": "integer"
+ },
+ "config": {
+ "description": "Contents of krb5.conf.",
+ "type": "string"
+ }
+ }
+ },
"workers": {
"description": "Airflow Worker configuration.",
"type": "object",
+ "additionalProperties": false,
"properties": {
"replicas": {
"description": "Number of airflow celery workers in StatefulSet.",
@@ -428,6 +496,7 @@
"keda": {
"description": "KEDA configuration.",
"type": "object",
+ "additionalProperties": false,
"properties": {
"enabled": {
"description": "Allow KEDA autoscaling. `Persistence.enabled` must be set to false to use KEDA.",
@@ -453,6 +522,7 @@
"persistence": {
"description": "Persistence configuration.",
"type": "object",
+ "additionalProperties": false,
"properties": {
"enabled": {
"description": "Enable persistent volumes.",
@@ -475,6 +545,16 @@
}
}
},
+ "kerberosSidecar": {
+ "type": "object",
+ "additionalProperties": false,
+ "properties": {
+ "enabled": {
+ "description": "Enable Kerberos sidecar for the worker.",
+ "type": "boolean"
+ }
+ }
+ },
"resources": {
"type": "object"
},
@@ -489,20 +569,21 @@
"serviceAccountAnnotations": {
"description": "Annotations to add to the worker kubernetes service account.",
"type": "object"
+ },
+ "extraVolumes": {
+ "description": "Mount additional volumes into workers.",
+ "type": "array"
+ },
+ "extraVolumeMounts": {
+ "description": "Mount additional volumes into workers.",
+ "type": "array"
}
- },
- "extraVolumes": {
- "description": "Mount additional volumes into workers.",
- "type": "array"
- },
- "extraVolumeMounts": {
- "description": "Mount additional volumes into workers.",
- "type": "array"
- }
+ }
},
"scheduler": {
"description": "Airflow scheduler settings.",
"type": "object",
+ "additionalProperties": false,
"properties": {
"replicas": {
"description": "Airflow 2.0 allows users to run multiple schedulers. This feature is only recommended for Mysql 8+ and postgres",
@@ -511,6 +592,7 @@
"podDisruptionBudget": {
"description": "Scheduler pod disruption budget.",
"type": "object",
+ "additionalProperties": false,
"properties": {
"enabled": {
"description": "Enable pod disruption budget.",
@@ -519,6 +601,7 @@
"config": {
"description": "Disruption budget configuration.",
"type": "object",
+ "additionalProperties": false,
"properties": {
"maxUnavailable": {
"description": "Max unavailable pods for scheduler.",
@@ -545,24 +628,26 @@
"serviceAccountAnnotations": {
"description": "Annotations to add to the scheduler kubernetes service account.",
"type": "object"
+ },
+ "extraVolumes": {
+ "description": "Mount additional volumes into scheduler.",
+ "type": "array"
+ },
+ "extraVolumeMounts": {
+ "description": "Mount additional volumes into scheduler.",
+ "type": "array"
}
- },
- "extraVolumes": {
- "description": "Mount additional volumes into scheduler.",
- "type": "array"
- },
- "extraVolumeMounts": {
- "description": "Mount additional volumes into scheduler.",
- "type": "array"
- }
+ }
},
"webserver": {
"description": "Airflow webserver settings.",
"type": "object",
+ "additionalProperties": false,
"properties": {
"livenessProbe": {
"description": "Liveness probe configuration.",
"type": "object",
+ "additionalProperties": false,
"properties": {
"initialDelaySeconds": {
"description": "Webserver Liveness probe initial delay.",
@@ -585,6 +670,7 @@
"readinessProbe": {
"description": "Readiness probe configuration.",
"type": "object",
+ "additionalProperties": false,
"properties": {
"initialDelaySeconds": {
"description": "Webserver Readiness probe initial delay.",
@@ -618,6 +704,7 @@
"defaultUser": {
"description": "Optional default airflow user information",
"type": "object",
+ "additionalProperties": false,
"properties": {
"enabled": {
"description": "Enable default user creation.",
@@ -667,6 +754,7 @@
"service": {
"description": "Webserver service configuration.",
"type": "object",
+ "additionalProperties": false,
"properties": {
"type": {
"description": "Webserver service type.",
@@ -687,6 +775,7 @@
"flower": {
"description": "Flower settings.",
"type": "object",
+ "additionalProperties": false,
"properties": {
"extraNetworkPolicies": {
"description": "Additional network policies as needed.",
@@ -698,6 +787,7 @@
"service": {
"description": "Flower service configuration.",
"type": "object",
+ "additionalProperties": false,
"properties": {
"type": {
"description": "Flower service type.",
@@ -710,6 +800,7 @@
"statsd": {
"description": "Statsd settings.",
"type": "object",
+ "additionalProperties": false,
"properties": {
"enabled": {
"description": "Enable statsd.",
@@ -725,6 +816,7 @@
"service": {
"description": "Statsd service configuration.",
"type": "object",
+ "additionalProperties": false,
"properties": {
"extraAnnotations": {
"description": "Extra annotations for the statsd service.",
@@ -737,6 +829,7 @@
"pgbouncer": {
"description": "Pgbouncer settings.",
"type": "object",
+ "additionalProperties": false,
"properties": {
"enabled": {
"description": "Enable pgbouncer.",
@@ -761,6 +854,7 @@
"podDisruptionBudget": {
"description": "Pgbouner pod disruption budget.",
"type": "object",
+ "additionalProperties": false,
"properties": {
"enabled": {
"description": "Enabled pod distribution budget.",
@@ -769,6 +863,7 @@
"config": {
"description": "Pod distribution configuration.",
"type": "object",
+ "additionalProperties": false,
"properties": {
"maxUnavailable": {
"description": "Max unavailable pods for pgbouncer.",
@@ -784,6 +879,7 @@
"service": {
"description": "Pgbouncer service configuration.",
"type": "object",
+ "additionalProperties": false,
"properties": {
"extraAnnotations": {
"description": "Extra annotations for the pgbouncer service.",
@@ -835,6 +931,7 @@
"redis": {
"description": "",
"type": "object",
+ "additionalProperties": false,
"properties": {
"terminationGracePeriodSeconds": {
"description": "Grace period for tasks to finish after SIGTERM is sent from Kubernetes.",
@@ -843,6 +940,7 @@
"persistence": {
"description": "Persistence configuration.",
"type": "object",
+ "additionalProperties": false,
"properties": {
"enabled": {
"description": "Enable persistent volumes.",
@@ -861,16 +959,6 @@
}
}
},
- "kerberosSidecar": {
- "description": "Run a side car in each worker pod to refresh Kerberos ccache with `airflow kerberos` according to the airflow security configuration",
- "type": "object",
- "properties": {
- "enabled": {
- "description": "Enable Kerberos side car on worker pods.",
- "type": "boolean"
- }
- }
- },
"resources": {
"type": "object"
},
@@ -904,6 +992,7 @@
"registry": {
"description": "Auth secret for a private registry. This is used if pulling airflow images from a private registry.",
"type": "object",
+ "additionalProperties": false,
"properties": {
"secretName": {
"description": "Registry connection string secret.",
@@ -921,6 +1010,7 @@
"elasticsearch": {
"description": "Elasticsearch logging configuration.",
"type": "object",
+ "additionalProperties": false,
"properties": {
"enabled": {
"description": "Enable elasticsearch task logging.",
@@ -942,6 +1032,7 @@
"ports": {
"description": "All ports used by chart.",
"type": "object",
+ "additionalProperties": false,
"properties": {
"flowerUI": {
"description": "Flower UI port.",
@@ -988,6 +1079,7 @@
"cleanup": {
"description": "This runs as a CronJob to cleanup old pods.",
"type": "object",
+ "additionalProperties": false,
"properties": {
"enabled": {
"description": "Enable cleanup.",
@@ -1032,13 +1124,22 @@
}
}
},
+ "multiNamespaceMode": {
+ "description": "Whether the KubernetesExecutor can launch workers and pods in multiple namespaces",
+ "type": "boolean"
+ },
+ "podTemplate": {
+ "description": "TODO ; also add type if you know it"
+ },
"dags": {
"description": "DAGs settings.",
"type": "object",
+ "additionalProperties": false,
"properties": {
"persistence": {
"description": "Persistence configuration.",
"type": "object",
+ "additionalProperties": false,
"properties": {
"enabled": {
"description": "Enable persistent volume for storing dags.",
@@ -1071,6 +1172,7 @@
"gitSync": {
"description": "Git sync settings.",
"type": "object",
+ "additionalProperties": false,
"properties": {
"enabled": {
"description": "Enable Git sync.",
@@ -1115,48 +1217,26 @@
"containerName": {
"description": "Git sync container name.",
"type": "string"
+ },
+ "uid": {
+ "description": "Git sync container run as user parameter.",
+ "type": "integer"
+ },
+ "credentialsSecret": {
+ "description": "TODO",
+ "type": ["string", "null"]
+ },
+ "sshKeySecret": {
+ "description": "TODO",
+ "type": ["string", "null"]
+ },
+ "knownHosts": {
+ "description": "TODO",
+ "type": ["string", "null"]
}
}
}
}
- },
- "kerberos": {
- "description": "Kerberos configurations for airflow",
- "type": "object",
- "properties": {
- "enabled": {
- "description": "Enable kerberos.",
- "type": "boolean"
- },
- "ccacheMountPath": {
- "description": "Path to mount shared volume for kerberos credentials cache.",
- "type": "string"
- },
- "ccacheFileName": {
- "description": "Name for kerberos credentials cache file.",
- "type": "string"
- },
- "configPath":{
- "description": "Path to mount krb5.conf kerberos configuration file.",
- "type": "string"
- },
- "keytabPath":{
- "description": "Path to mount the keytab for refreshing credentials in the kerberos sidecar.",
- "type": "string"
- },
- "principal":{
- "description": "Principal to use when refreshing kerberos credentials.",
- "type": "string"
- },
- "reinitFrequency": {
- "description": "How often (in seconds) airflow kerberos will reinitialize the credentials cache.",
- "type": "integer"
- },
- "config": {
- "description": "Contents of krb5.conf.",
- "type": "string"
- }
- }
}
}
}
diff --git a/chart/values.yaml b/chart/values.yaml
index 54a78d6..4855214 100644
--- a/chart/values.yaml
+++ b/chart/values.yaml
@@ -112,7 +112,7 @@ rbacEnabled: true
# Options: SequentialExecutor, LocalExecutor, CeleryExecutor, KubernetesExecutor
executor: "KubernetesExecutor"
-# If this is true and using LocalExecutor/SequentialExecutor/KubernetesExecutor, the scheudler's
+# If this is true and using LocalExecutor/SequentialExecutor/KubernetesExecutor, the scheduler's
# service account will have access to communicate with the api-server and launch pods.
# If this is true and using the CeleryExecutor, the workers will be able to launch pods.
allowPodLaunching: true
@@ -450,13 +450,21 @@ pgbouncer:
config:
maxUnavailable: 1
+ # Limit the resources to pgbouncerExported.
+ # When you specify the resource request the scheduler uses this information to decide which node to place
+ # the Pod on. When you specify a resource limit for a Container, the kubelet enforces those limits so
+ # that the running container is not allowed to use more of that resource than the limit you set.
+ # See: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/
+ # Example:
+ #
+ # resource:
+ # limits:
+ # cpu: 100m
+ # memory: 128Mi
+ # requests:
+ # cpu: 100m
+ # memory: 128Mi
resources: {}
- # limits:
- # cpu: 100m
- # memory: 128Mi
- # requests:
- # cpu: 100m
- # memory: 128Mi
service:
extraAnnotations: {}
@@ -510,11 +518,13 @@ redis:
registry:
secretName: ~
+ # Example:
+ # connection:
+ # user: ~
+ # pass: ~
+ # host: ~
+ # email: ~
connection: {}
- # user: ~
- # pass: ~
- # host: ~
- # email: ~
# Elasticsearch logging configuration
elasticsearch:
@@ -523,11 +533,14 @@ elasticsearch:
# A secret containing the connection
secretName: ~
# Or an object representing the connection
+ # Example:
+ # connection:
+ # user: ~
+ # pass: ~
+ # host: ~
+ # port: ~
connection: {}
- # user: ~
- # pass: ~
- # host: ~
- # port: ~
+
# All ports used by chart
ports:
@@ -563,7 +576,7 @@ postgresql:
#
# Please note that these values are passed through the `tpl` function, so are
# all subject to being rendered as go templates. If you need to include a
-# litera `{{` in a value, it must be expessed like this:
+# literal `{{` in a value, it must be expressed like this:
#
# a: '{{ "{{ not a template }}" }}'
#
@@ -619,38 +632,11 @@ config:
pod_template_file: '{{ include "airflow_pod_template_file" . }}/pod_template_file.yaml'
worker_container_repository: '{{ .Values.images.airflow.repository | default .Values.defaultAirflowRepository }}'
worker_container_tag: '{{ .Values.images.airflow.tag | default .Values.defaultAirflowTag }}'
- worker_container_image_pull_policy: '{{ .Values.images.airflow.pullPolicy }}'
- worker_service_account_name: '{{ .Release.Name }}-worker-serviceaccount'
- image_pull_secrets: '{{ template "registry_secret" . }}'
- dags_in_image: '{{ ternary "False" "True" (or .Values.dags.gitSync.enabled .Values.dags.persistence.enabled) }}'
delete_worker_pods: 'True'
- multi_namespace_mode: '{{ .Values.multiNamespaceMode }}'
- delete_worker_pods_on_failure: 'False'
- run_as_user: '{{ .Values.uid }}'
- fs_group: '{{ .Values.gid }}'
- git_dags_folder_mount_point: '{{- if or .Values.dags.gitSync.enabled .Values.dags.persistence.enabled }}{{ include "airflow_dags_mount_path" . }}{{end}}'
- dags_volume_mount_point: '{{- if or .Values.dags.gitSync.enabled .Values.dags.persistence.enabled }}{{ include "airflow_dags_mount_path" . }}{{ end }}'
- dags_volume_claim: '{{- if .Values.dags.persistence.enabled }}{{ include "airflow_dags_volume_claim" . }}{{ end }}'
- dags_volume_subpath: '{{- if .Values.dags.persistence.enabled }}{{.Values.dags.gitSync.dest }}/{{ .Values.dags.gitSync.subPath }}{{ end }}'
- git_repo: '{{- if and .Values.dags.gitSync.enabled (not .Values.dags.persistence.enabled) }}{{ .Values.dags.gitSync.repo }}{{ end }}'
- git_branch: '{{ .Values.dags.gitSync.branch }}'
- git_sync_rev: '{{ .Values.dags.gitSync.rev }}'
- git_sync_depth: '{{ .Values.dags.gitSync.depth }}'
- git_sync_root: '{{ .Values.dags.gitSync.root }}'
- git_sync_dest: '{{ .Values.dags.gitSync.dest }}'
- git_sync_container_repository: '{{ .Values.dags.gitSync.containerRepository }}'
- git_sync_container_tag: '{{ .Values.dags.gitSync.containerTag }}'
- git_sync_init_container_name: '{{ .Values.dags.gitSync.containerName }}'
- git_sync_run_as_user: '{{ .Values.uid }}'
- git_ssh_known_hosts_configmap_name: '{{- if .Values.dags.gitSync.knownHosts }}{{ include "airflow_config" . }}{{ end }}'
- git_ssh_key_secret_name: '{{- if .Values.dags.gitSync.sshKeySecret }}{{ .Values.dags.gitSync.sshKeySecret }}{{ end }}'
- git_sync_credentials_secret: '{{- if .Values.dags.gitSync.credentialsSecret }}{{ .Values.dags.gitSync.credentialsSecret }}{{ end }}'
- kubernetes_secrets:
- AIRFLOW__CORE__SQL_ALCHEMY_CONN: '{{ printf "%s=connection" (include "airflow_metadata_secret" .) }}'
- AIRFLOW__CORE__FERNET_KEY: '{{ printf "%s=fernet-key" (include "fernet_key_secret" .) }}'
+ multi_namespace_mode: '{{ if .Values.multiNamespaceMode }}True{{ else }}False{{ end }}'
# yamllint enable rule:line-length
-multiNamespaceMode: 'False'
+multiNamespaceMode: false
podTemplate: ~
@@ -695,6 +681,7 @@ dags:
# GIT_SYNC_USERNAME: <base64_encoded_git_username>
# GIT_SYNC_PASSWORD: <base64_encoded_git_password>
# and specify the name of the secret below
+ #
# credentialsSecret: git-credentials
#
#
@@ -710,8 +697,10 @@ dags:
# gitSshKey: <base64_encoded_data>
# and specify the name of the secret below
# sshKeySecret: airflow-ssh-secret
+ #
# If you are using an ssh private key, you can additionally
# specify the content of your known_hosts file, example:
+ #
# knownHosts: |
# <host1>,<ip1> <key1>
# <host2>,<ip2> <key2>
[airflow] 21/32: fix helm chart worker deployment without kerberos
(#11681)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 7a72fb78c7897ef1ac802bc6f638998794089c0c
Author: Florent Chehab <fc...@meilleursagents.com>
AuthorDate: Sat Oct 31 18:48:55 2020 +0100
fix helm chart worker deployment without kerberos (#11681)
Follow up to #11130 : we shouldn't mount the `kerberos-keytab` volume
in the worker deployment if we are not using
kerberos in the first place.
(the previous behavior is breaking the chart)
(cherry picked from commit 4c547180ca63472f60445758988ab815cc66becf)
---
chart/templates/workers/worker-deployment.yaml | 2 ++
chart/tests/test_celery_kubernetes_executor.py | 8 +++----
chart/tests/test_git_sync_worker.py | 8 +++----
chart/tests/test_kerberos.py | 32 ++++++++++++++++++++++++++
4 files changed, 42 insertions(+), 8 deletions(-)
diff --git a/chart/templates/workers/worker-deployment.yaml b/chart/templates/workers/worker-deployment.yaml
index dc2dfb4..53998da 100644
--- a/chart/templates/workers/worker-deployment.yaml
+++ b/chart/templates/workers/worker-deployment.yaml
@@ -207,9 +207,11 @@ spec:
{{- if .Values.workers.extraVolumes }}
{{ toYaml .Values.workers.extraVolumes | indent 8 }}
{{- end }}
+ {{- if .Values.kerberos.enabled }}
- name: kerberos-keytab
secret:
secretName: {{ include "kerberos_keytab_secret" . | quote }}
+ {{- end }}
- name: config
configMap:
name: {{ template "airflow_config" . }}
diff --git a/chart/tests/test_celery_kubernetes_executor.py b/chart/tests/test_celery_kubernetes_executor.py
index 7eae16e..57c3980 100644
--- a/chart/tests/test_celery_kubernetes_executor.py
+++ b/chart/tests/test_celery_kubernetes_executor.py
@@ -31,8 +31,8 @@ class CeleryKubernetesExecutorTest(unittest.TestCase):
show_only=["templates/workers/worker-deployment.yaml"],
)
- self.assertEqual("config", jmespath.search("spec.template.spec.volumes[1].name", docs[0]))
- self.assertEqual("dags", jmespath.search("spec.template.spec.volumes[2].name", docs[0]))
+ self.assertEqual("config", jmespath.search("spec.template.spec.volumes[0].name", docs[0]))
+ self.assertEqual("dags", jmespath.search("spec.template.spec.volumes[1].name", docs[0]))
def test_should_create_a_worker_deployment_with_the_celery_kubernetes_executor(self):
docs = render_chart(
@@ -43,5 +43,5 @@ class CeleryKubernetesExecutorTest(unittest.TestCase):
show_only=["templates/workers/worker-deployment.yaml"],
)
- self.assertEqual("config", jmespath.search("spec.template.spec.volumes[1].name", docs[0]))
- self.assertEqual("dags", jmespath.search("spec.template.spec.volumes[2].name", docs[0]))
+ self.assertEqual("config", jmespath.search("spec.template.spec.volumes[0].name", docs[0]))
+ self.assertEqual("dags", jmespath.search("spec.template.spec.volumes[1].name", docs[0]))
diff --git a/chart/tests/test_git_sync_worker.py b/chart/tests/test_git_sync_worker.py
index 714f385..a70d311 100644
--- a/chart/tests/test_git_sync_worker.py
+++ b/chart/tests/test_git_sync_worker.py
@@ -31,8 +31,8 @@ class GitSyncWorkerTest(unittest.TestCase):
show_only=["templates/workers/worker-deployment.yaml"],
)
- self.assertEqual("config", jmespath.search("spec.template.spec.volumes[1].name", docs[0]))
- self.assertEqual("dags", jmespath.search("spec.template.spec.volumes[2].name", docs[0]))
+ self.assertEqual("config", jmespath.search("spec.template.spec.volumes[0].name", docs[0]))
+ self.assertEqual("dags", jmespath.search("spec.template.spec.volumes[1].name", docs[0]))
def test_should_add_dags_volume_to_the_worker_if_git_sync_is_enabled_and_peristence_is_disabled(self):
docs = render_chart(
@@ -43,8 +43,8 @@ class GitSyncWorkerTest(unittest.TestCase):
show_only=["templates/workers/worker-deployment.yaml"],
)
- self.assertEqual("config", jmespath.search("spec.template.spec.volumes[1].name", docs[0]))
- self.assertEqual("dags", jmespath.search("spec.template.spec.volumes[2].name", docs[0]))
+ self.assertEqual("config", jmespath.search("spec.template.spec.volumes[0].name", docs[0]))
+ self.assertEqual("dags", jmespath.search("spec.template.spec.volumes[1].name", docs[0]))
def test_should_add_git_sync_container_to_worker_if_persistence_is_not_enabled_but_git_sync_is(self):
docs = render_chart(
diff --git a/chart/tests/test_kerberos.py b/chart/tests/test_kerberos.py
new file mode 100644
index 0000000..b0cf88d
--- /dev/null
+++ b/chart/tests/test_kerberos.py
@@ -0,0 +1,32 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import json
+import unittest
+
+from tests.helm_template_generator import render_chart
+
+
+class KerberosTest(unittest.TestCase):
+ def test_kerberos_not_mentioned_in_render_if_disabled(self):
+ k8s_objects = render_chart(name="NO-KERBEROS", values={"kerberos": {'enabled': False}})
+ # ignore airflow config map
+ k8s_objects_to_consider = [
+ obj for obj in k8s_objects if obj["metadata"]["name"] != "NO-KERBEROS-airflow-config"
+ ]
+ k8s_objects_to_consider_str = json.dumps(k8s_objects_to_consider)
+ self.assertNotIn("kerberos", k8s_objects_to_consider_str)
[airflow] 07/32: Add capability of adding service account
annotations to Helm Chart (#11387)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 5ce3da6a41f86cc7daf16cca0014cedc306151b7
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Fri Oct 9 22:54:21 2020 +0200
Add capability of adding service account annotations to Helm Chart (#11387)
We can now add annotations to the service accounts in a generic
way. This allows for example to add Workflow Identitty in GKE
environment but it is not limited to it.
Co-authored-by: Kamil BreguĊa <ka...@polidea.com>
Co-authored-by: Jacob Ferriero <jf...@google.com>
Co-authored-by: Kamil BreguĊa <ka...@polidea.com>
(cherry picked from commit 29a145cd6968a4fab5d6cf3197773e90ba91a642)
---
chart/README.md | 6 +++++-
chart/templates/scheduler/scheduler-serviceaccount.yaml | 6 ++++++
chart/templates/webserver/webserver-serviceaccount.yaml | 6 ++++++
chart/templates/workers/worker-serviceaccount.yaml | 8 +++++++-
chart/values.schema.json | 12 ++++++++++++
chart/values.yaml | 9 ++++++++-
6 files changed, 44 insertions(+), 3 deletions(-)
diff --git a/chart/README.md b/chart/README.md
index ef1caa3..266ef58 100644
--- a/chart/README.md
+++ b/chart/README.md
@@ -126,6 +126,9 @@ The following tables lists the configurable parameters of the Airflow chart and
| `labels` | Common labels to add to all objects defined in this chart | `{}` |
| `privateRegistry.enabled` | Enable usage of a private registry for Airflow base image | `false` |
| `privateRegistry.repository` | Repository where base image lives (eg: quay.io) | `~` |
+| `ingress.enabled` | Enable Kubernetes Ingress support | `false` |
+| `ingress.web.*` | Configs for the Ingress of the web Service | Please refer to `values.yaml` |
+| `ingress.flower.*` | Configs for the Ingress of the flower Service | Please refer to `values.yaml` |
| `networkPolicies.enabled` | Enable Network Policies to restrict traffic | `true` |
| `airflowHome` | Location of airflow home directory | `/opt/airflow` |
| `rbacEnabled` | Deploy pods with Kubernetes RBAC enabled | `true` |
@@ -207,6 +210,7 @@ The following tables lists the configurable parameters of the Airflow chart and
| `dags.persistence.*` | Dag persistence configuration | Please refer to `values.yaml` |
| `dags.gitSync.*` | Git sync configuration | Please refer to `values.yaml` |
| `multiNamespaceMode` | Whether the KubernetesExecutor can launch pods in multiple namespaces | `False` |
+| `serviceAccountAnnottions.*` | Map of annotations for worker, webserver, scheduler kubernetes service accounts | {} |
Specify each parameter using the `--set key=value[,key=value]` argument to `helm install`. For example,
@@ -221,7 +225,7 @@ helm install --name my-release \
KEDA stands for Kubernetes Event Driven Autoscaling. [KEDA](https://github.com/kedacore/keda) is a custom controller that allows users to create custom bindings
to the Kubernetes [Horizontal Pod Autoscaler](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/).
We've built an experimental scaler that allows users to create scalers based on postgreSQL queries. For the moment this exists
-on a seperate branch, but will be merged upstream soon. To install our custom version of KEDA on your cluster, please run
+on a separate branch, but will be merged upstream soon. To install our custom version of KEDA on your cluster, please run
```bash
helm repo add kedacore https://kedacore.github.io/charts
diff --git a/chart/templates/scheduler/scheduler-serviceaccount.yaml b/chart/templates/scheduler/scheduler-serviceaccount.yaml
index bfd9d3b..5dfa1dc 100644
--- a/chart/templates/scheduler/scheduler-serviceaccount.yaml
+++ b/chart/templates/scheduler/scheduler-serviceaccount.yaml
@@ -28,6 +28,12 @@ metadata:
release: {{ .Release.Name }}
chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
heritage: {{ .Release.Service }}
+ {{- with .Values.scheduler.serviceAccountAnnotations }}
+ annotations:
+ {{- range $key, $value := . }}
+ {{- printf "%s: %s" $key (tpl $value $ | quote) | nindent 4 }}
+ {{- end }}
+ {{- end }}
{{- with .Values.labels }}
{{ toYaml . | indent 4 }}
{{- end }}
diff --git a/chart/templates/webserver/webserver-serviceaccount.yaml b/chart/templates/webserver/webserver-serviceaccount.yaml
index ea86713..ba99cea 100644
--- a/chart/templates/webserver/webserver-serviceaccount.yaml
+++ b/chart/templates/webserver/webserver-serviceaccount.yaml
@@ -27,6 +27,12 @@ metadata:
release: {{ .Release.Name }}
chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
heritage: {{ .Release.Service }}
+ {{- with .Values.webserver.serviceAccountAnnotations }}
+ annotations:
+ {{- range $key, $value := . }}
+ {{- printf "%s: %s" $key (tpl $value $ | quote) | nindent 4 }}
+ {{- end }}
+ {{- end }}
{{- with .Values.labels }}
{{ toYaml . | indent 4 }}
{{- end }}
diff --git a/chart/templates/workers/worker-serviceaccount.yaml b/chart/templates/workers/worker-serviceaccount.yaml
index 8735097..4a7542e 100644
--- a/chart/templates/workers/worker-serviceaccount.yaml
+++ b/chart/templates/workers/worker-serviceaccount.yaml
@@ -28,7 +28,13 @@ metadata:
release: {{ .Release.Name }}
chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
heritage: {{ .Release.Service }}
-{{- with .Values.labels }}
+ {{- with .Values.workers.serviceAccountAnnotations }}
+ annotations:
+ {{- range $key, $value := . }}
+ {{- printf "%s: %s" $key (tpl $value $ | quote) | nindent 4 }}
+ {{- end }}
+ {{- end }}
+ {{- with .Values.labels }}
{{ toYaml . | indent 4 }}
{{- end }}
{{- end }}
diff --git a/chart/values.schema.json b/chart/values.schema.json
index 9776116..a5eda19 100644
--- a/chart/values.schema.json
+++ b/chart/values.schema.json
@@ -467,6 +467,10 @@
"safeToEvict": {
"description": "This setting tells Kubernetes that it's ok to evict when it wants to scale a node down.",
"type": "boolean"
+ },
+ "serviceAccountAnnotations": {
+ "description": "Annotations to add to the worker kubernetes service account.",
+ "type": "object"
}
}
},
@@ -507,6 +511,10 @@
"safeToEvict": {
"description": "This setting tells Kubernetes that its ok to evict when it wants to scale a node down.",
"type": "boolean"
+ },
+ "serviceAccountAnnotations": {
+ "description": "Annotations to add to the scheduler kubernetes service account.",
+ "type": "object"
}
}
},
@@ -631,6 +639,10 @@
"type": "object"
}
}
+ },
+ "serviceAccountAnnotations": {
+ "description": "Annotations to add to the webserver kubernetes service account.",
+ "type": "object"
}
}
},
diff --git a/chart/values.yaml b/chart/values.yaml
index f7c5e67..800f58a 100644
--- a/chart/values.yaml
+++ b/chart/values.yaml
@@ -233,7 +233,6 @@ kerberos:
admin_server = admin_server.foo.com
}
-
# Airflow Worker Config
workers:
# Number of airflow celery workers in StatefulSet
@@ -285,6 +284,8 @@ workers:
# This setting tells kubernetes that its ok to evict
# when it wants to scale a node down.
safeToEvict: true
+ # Annotations to add to worker kubernetes service account.
+ serviceAccountAnnotations: {}
# Airflow scheduler settings
scheduler:
@@ -312,6 +313,9 @@ scheduler:
# when it wants to scale a node down.
safeToEvict: true
+ # Annotations to add to scheduler kubernetes service account.
+ serviceAccountAnnotations: {}
+
# Airflow webserver settings
webserver:
livenessProbe:
@@ -372,6 +376,9 @@ webserver:
## service annotations
annotations: {}
+ # Annotations to add to webserver kubernetes service account.
+ serviceAccountAnnotations: {}
+
# Flower settings
flower:
# Additional network policies as needed
[airflow] 28/32: Move metrics configuration to new section -
metrics (#12165)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 44b50770f7f449da324e8bbc303a5f95a930be31
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Sun Nov 15 03:46:54 2020 +0100
Move metrics configuration to new section - metrics (#12165)
* Move metrics configuration to new section
* fixup! Move metrics configuration to new section
* fixup! fixup! Move metrics configuration to new section
* Apply suggestions from code review
Co-authored-by: Xiaodong DENG <xd...@gmail.com>
* fixup! Apply suggestions from code review
Co-authored-by: Xiaodong DENG <xd...@gmail.com>
(cherry-picked from commit 833ba453de9490bce742ebf4b648eec48caf2037)
---
chart/values.yaml | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/chart/values.yaml b/chart/values.yaml
index 1e3b81f..0f5b313 100644
--- a/chart/values.yaml
+++ b/chart/values.yaml
@@ -600,6 +600,11 @@ config:
auth_backend: airflow.api.auth.backend.deny_all
logging:
logging_level: DEBUG
+ metrics:
+ statsd_on: '{{ ternary "True" "False" .Values.statsd.enabled }}'
+ statsd_port: 9125
+ statsd_prefix: airflow
+ statsd_host: '{{ printf "%s-statsd" .Release.Name }}'
webserver:
enable_proxy_fix: 'True'
expose_config: 'True'
@@ -610,6 +615,7 @@ config:
scheduler:
scheduler_heartbeat_sec: 5
+ # For Airflow 1.10, backward compatibility
statsd_on: '{{ ternary "True" "False" .Values.statsd.enabled }}'
statsd_port: 9125
statsd_prefix: airflow
[airflow] 23/32: fix helm scheduler deployment / scheduler logs
(#11685)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit ffcbf6fa66199e4ddaf90fa6b7c70444b49108f8
Author: Florent Chehab <fc...@meilleursagents.com>
AuthorDate: Sat Oct 31 18:55:50 2020 +0100
fix helm scheduler deployment / scheduler logs (#11685)
Based on the airflow image entrypoint, we should use airflow commands directly.
The container exits otherwise.
(cherry picked from commit 069b1f71cf9fbeaea3af2dfdc14681996dd09f4b)
---
.../templates/scheduler/scheduler-deployment.yaml | 2 +-
chart/tests/test_basic_helm_chart.py | 31 ++++++++++++++++++++++
2 files changed, 32 insertions(+), 1 deletion(-)
diff --git a/chart/templates/scheduler/scheduler-deployment.yaml b/chart/templates/scheduler/scheduler-deployment.yaml
index 1ed1c3a..771dd1a 100644
--- a/chart/templates/scheduler/scheduler-deployment.yaml
+++ b/chart/templates/scheduler/scheduler-deployment.yaml
@@ -169,7 +169,7 @@ spec:
- name: scheduler-logs
image: {{ template "airflow_image" . }}
imagePullPolicy: {{ .Values.images.airflow.pullPolicy }}
- args: ["airflow", "serve_logs"]
+ args: ["serve_logs"]
ports:
- name: worker-logs
containerPort: {{ .Values.ports.workerLogs }}
diff --git a/chart/tests/test_basic_helm_chart.py b/chart/tests/test_basic_helm_chart.py
index 767a073..89a6f5a 100644
--- a/chart/tests/test_basic_helm_chart.py
+++ b/chart/tests/test_basic_helm_chart.py
@@ -16,6 +16,7 @@
# under the License.
import unittest
+from typing import Any, Dict, List, Union
import jmespath
@@ -84,3 +85,33 @@ class TestBaseChartTest(unittest.TestCase):
]
self.assertNotIn(('Job', 'TEST-BASIC-create-user'), list_of_kind_names_tuples)
self.assertEqual(OBJECT_COUNT_IN_BASIC_DEPLOYMENT - 1, len(k8s_objects))
+
+ def test_chart_is_consistent_with_official_airflow_image(self):
+ def get_k8s_objs_with_image(obj: Union[List[Any], Dict[str, Any]]) -> List[Dict[str, Any]]:
+ """
+ Recursive helper to retrieve all the k8s objects that have an "image" key
+ inside k8s obj or list of k8s obj
+ """
+ out = []
+ if isinstance(obj, list):
+ for item in obj:
+ out += get_k8s_objs_with_image(item)
+ if isinstance(obj, dict):
+ if "image" in obj:
+ out += [obj]
+ # include sub objs, just in case
+ for val in obj.values():
+ out += get_k8s_objs_with_image(val)
+ return out
+
+ image_repo = "test-airflow-repo/airflow"
+ k8s_objects = render_chart("TEST-BASIC", {"defaultAirflowRepository": image_repo})
+
+ objs_with_image = get_k8s_objs_with_image(k8s_objects)
+ for obj in objs_with_image:
+ image: str = obj["image"] # pylint: disable=invalid-sequence-index
+ if image.startswith(image_repo):
+ # Make sure that a command is not specified
+ self.assertNotIn("command", obj)
+ # Make sure that the first arg is never airflow
+ self.assertNotEqual(obj["args"][0], "airflow") # pylint: disable=invalid-sequence-index
[airflow] 14/32: Improvements for pod template file with git sync
container (#11511)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 5ee07b9c44ac45b133c9a7910ff0a02c4af5b824
Author: Fernando Llaca <fe...@gmail.com>
AuthorDate: Mon Oct 19 23:16:29 2020 +0200
Improvements for pod template file with git sync container (#11511)
* Helm chart fixes in pod template
- default pod_template image to `defaultAirflowRepository:defaultAirflowTag`
- fix never-ending git-sync init containers
- fix broken reference to volume
* Fix helm chart test
(cherry picked from commit 52b4733b8297c8a08210aead18c661a9d58f3f6c)
---
chart/files/pod-template-file.kubernetes-helm-yaml | 6 ++---
chart/templates/_helpers.yaml | 8 +++++++
chart/tests/migrate-database-job_test.yaml | 27 ----------------------
chart/values.yaml | 4 ++++
4 files changed, 15 insertions(+), 30 deletions(-)
diff --git a/chart/files/pod-template-file.kubernetes-helm-yaml b/chart/files/pod-template-file.kubernetes-helm-yaml
index b19edf1..2206c57 100644
--- a/chart/files/pod-template-file.kubernetes-helm-yaml
+++ b/chart/files/pod-template-file.kubernetes-helm-yaml
@@ -22,7 +22,7 @@ metadata:
spec:
{{- if .Values.dags.gitSync.enabled }}
initContainers:
-{{- include "git_sync_container" . | indent 8 }}
+{{- include "git_sync_container" (dict "Values" .Values "is_init" "true") | indent 8 }}
{{- end }}
containers:
- args: []
@@ -32,7 +32,7 @@ spec:
value: LocalExecutor
{{- include "standard_airflow_environment" . | indent 4 }}
envFrom: []
- image: dummy_image
+ image: {{ template "pod_template_image" . }}
imagePullPolicy: {{ .Values.images.airflow.pullPolicy }}
name: base
ports: []
@@ -56,7 +56,7 @@ spec:
{{- end }}
{{- if or .Values.dags.gitSync.enabled .Values.dags.persistence.enabled }}
- mountPath: {{ include "airflow_dags_mount_path" . }}
- name: airflow-dags
+ name: dags
readOnly: true
{{- if .Values.dags.persistence.enabled }}
subPath: {{.Values.dags.gitSync.dest }}/{{ .Values.dags.gitSync.subPath }}
diff --git a/chart/templates/_helpers.yaml b/chart/templates/_helpers.yaml
index 4f3e0a0..6dbdc04 100644
--- a/chart/templates/_helpers.yaml
+++ b/chart/templates/_helpers.yaml
@@ -152,6 +152,10 @@
value: {{ .Values.dags.gitSync.wait | quote }}
- name: GIT_SYNC_MAX_SYNC_FAILURES
value: {{ .Values.dags.gitSync.maxFailures | quote }}
+ {{- if .is_init }}
+ - name: GIT_SYNC_ONE_TIME
+ value: "true"
+ {{- end }}
volumeMounts:
- name: dags
mountPath: {{ .Values.dags.gitSync.root }}
@@ -174,6 +178,10 @@
{{ printf "%s:%s" (.Values.images.airflow.repository | default .Values.defaultAirflowRepository) (.Values.images.airflow.tag | default .Values.defaultAirflowTag) }}
{{- end }}
+{{ define "pod_template_image" -}}
+{{ printf "%s:%s" (.Values.images.pod_template.repository | default .Values.defaultAirflowRepository) (.Values.images.pod_template.tag | default .Values.defaultAirflowTag) }}
+{{- end }}
+
# This helper is used for airflow containers that do not need the users code.
{{ define "default_airflow_image" -}}
{{ printf "%s:%s" .Values.defaultAirflowRepository .Values.defaultAirflowTag }}
diff --git a/chart/tests/migrate-database-job_test.yaml b/chart/tests/migrate-database-job_test.yaml
deleted file mode 100644
index 117b7a7..0000000
--- a/chart/tests/migrate-database-job_test.yaml
+++ /dev/null
@@ -1,27 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
-templates:
- - migrate-database-job.yaml
-tests:
- - it: should run by default
- asserts:
- - isKind:
- of: Job
- - equal:
- path: spec.template.spec.containers[0].name
- value: run-airflow-migrations
diff --git a/chart/values.yaml b/chart/values.yaml
index 7b609a6..1556341 100644
--- a/chart/values.yaml
+++ b/chart/values.yaml
@@ -123,6 +123,10 @@ images:
repository: ~
tag: ~
pullPolicy: IfNotPresent
+ pod_template:
+ repository: ~
+ tag: ~
+ pullPolicy: IfNotPresent
flower:
repository: ~
tag: ~
[airflow] 31/32: Mount airflow.cfg to pod_template_file (#12311)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 1bd745e1ca4897e419c3bfd8735e9fed46f15e1b
Author: Daniel Imberman <da...@gmail.com>
AuthorDate: Thu Nov 12 20:58:13 2020 -0800
Mount airflow.cfg to pod_template_file (#12311)
* Mount airflow.cfg to pod_template_file
k8sexecutor workers were launching without an airflow.cfg,
this was preventing logs from being sent to distributed logging systems.
* consistent naming
(cherry picked from commit ce06be4752708b35b074832b0f227177cad055fd)
---
chart/files/pod-template-file.kubernetes-helm-yaml | 15 +++++++++++----
chart/tests/test_pod_template_file.py | 21 +++++++++++++++++++++
2 files changed, 32 insertions(+), 4 deletions(-)
diff --git a/chart/files/pod-template-file.kubernetes-helm-yaml b/chart/files/pod-template-file.kubernetes-helm-yaml
index b4ec9a5..5c4fb92 100644
--- a/chart/files/pod-template-file.kubernetes-helm-yaml
+++ b/chart/files/pod-template-file.kubernetes-helm-yaml
@@ -40,6 +40,16 @@ spec:
volumeMounts:
- mountPath: {{ template "airflow_logs" . }}
name: airflow-logs
+ - name: config
+ mountPath: {{ template "airflow_config_path" . }}
+ subPath: airflow.cfg
+ readOnly: true
+{{- if .Values.scheduler.airflowLocalSettings }}
+ - name: config
+ mountPath: {{ template "airflow_local_setting_path" . }}
+ subPath: airflow_local_settings.py
+ readOnly: true
+{{- end }}
{{- if .Values.dags.gitSync.knownHosts }}
- mountPath: /etc/git-secret/known_hosts
name: {{ .Values.dags.gitSync.knownHosts }}
@@ -92,7 +102,4 @@ spec:
{{- end }}
- configMap:
name: {{ include "airflow_config" . }}
- name: airflow-config
- - configMap:
- name: {{ include "airflow_config" . }}
- name: airflow-local-settings
+ name: config
diff --git a/chart/tests/test_pod_template_file.py b/chart/tests/test_pod_template_file.py
index 3c61733..fb0dd45 100644
--- a/chart/tests/test_pod_template_file.py
+++ b/chart/tests/test_pod_template_file.py
@@ -185,6 +185,27 @@ class PodTemplateFileTest(unittest.TestCase):
self.assertEqual("dummy_image:latest", jmespath.search("spec.containers[0].image", docs[0]))
self.assertEqual("base", jmespath.search("spec.containers[0].name", docs[0]))
+ def test_mount_airflow_cfg(self):
+ docs = render_chart(
+ values={},
+ show_only=["templates/pod-template-file.yaml"],
+ )
+
+ self.assertRegex(docs[0]["kind"], "Pod")
+ self.assertDictEqual(
+ {'configMap': {'name': 'RELEASE-NAME-airflow-config'}, 'name': 'airflow-config'},
+ jmespath.search("spec.volumes[1]", docs[0]),
+ )
+ self.assertDictEqual(
+ {
+ 'name': 'config',
+ 'mountPath': '/opt/airflow/airflow.cfg',
+ 'subPath': 'airflow.cfg',
+ 'readOnly': True,
+ },
+ jmespath.search("spec.containers[0].volumeMounts[1]", docs[0]),
+ )
+
def test_should_create_valid_affinity_and_node_selector(self):
docs = render_chart(
values={
[airflow] 20/32: Add Flower Authentication to Helm Chart (#11836)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit a01bd324b11382f87f61c3ae1b96bdcc700c7f89
Author: Songkran Nethan <ka...@gmail.com>
AuthorDate: Sun Nov 1 00:37:45 2020 +0700
Add Flower Authentication to Helm Chart (#11836)
(cherry picked from commit 644ac1b06019bcd1c1c540373051c31b766efccf)
---
chart/templates/_helpers.yaml | 4 ++
chart/templates/flower/flower-deployment.yaml | 29 +++++++++---
chart/templates/secrets/flower-secret.yaml | 36 ++++++++++++++
chart/tests/test_flower_authorization.py | 68 +++++++++++++++++++++++++++
chart/values.schema.json | 21 +++++++++
chart/values.yaml | 7 +++
6 files changed, 159 insertions(+), 6 deletions(-)
diff --git a/chart/templates/_helpers.yaml b/chart/templates/_helpers.yaml
index beb26a7..9914127 100644
--- a/chart/templates/_helpers.yaml
+++ b/chart/templates/_helpers.yaml
@@ -252,6 +252,10 @@
{{ default (printf "%s-elasticsearch" .Release.Name) .Values.elasticsearch.secretName }}
{{- end }}
+{{ define "flower_secret" -}}
+{{ default (printf "%s-flower" .Release.Name) .Values.flower.secretName }}
+{{- end }}
+
{{ define "kerberos_keytab_secret" -}}
{{ .Release.Name }}-kerberos-keytab
{{- end }}
diff --git a/chart/templates/flower/flower-deployment.yaml b/chart/templates/flower/flower-deployment.yaml
index 887b70d..c5d1f91 100644
--- a/chart/templates/flower/flower-deployment.yaml
+++ b/chart/templates/flower/flower-deployment.yaml
@@ -81,19 +81,36 @@ spec:
containerPort: {{ .Values.ports.flowerUI }}
livenessProbe:
failureThreshold: 10
- httpGet:
- path: /
- port: {{ .Values.ports.flowerUI }}
+ exec:
+ command:
+ - curl
+ {{- if (or .Values.flower.secretName (and .Values.flower.username .Values.flower.password))}}
+ - "--user"
+ - $AIRFLOW__CELERY__FLOWER_BASIC_AUTH
+ {{- end }}
+ - {{ printf "localhost:%s" (.Values.ports.flowerUI | toString) }}
initialDelaySeconds: 10
periodSeconds: 5
readinessProbe:
failureThreshold: 10
- httpGet:
- path: /
- port: {{ .Values.ports.flowerUI }}
+ exec:
+ command:
+ - curl
+ {{- if (or .Values.flower.secretName (and .Values.flower.username .Values.flower.password))}}
+ - "--user"
+ - $AIRFLOW__CELERY__FLOWER_BASIC_AUTH
+ {{- end }}
+ - {{ printf "localhost:%s" (.Values.ports.flowerUI | toString) }}
initialDelaySeconds: 10
periodSeconds: 5
env:
+ {{- if (or .Values.flower.secretName (and .Values.flower.username .Values.flower.password))}}
+ - name: AIRFLOW__CELERY__FLOWER_BASIC_AUTH
+ valueFrom:
+ secretKeyRef:
+ name: {{ template "flower_secret" . }}
+ key: basicAuth
+ {{- end }}
{{- include "standard_airflow_environment" . | indent 10 }}
volumes:
- name: config
diff --git a/chart/templates/secrets/flower-secret.yaml b/chart/templates/secrets/flower-secret.yaml
new file mode 100644
index 0000000..c88734d
--- /dev/null
+++ b/chart/templates/secrets/flower-secret.yaml
@@ -0,0 +1,36 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+################################
+## Flower Secret
+#################################
+{{- if (and (not .Values.flower.secretName) .Values.flower.username .Values.flower.password) }}
+kind: Secret
+apiVersion: v1
+metadata:
+ name: {{ .Release.Name }}-flower
+ labels:
+ release: {{ .Release.Name }}
+ chart: {{ .Chart.Name }}
+ heritage: {{ .Release.Service }}
+{{- with .Values.labels }}
+{{ toYaml . | indent 4 }}
+{{- end }}
+type: Opaque
+data:
+ basicAuth: {{ (printf "%s:%s" .Values.flower.username .Values.flower.password) | b64enc | quote }}
+{{- end }}
diff --git a/chart/tests/test_flower_authorization.py b/chart/tests/test_flower_authorization.py
new file mode 100644
index 0000000..f0cc5b0
--- /dev/null
+++ b/chart/tests/test_flower_authorization.py
@@ -0,0 +1,68 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import unittest
+
+import jmespath
+from tests.helm_template_generator import render_chart
+
+
+class FlowerAuthorizationTest(unittest.TestCase):
+ def test_should_create_flower_deployment_with_authorization(self):
+ docs = render_chart(
+ values={
+ "executor": "CeleryExecutor",
+ "flower": {"username": "flower", "password": "fl0w3r"},
+ "ports": {"flowerUI": 7777},
+ },
+ show_only=["templates/flower/flower-deployment.yaml"],
+ )
+
+ self.assertEqual(
+ "AIRFLOW__CELERY__FLOWER_BASIC_AUTH",
+ jmespath.search("spec.template.spec.containers[0].env[0].name", docs[0]),
+ )
+ self.assertEqual(
+ ['curl', '--user', '$AIRFLOW__CELERY__FLOWER_BASIC_AUTH', 'localhost:7777'],
+ jmespath.search("spec.template.spec.containers[0].livenessProbe.exec.command", docs[0]),
+ )
+ self.assertEqual(
+ ['curl', '--user', '$AIRFLOW__CELERY__FLOWER_BASIC_AUTH', 'localhost:7777'],
+ jmespath.search("spec.template.spec.containers[0].readinessProbe.exec.command", docs[0]),
+ )
+
+ def test_should_create_flower_deployment_without_authorization(self):
+ docs = render_chart(
+ values={
+ "executor": "CeleryExecutor",
+ "ports": {"flowerUI": 7777},
+ },
+ show_only=["templates/flower/flower-deployment.yaml"],
+ )
+
+ self.assertEqual(
+ "AIRFLOW__CORE__FERNET_KEY",
+ jmespath.search("spec.template.spec.containers[0].env[0].name", docs[0]),
+ )
+ self.assertEqual(
+ ['curl', 'localhost:7777'],
+ jmespath.search("spec.template.spec.containers[0].livenessProbe.exec.command", docs[0]),
+ )
+ self.assertEqual(
+ ['curl', 'localhost:7777'],
+ jmespath.search("spec.template.spec.containers[0].readinessProbe.exec.command", docs[0]),
+ )
diff --git a/chart/values.schema.json b/chart/values.schema.json
index 2f25892..7881c82 100644
--- a/chart/values.schema.json
+++ b/chart/values.schema.json
@@ -784,6 +784,27 @@
"resources": {
"type": "object"
},
+ "secretName": {
+ "description": "A secret containing the user and password pair.",
+ "type": [
+ "string",
+ "null"
+ ]
+ },
+ "username": {
+ "description": "Username use to access flower.",
+ "type": [
+ "string",
+ "null"
+ ]
+ },
+ "password": {
+ "description": "Password use to access flower.",
+ "type": [
+ "string",
+ "null"
+ ]
+ },
"service": {
"description": "Flower service configuration.",
"type": "object",
diff --git a/chart/values.yaml b/chart/values.yaml
index 4855214..1e3b81f 100644
--- a/chart/values.yaml
+++ b/chart/values.yaml
@@ -409,6 +409,13 @@ flower:
# cpu: 100m
# memory: 128Mi
+ # A secret containing the connection
+ secretName: ~
+
+ # Else, if username and password are set, create secret from username and password
+ username: ~
+ password: ~
+
service:
type: ClusterIP
[airflow] 01/32: Fix gitSync user in the helm Chart (#11127)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit a1077dca5a77e7debb416c54cd9db9cb7e1edc55
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Fri Sep 25 12:31:45 2020 +0200
Fix gitSync user in the helm Chart (#11127)
There was a problem with user in Git Sync mode of the Helm Chart
in connection with the git sync image and official Airflow
image. Since we are using the official image, most of the
containers are run with the "50000" user, but the git-sync image
used by the git sync user is 65533 so we have to set it as
default. We also exposed that value as parameter, so that
another image could be used here as well.
(cherry picked from commit ce6b257de7b442e9103728c2a293df23ece909d2)
---
chart/templates/_helpers.yaml | 2 ++
chart/values.yaml | 1 +
2 files changed, 3 insertions(+)
diff --git a/chart/templates/_helpers.yaml b/chart/templates/_helpers.yaml
index 49c3b3f..195d484 100644
--- a/chart/templates/_helpers.yaml
+++ b/chart/templates/_helpers.yaml
@@ -105,6 +105,8 @@
{{- define "git_sync_container"}}
- name: {{ .Values.dags.gitSync.containerName }}
image: "{{ .Values.dags.gitSync.containerRepository }}:{{ .Values.dags.gitSync.containerTag }}"
+ securityContext:
+ runAsUser: {{ .Values.dags.gitSync.uid }}
env:
{{- if .Values.dags.gitSync.sshKeySecret }}
- name: GIT_SSH_KEY_FILE
diff --git a/chart/values.yaml b/chart/values.yaml
index b8c4fcf..c0b9ff5 100644
--- a/chart/values.yaml
+++ b/chart/values.yaml
@@ -601,3 +601,4 @@ dags:
containerRepository: k8s.gcr.io/git-sync
containerTag: v3.1.6
containerName: git-sync
+ uid: 65533
[airflow] 24/32: Add Kubernetes cleanup-pods CLI command for Helm
Chart (#11802)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit a70cdd15fbbe413e7483f7566bc9e9a249d5c405
Author: Kaxil Naik <ka...@gmail.com>
AuthorDate: Tue Nov 3 15:28:51 2020 +0000
Add Kubernetes cleanup-pods CLI command for Helm Chart (#11802)
closes: https://github.com/apache/airflow/issues/11146
(cherry picked from commit 980c7252c0f28c251e9f87d736cd88d6027f3da3)
---
chart/templates/cleanup/cleanup-cronjob.yaml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/chart/templates/cleanup/cleanup-cronjob.yaml b/chart/templates/cleanup/cleanup-cronjob.yaml
index 3f44976..3ad3d4f 100644
--- a/chart/templates/cleanup/cleanup-cronjob.yaml
+++ b/chart/templates/cleanup/cleanup-cronjob.yaml
@@ -61,7 +61,7 @@ spec:
image: {{ template "default_airflow_image" . }}
imagePullPolicy: {{ .Values.images.airflow.pullPolicy }}
# Don't use entry point here, we don't need to wait on pg-bouncer etc being available.
- command: ["airflow-cleanup-pods", "--namespace={{ .Release.Namespace }}"]
+ args: ["kubernetes", "cleanup-pods", "--namespace={{ .Release.Namespace }}"]
env:
{{- include "standard_airflow_environment" . | indent 12 }}
{{- end }}
[airflow] 29/32: Fix spelling (#12253)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 13442dbe20b3e0e1b069a59814f8d9161c4d714f
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Sun Nov 15 03:48:38 2020 +0100
Fix spelling (#12253)
Author: John Bampton <jb...@users.noreply.github.com>
(cherry-picked from commit 4f9439dec13d3118d5423bac246064dea7a95002)
---
chart/README.md | 2 +-
chart/tests/test_celery_kubernetes_executor.py | 2 +-
chart/tests/test_git_sync_webserver.py | 4 ++--
chart/tests/test_git_sync_worker.py | 4 ++--
4 files changed, 6 insertions(+), 6 deletions(-)
diff --git a/chart/README.md b/chart/README.md
index db59db7..1ae5d6c 100644
--- a/chart/README.md
+++ b/chart/README.md
@@ -298,7 +298,7 @@ helm list -n airflow
```
Run `kubectl port-forward svc/airflow-webserver 8080:8080 -n airflow`
-to port-forward the Airflow UI to http://localhost:8080/ to cofirm Airflow is working.
+to port-forward the Airflow UI to http://localhost:8080/ to confirm Airflow is working.
**Build a Docker image from your DAGs:**
diff --git a/chart/tests/test_celery_kubernetes_executor.py b/chart/tests/test_celery_kubernetes_executor.py
index fb21929..6c54e80 100644
--- a/chart/tests/test_celery_kubernetes_executor.py
+++ b/chart/tests/test_celery_kubernetes_executor.py
@@ -23,7 +23,7 @@ from tests.helm_template_generator import render_chart
class CeleryKubernetesExecutorTest(unittest.TestCase):
- def test_should_create_a_worker_deployment_with_the_celery_executo(self):
+ def test_should_create_a_worker_deployment_with_the_celery_executor(self):
docs = render_chart(
values={
"executor": "CeleryExecutor",
diff --git a/chart/tests/test_git_sync_webserver.py b/chart/tests/test_git_sync_webserver.py
index 75ec51b..09c9aa3 100644
--- a/chart/tests/test_git_sync_webserver.py
+++ b/chart/tests/test_git_sync_webserver.py
@@ -23,7 +23,7 @@ from tests.helm_template_generator import render_chart
class GitSyncWebserverTest(unittest.TestCase):
- def test_should_add_dags_volume_to_the_webserver_if_git_sync_and_peristence_is_enabled(self):
+ def test_should_add_dags_volume_to_the_webserver_if_git_sync_and_persistence_is_enabled(self):
docs = render_chart(
values={"dags": {"gitSync": {"enabled": True}, "persistence": {"enabled": True}}},
show_only=["templates/webserver/webserver-deployment.yaml"],
@@ -31,7 +31,7 @@ class GitSyncWebserverTest(unittest.TestCase):
self.assertEqual("dags", jmespath.search("spec.template.spec.volumes[1].name", docs[0]))
- def test_should_add_dags_volume_to_the_webserver_if_git_sync_is_enabled_and_peristence_is_disabled(self):
+ def test_should_add_dags_volume_to_the_webserver_if_git_sync_is_enabled_and_persistence_is_disabled(self):
docs = render_chart(
values={"dags": {"gitSync": {"enabled": True}, "persistence": {"enabled": False}}},
show_only=["templates/webserver/webserver-deployment.yaml"],
diff --git a/chart/tests/test_git_sync_worker.py b/chart/tests/test_git_sync_worker.py
index e5036d7..a56b0dc 100644
--- a/chart/tests/test_git_sync_worker.py
+++ b/chart/tests/test_git_sync_worker.py
@@ -23,7 +23,7 @@ from tests.helm_template_generator import render_chart
class GitSyncWorkerTest(unittest.TestCase):
- def test_should_add_dags_volume_to_the_worker_if_git_sync_and_peristence_is_enabled(self):
+ def test_should_add_dags_volume_to_the_worker_if_git_sync_and_persistence_is_enabled(self):
docs = render_chart(
values={
"executor": "CeleryExecutor",
@@ -35,7 +35,7 @@ class GitSyncWorkerTest(unittest.TestCase):
self.assertEqual("config", jmespath.search("spec.template.spec.volumes[0].name", docs[0]))
self.assertEqual("dags", jmespath.search("spec.template.spec.volumes[1].name", docs[0]))
- def test_should_add_dags_volume_to_the_worker_if_git_sync_is_enabled_and_peristence_is_disabled(self):
+ def test_should_add_dags_volume_to_the_worker_if_git_sync_is_enabled_and_persistence_is_disabled(self):
docs = render_chart(
values={
"executor": "CeleryExecutor",
[airflow] 32/32: Fix helm unit test for pod_template_file (#12345)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit eeb913488701486f933583ef17a1e37fa731ee2b
Author: Daniel Imberman <da...@gmail.com>
AuthorDate: Fri Nov 13 10:05:24 2020 -0800
Fix helm unit test for pod_template_file (#12345)
Fixes bug in unittest that is causing master to fail.
(cherry picked from commit 64978e323f0d6951a079f1dbe7d6b80127ffc19d)
---
chart/tests/test_pod_template_file.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/chart/tests/test_pod_template_file.py b/chart/tests/test_pod_template_file.py
index fb0dd45..768638f 100644
--- a/chart/tests/test_pod_template_file.py
+++ b/chart/tests/test_pod_template_file.py
@@ -193,7 +193,7 @@ class PodTemplateFileTest(unittest.TestCase):
self.assertRegex(docs[0]["kind"], "Pod")
self.assertDictEqual(
- {'configMap': {'name': 'RELEASE-NAME-airflow-config'}, 'name': 'airflow-config'},
+ {'configMap': {'name': 'RELEASE-NAME-airflow-config'}, 'name': 'config'},
jmespath.search("spec.volumes[1]", docs[0]),
)
self.assertDictEqual(
[airflow] 06/32: Add CeleryKubernetesExecutor to helm chart (#11288)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 4c2cb60e6294475f252979832e3842a7f3500667
Author: Daniel Imberman <da...@gmail.com>
AuthorDate: Mon Oct 5 15:46:10 2020 -0700
Add CeleryKubernetesExecutor to helm chart (#11288)
Users of the CeleryKubernetesExecutor will require both
Celery and Kubernetes features to launch tasks.
This PR will also serve as the basis for integration tests for this
executor
Co-authored-by: Daniel Imberman <da...@astronomer.io>
(cherry picked from commit 48474600ccd7c586eb5dd8ea9aa2d818db7de505)
---
chart/templates/NOTES.txt | 16 ++++++--
chart/templates/_helpers.yaml | 6 +--
chart/templates/configmap.yaml | 2 +-
chart/templates/flower/flower-deployment.yaml | 2 +-
.../flower-ingress.yaml} | 43 ++++++++++++----------
chart/templates/flower/flower-service.yaml | 2 +-
chart/templates/rbac/pod-launcher-rolebinding.yaml | 4 +-
chart/templates/redis/redis-networkpolicy.yaml | 2 +-
chart/templates/redis/redis-service.yaml | 2 +-
chart/templates/redis/redis-statefulset.yaml | 2 +-
.../templates/scheduler/scheduler-deployment.yaml | 2 -
chart/templates/workers/worker-deployment.yaml | 2 +-
chart/templates/workers/worker-kedaautoscaler.yaml | 2 +-
chart/templates/workers/worker-networkpolicy.yaml | 2 +-
chart/templates/workers/worker-service.yaml | 2 +-
15 files changed, 51 insertions(+), 40 deletions(-)
diff --git a/chart/templates/NOTES.txt b/chart/templates/NOTES.txt
index 3df1e3b..afca72a 100644
--- a/chart/templates/NOTES.txt
+++ b/chart/templates/NOTES.txt
@@ -20,10 +20,18 @@ under the License.
Your release is named {{ .Release.Name }}.
-You can now access your dashboard(s) by executing the following command(s) and visiting the corresponding port at localhost in your browser:
+{{- if .Values.ingress.enabled }}
+You can now access your dashboard(s) by following defined Ingress urls:
-Airflow dashboard: kubectl port-forward svc/{{ .Release.Name }}-webserver {{ .Values.ports.airflowUI }}:{{ .Values.ports.airflowUI }}
-{{- if eq .Values.executor "CeleryExecutor"}}
-Flower dashboard: kubectl port-forward svc/{{ .Release.Name }}-flower {{ .Values.ports.flowerUI }}:{{ .Values.ports.flowerUI }}
+Airflow dashboard: http{{ if .Values.ingress.web.tls.enabled }}s{{ end }}://{{ .Values.ingress.web.host }}{{ .Values.ingress.web.path }}/
+{{- if or (eq .Values.executor "CeleryExecutor") (eq .Values.executor "CeleryKubernetesExecutor") }}
+Flower dashboard: http{{ if .Values.ingress.flower.tls.enabled }}s{{ end }}://{{ .Values.ingress.flower.host }}{{ .Values.ingress.flower.path }}/
+{{- end }}
+{{- else }}
+You can now access your dashboard(s) by executing the following command(s) and visiting the corresponding port at localhost in your browser:
+Airflow dashboard: kubectl port-forward svc/{{ .Release.Name }}-webserver {{ .Values.ports.airflowUI }}:{{ .Values.ports.airflowUI }} --namespace {{ .Release.Namespace }}
+{{- if or (eq .Values.executor "CeleryExecutor") (eq .Values.executor "CeleryKubernetesExecutor")}}
+Flower dashboard: kubectl port-forward svc/{{ .Release.Name }}-flower {{ .Values.ports.flowerUI }}:{{ .Values.ports.flowerUI }} --namespace {{ .Release.Namespace }}
+{{- end }}
{{- end }}
diff --git a/chart/templates/_helpers.yaml b/chart/templates/_helpers.yaml
index 5d3ae73..02b255a 100644
--- a/chart/templates/_helpers.yaml
+++ b/chart/templates/_helpers.yaml
@@ -33,7 +33,7 @@
secretKeyRef:
name: {{ template "airflow_metadata_secret" . }}
key: connection
- {{- if eq .Values.executor "CeleryExecutor" }}
+ {{- if or (eq .Values.executor "CeleryExecutor") (eq .Values.executor "CeleryKubernetesExecutor") }}
- name: AIRFLOW__CELERY__CELERY_RESULT_BACKEND
valueFrom:
secretKeyRef:
@@ -72,7 +72,7 @@
{{- range $i, $config := .Values.env }}
- name: {{ $config.name }}
value: {{ $config.value | quote }}
- {{- if eq $.Values.executor "KubernetesExecutor" }}
+ {{- if or (eq $.Values.executor "KubernetesExecutor") (eq $.Values.executor "CeleryKubernetesExecutor") }}
- name: AIRFLOW__KUBERNETES_ENVIRONMENT_VARIABLES__{{ $config.name }}
value: {{ $config.value | quote }}
{{- end }}
@@ -85,7 +85,7 @@
name: {{ $config.secretName }}
key: {{ default "value" $config.secretKey }}
{{- end }}
- {{- if eq .Values.executor "KubernetesExecutor" }}
+ {{- if or (eq $.Values.executor "KubernetesExecutor") (eq $.Values.executor "CeleryKubernetesExecutor") }}
{{- range $i, $config := .Values.secret }}
- name: AIRFLOW__KUBERNETES_SECRETS__{{ $config.envName }}
value: {{ printf "%s=%s" $config.secretName $config.secretKey }}
diff --git a/chart/templates/configmap.yaml b/chart/templates/configmap.yaml
index b5bc656..2cf4502 100644
--- a/chart/templates/configmap.yaml
+++ b/chart/templates/configmap.yaml
@@ -55,7 +55,7 @@ data:
known_hosts: |
{{ .Values.dags.gitSync.knownHosts | nindent 4 }}
{{- end }}
-{{- if eq .Values.executor "KubernetesExecutor" }}
+{{- if or (eq $.Values.executor "KubernetesExecutor") (eq $.Values.executor "CeleryKubernetesExecutor") }}
pod_template_file.yaml: |-
{{- if .Values.podTemplate }}
{{ .Values.podTemplate | nindent 4 }}
diff --git a/chart/templates/flower/flower-deployment.yaml b/chart/templates/flower/flower-deployment.yaml
index 145c80b..887b70d 100644
--- a/chart/templates/flower/flower-deployment.yaml
+++ b/chart/templates/flower/flower-deployment.yaml
@@ -18,7 +18,7 @@
################################
## Airflow Flower Deployment
#################################
-{{- if eq .Values.executor "CeleryExecutor" }}
+{{- if or (eq .Values.executor "CeleryExecutor") (eq .Values.executor "CeleryKubernetesExecutor") }}
kind: Deployment
apiVersion: apps/v1
metadata:
diff --git a/chart/templates/workers/worker-service.yaml b/chart/templates/flower/flower-ingress.yaml
similarity index 54%
copy from chart/templates/workers/worker-service.yaml
copy to chart/templates/flower/flower-ingress.yaml
index 2c7768f..228631b 100644
--- a/chart/templates/workers/worker-service.yaml
+++ b/chart/templates/flower/flower-ingress.yaml
@@ -16,31 +16,36 @@
# under the License.
################################
-## Airflow Worker Service
+## Airflow Flower Ingress
#################################
-{{- if eq .Values.executor "CeleryExecutor" }}
-kind: Service
-apiVersion: v1
+{{- if and .Values.ingress.enabled (or (eq .Values.executor "CeleryExecutor") (eq .Values.executor "CeleryKubernetesExecutor")) }}
+apiVersion: networking.k8s.io/v1beta1
+kind: Ingress
metadata:
- name: {{ .Release.Name }}-worker
+ name: {{ .Release.Name }}-flower-ingress
labels:
tier: airflow
- component: worker
+ component: flower-ingress
release: {{ .Release.Name }}
chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
heritage: {{ .Release.Service }}
-{{- with .Values.labels }}
-{{ toYaml . | indent 4 }}
-{{- end }}
+ annotations:
+ {{- range $key, $value := .Values.ingress.flower.annotations }}
+ {{ $key }}: {{ $value | quote }}
+ {{- end }}
spec:
- clusterIP: None
- selector:
- tier: airflow
- component: worker
- release: {{ .Release.Name }}
- ports:
- - name: worker-logs
- protocol: TCP
- port: {{ .Values.ports.workerLogs }}
- targetPort: {{ .Values.ports.workerLogs }}
+ {{- if .Values.ingress.flower.tls.enabled }}
+ tls:
+ - hosts:
+ - {{ .Values.ingress.flower.host }}
+ secretName: {{ .Values.ingress.flower.tls.secretName }}
+ {{- end }}
+ rules:
+ - http:
+ paths:
+ - path: {{ .Values.ingress.flower.path }}
+ backend:
+ serviceName: {{ .Release.Name }}-flower
+ servicePort: flower-ui
+ host: {{ .Values.ingress.flower.host }}
{{- end }}
diff --git a/chart/templates/flower/flower-service.yaml b/chart/templates/flower/flower-service.yaml
index 187046b8f..b723a12 100644
--- a/chart/templates/flower/flower-service.yaml
+++ b/chart/templates/flower/flower-service.yaml
@@ -18,7 +18,7 @@
################################
## Airflow Flower Service Component
#################################
-{{- if eq .Values.executor "CeleryExecutor" }}
+{{- if or (eq .Values.executor "CeleryExecutor") (eq .Values.executor "CeleryKubernetesExecutor") }}
kind: Service
apiVersion: v1
metadata:
diff --git a/chart/templates/rbac/pod-launcher-rolebinding.yaml b/chart/templates/rbac/pod-launcher-rolebinding.yaml
index 6a1a5be..a3eab66 100644
--- a/chart/templates/rbac/pod-launcher-rolebinding.yaml
+++ b/chart/templates/rbac/pod-launcher-rolebinding.yaml
@@ -19,8 +19,8 @@
## Airflow Pod Launcher Role Binding
#################################
{{- if and .Values.rbacEnabled .Values.allowPodLaunching }}
-{{- $grantScheduler := or (eq .Values.executor "LocalExecutor") (eq .Values.executor "SequentialExecutor") (eq .Values.executor "KubernetesExecutor") }}
-{{- $grantWorker := or (eq .Values.executor "CeleryExecutor") (eq .Values.executor "KubernetesExecutor") }}
+{{- $grantScheduler := or (eq .Values.executor "LocalExecutor") (eq .Values.executor "SequentialExecutor") (eq .Values.executor "KubernetesExecutor") (eq .Values.executor "CeleryKubernetesExecutor")}}
+{{- $grantWorker := or (eq .Values.executor "CeleryExecutor") (eq .Values.executor "KubernetesExecutor") (eq .Values.executor "CeleryKubernetesExecutor") }}
{{- if .Values.multiNamespaceMode }}
kind: ClusterRoleBinding
{{- else }}
diff --git a/chart/templates/redis/redis-networkpolicy.yaml b/chart/templates/redis/redis-networkpolicy.yaml
index b0a9e49..a7a7950 100644
--- a/chart/templates/redis/redis-networkpolicy.yaml
+++ b/chart/templates/redis/redis-networkpolicy.yaml
@@ -18,7 +18,7 @@
################################
## Airflow Redis NetworkPolicy
#################################
-{{- if (and .Values.networkPolicies.enabled (eq .Values.executor "CeleryExecutor")) }}
+{{- if (and .Values.networkPolicies.enabled (or (eq .Values.executor "CeleryExecutor") (eq .Values.executor "CeleryKubernetesExecutor"))) }}
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
diff --git a/chart/templates/redis/redis-service.yaml b/chart/templates/redis/redis-service.yaml
index ac4bb21..b6e79db 100644
--- a/chart/templates/redis/redis-service.yaml
+++ b/chart/templates/redis/redis-service.yaml
@@ -18,7 +18,7 @@
################################
## Airflow Redis Service
#################################
-{{- if eq .Values.executor "CeleryExecutor" }}
+{{- if or (eq .Values.executor "CeleryExecutor") (eq .Values.executor "CeleryKubernetesExecutor") }}
kind: Service
apiVersion: v1
metadata:
diff --git a/chart/templates/redis/redis-statefulset.yaml b/chart/templates/redis/redis-statefulset.yaml
index e1eff0c..6cd7361 100644
--- a/chart/templates/redis/redis-statefulset.yaml
+++ b/chart/templates/redis/redis-statefulset.yaml
@@ -18,7 +18,7 @@
################################
## Airflow Redis StatefulSet
#################################
-{{- if eq .Values.executor "CeleryExecutor" }}
+{{- if or (eq .Values.executor "CeleryExecutor") (eq .Values.executor "CeleryKubernetesExecutor") }}
kind: StatefulSet
apiVersion: apps/v1
metadata:
diff --git a/chart/templates/scheduler/scheduler-deployment.yaml b/chart/templates/scheduler/scheduler-deployment.yaml
index f2b4a99..794a473 100644
--- a/chart/templates/scheduler/scheduler-deployment.yaml
+++ b/chart/templates/scheduler/scheduler-deployment.yaml
@@ -21,8 +21,6 @@
# Are we using a local/sequenial executor?
{{- $local := or (eq .Values.executor "LocalExecutor") (eq .Values.executor "SequentialExecutor") }}
-# Are we using the kubernetes executor?
-{{- $kube := eq .Values.executor "KubernetesExecutor" }}
# Is persistence enabled on the _workers_?
# This is important because in $local mode, the scheduler assumes the role of the worker
{{- $persistence := .Values.workers.persistence.enabled }}
diff --git a/chart/templates/workers/worker-deployment.yaml b/chart/templates/workers/worker-deployment.yaml
index fe07e20..38b3a9a 100644
--- a/chart/templates/workers/worker-deployment.yaml
+++ b/chart/templates/workers/worker-deployment.yaml
@@ -19,7 +19,7 @@
## Airflow Worker Deployment
#################################
{{- $persistence := .Values.workers.persistence.enabled }}
-{{- if eq .Values.executor "CeleryExecutor" }}
+{{- if or (eq .Values.executor "CeleryExecutor") (eq .Values.executor "CeleryKubernetesExecutor") }}
kind: {{ if $persistence }}StatefulSet{{ else }}Deployment{{ end }}
apiVersion: apps/v1
metadata:
diff --git a/chart/templates/workers/worker-kedaautoscaler.yaml b/chart/templates/workers/worker-kedaautoscaler.yaml
index 891afbf..c733618 100644
--- a/chart/templates/workers/worker-kedaautoscaler.yaml
+++ b/chart/templates/workers/worker-kedaautoscaler.yaml
@@ -18,7 +18,7 @@
################################
## Airflow Worker KEDA Scaler
#################################
-{{- if (and .Values.workers.keda.enabled (eq .Values.executor "CeleryExecutor")) }}
+{{- if (and .Values.workers.keda.enabled ( or (eq .Values.executor "CeleryExecutor")) (eq .Values.executor "CeleryKubernetesExecutor")) }}
apiVersion: keda.k8s.io/v1alpha1
kind: ScaledObject
metadata:
diff --git a/chart/templates/workers/worker-networkpolicy.yaml b/chart/templates/workers/worker-networkpolicy.yaml
index f6420e3..2c5e123 100644
--- a/chart/templates/workers/worker-networkpolicy.yaml
+++ b/chart/templates/workers/worker-networkpolicy.yaml
@@ -18,7 +18,7 @@
################################
## Airflow Worker NetworkPolicy
#################################
-{{- if (and .Values.networkPolicies.enabled (eq .Values.executor "CeleryExecutor")) }}
+{{- if (and .Values.networkPolicies.enabled ( or (eq .Values.executor "CeleryExecutor")) (eq .Values.executor "CeleryKubernetesExecutor")) }}
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
diff --git a/chart/templates/workers/worker-service.yaml b/chart/templates/workers/worker-service.yaml
index 2c7768f..8923f53 100644
--- a/chart/templates/workers/worker-service.yaml
+++ b/chart/templates/workers/worker-service.yaml
@@ -18,7 +18,7 @@
################################
## Airflow Worker Service
#################################
-{{- if eq .Values.executor "CeleryExecutor" }}
+{{- if or (eq .Values.executor "CeleryExecutor") (eq .Values.executor "CeleryKubernetesExecutor") }}
kind: Service
apiVersion: v1
metadata:
[airflow] 17/32: Consistent use images in Helm Chart (#11701)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 0df5e99b82e1de5fba72e77e20f5578809c7e6ad
Author: Kamil BreguĊa <mi...@users.noreply.github.com>
AuthorDate: Wed Oct 21 12:12:14 2020 +0200
Consistent use images in Helm Chart (#11701)
(cherry picked from commit 883eadfc1d13a16b662fd2b235917a8878c8e0f0)
---
chart/templates/_helpers.yaml | 7 +++-
.../templates/pgbouncer/pgbouncer-deployment.yaml | 5 +++
chart/templates/redis/redis-statefulset.yaml | 6 +++-
.../templates/scheduler/scheduler-deployment.yaml | 1 +
chart/templates/statsd/statsd-deployment.yaml | 6 +++-
chart/templates/workers/worker-deployment.yaml | 1 +
chart/tests/scheduler_test.yaml | 38 ----------------------
chart/tests/worker_test.yaml | 38 ----------------------
chart/values.schema.json | 26 ++++++++++-----
chart/values.yaml | 7 ++--
10 files changed, 45 insertions(+), 90 deletions(-)
diff --git a/chart/templates/_helpers.yaml b/chart/templates/_helpers.yaml
index 6dbdc04..beb26a7 100644
--- a/chart/templates/_helpers.yaml
+++ b/chart/templates/_helpers.yaml
@@ -104,7 +104,8 @@
{{/* Git sync container */}}
{{- define "git_sync_container"}}
- name: {{ .Values.dags.gitSync.containerName }}
- image: "{{ .Values.dags.gitSync.containerRepository }}:{{ .Values.dags.gitSync.containerTag }}"
+ image: {{ template "git_sync_image" . }}
+ imagePullPolicy: {{ .Values.images.gitSync.pullPolicy }}
securityContext:
runAsUser: {{ .Values.dags.gitSync.uid }}
env:
@@ -207,6 +208,10 @@
{{ printf "%s:%s" .Values.images.pgbouncerExporter.repository .Values.images.pgbouncerExporter.tag }}
{{- end }}
+{{ define "git_sync_image" -}}
+{{ printf "%s:%s" .Values.images.gitSync.repository .Values.images.gitSync.tag }}
+{{- end }}
+
{{ define "fernet_key_secret" -}}
{{ default (printf "%s-fernet-key" .Release.Name) .Values.fernetKeySecretName }}
{{- end }}
diff --git a/chart/templates/pgbouncer/pgbouncer-deployment.yaml b/chart/templates/pgbouncer/pgbouncer-deployment.yaml
index 9237e6c..18e1b21 100644
--- a/chart/templates/pgbouncer/pgbouncer-deployment.yaml
+++ b/chart/templates/pgbouncer/pgbouncer-deployment.yaml
@@ -63,6 +63,10 @@ spec:
tolerations:
{{ toYaml .Values.tolerations | indent 8 }}
restartPolicy: Always
+ {{- if or .Values.registry.secretName .Values.registry.connection }}
+ imagePullSecrets:
+ - name: {{ template "registry_secret" . }}
+ {{- end }}
containers:
- name: pgbouncer
image: {{ template "pgbouncer_image" . }}
@@ -117,6 +121,7 @@ spec:
command: ["/bin/sh", "-c", "killall -INT pgbouncer && sleep 120"]
- name: metrics-exporter
image: {{ template "pgbouncer_exporter_image" . }}
+ imagePullPolicy: {{ .Values.images.pgbouncerExporter.pullPolicy }}
env:
- name: DATABASE_URL
valueFrom:
diff --git a/chart/templates/redis/redis-statefulset.yaml b/chart/templates/redis/redis-statefulset.yaml
index 6cd7361..6df78b4 100644
--- a/chart/templates/redis/redis-statefulset.yaml
+++ b/chart/templates/redis/redis-statefulset.yaml
@@ -59,9 +59,13 @@ spec:
{{ toYaml .Values.affinity | indent 8 }}
tolerations:
{{ toYaml .Values.tolerations | indent 8 }}
+ {{- if or .Values.registry.secretName .Values.registry.connection }}
+ imagePullSecrets:
+ - name: {{ template "registry_secret" . }}
+ {{- end }}
containers:
- name: redis
- image: {{- include "redis_image" . | indent 1 }}
+ image: {{ template "redis_image" . }}
imagePullPolicy: {{ .Values.images.redis.pullPolicy }}
command: ["/bin/sh"]
resources:
diff --git a/chart/templates/scheduler/scheduler-deployment.yaml b/chart/templates/scheduler/scheduler-deployment.yaml
index 2cf3dce..1ed1c3a 100644
--- a/chart/templates/scheduler/scheduler-deployment.yaml
+++ b/chart/templates/scheduler/scheduler-deployment.yaml
@@ -168,6 +168,7 @@ spec:
# we don't have elasticsearch enabled.
- name: scheduler-logs
image: {{ template "airflow_image" . }}
+ imagePullPolicy: {{ .Values.images.airflow.pullPolicy }}
args: ["airflow", "serve_logs"]
ports:
- name: worker-logs
diff --git a/chart/templates/statsd/statsd-deployment.yaml b/chart/templates/statsd/statsd-deployment.yaml
index d7f5464..181c910 100644
--- a/chart/templates/statsd/statsd-deployment.yaml
+++ b/chart/templates/statsd/statsd-deployment.yaml
@@ -56,9 +56,13 @@ spec:
tolerations:
{{ toYaml .Values.tolerations | indent 8 }}
restartPolicy: Always
+ {{- if or .Values.registry.secretName .Values.registry.connection }}
+ imagePullSecrets:
+ - name: {{ template "registry_secret" . }}
+ {{- end }}
containers:
- name: statsd
- image: {{- include "statsd_image" . | indent 1 }}
+ image: {{ template "statsd_image" . }}
imagePullPolicy: {{ .Values.images.statsd.pullPolicy }}
args:
- "-statsd.mapping-config=/etc/statsd-exporter/mappings.yml"
diff --git a/chart/templates/workers/worker-deployment.yaml b/chart/templates/workers/worker-deployment.yaml
index 47aa18e..dc2dfb4 100644
--- a/chart/templates/workers/worker-deployment.yaml
+++ b/chart/templates/workers/worker-deployment.yaml
@@ -152,6 +152,7 @@ spec:
{{- if $persistence }}
- name: worker-gc
image: {{ template "airflow_image" . }}
+ imagePullPolicy: {{ .Values.images.airflow.pullPolicy }}
args: ["bash", "/clean-logs"]
volumeMounts:
- name: logs
diff --git a/chart/tests/scheduler_test.yaml b/chart/tests/scheduler_test.yaml
deleted file mode 100644
index 327b6e3..0000000
--- a/chart/tests/scheduler_test.yaml
+++ /dev/null
@@ -1,38 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
----
-templates:
- - scheduler/scheduler-deployment.yaml
-tests:
- - it: should add extraVolume and extraVolumeMount
- set:
- executor: CeleryExecutor
- scheduler:
- extraVolumes:
- - name: test-volume
- emptyDir: {}
- extraVolumeMounts:
- - name: test-volume
- mountPath: /opt/test
- asserts:
- - equal:
- path: spec.template.spec.volumes[1].name
- value: test-volume
- - equal:
- path: spec.template.spec.containers[0].volumeMounts[3].name
- value: test-volume
diff --git a/chart/tests/worker_test.yaml b/chart/tests/worker_test.yaml
deleted file mode 100644
index 29c4b97..0000000
--- a/chart/tests/worker_test.yaml
+++ /dev/null
@@ -1,38 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
----
-templates:
- - workers/worker-deployment.yaml
-tests:
- - it: should add extraVolume and extraVolumeMount
- set:
- executor: CeleryExecutor
- workers:
- extraVolumes:
- - name: test-volume
- emptyDir: {}
- extraVolumeMounts:
- - name: test-volume
- mountPath: /opt/test
- asserts:
- - equal:
- path: spec.template.spec.volumes[0].name
- value: test-volume
- - equal:
- path: spec.template.spec.containers[0].volumeMounts[0].name
- value: test-volume
diff --git a/chart/values.schema.json b/chart/values.schema.json
index b2f7288..7289374 100644
--- a/chart/values.schema.json
+++ b/chart/values.schema.json
@@ -288,6 +288,24 @@
"type": "string"
}
}
+ },
+ "gitSync": {
+ "description": "Configuration of the gitSync image.",
+ "type": "object",
+ "properties": {
+ "repository": {
+ "description": "The gitSync image repository.",
+ "type": "string"
+ },
+ "tag": {
+ "description": "The gitSync image tag.",
+ "type": "string"
+ },
+ "pullPolicy": {
+ "description": "The gitSync image pull policy.",
+ "type": "string"
+ }
+ }
}
}
},
@@ -1108,14 +1126,6 @@
"description": "Interval between git sync attempts in seconds.",
"type": "integer"
},
- "containerRepository": {
- "description": "Git sync image repository.",
- "type": "string"
- },
- "containerTag": {
- "description": "Git sync image tag.",
- "type": "string"
- },
"containerName": {
"description": "Git sync container name.",
"type": "string"
diff --git a/chart/values.yaml b/chart/values.yaml
index 1556341..b8bd0f8 100644
--- a/chart/values.yaml
+++ b/chart/values.yaml
@@ -147,6 +147,10 @@ images:
repository: apache/airflow
tag: airflow-pgbouncer-exporter-2020.09.25-0.5.0
pullPolicy: IfNotPresent
+ gitSync:
+ repository: k8s.gcr.io/git-sync
+ tag: v3.1.6
+ pullPolicy: IfNotPresent
# Environment variables for all airflow containers
env: []
@@ -734,8 +738,5 @@ dags:
# <host2>,<ip2> <key2>
# interval between git sync attempts in seconds
wait: 60
- # git sync image details
- containerRepository: k8s.gcr.io/git-sync
- containerTag: v3.1.6
containerName: git-sync
uid: 65533