You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by po...@apache.org on 2020/11/15 09:37:43 UTC
[airflow] branch v1-10-test updated (eeb9134 -> 36bfa04)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a change to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git.
omit eeb9134 Fix helm unit test for pod_template_file (#12345)
omit 1bd745e Mount airflow.cfg to pod_template_file (#12311)
omit fda5f7ad Fix indentation for affinities in helm chart (#12288)
omit 13442db Fix spelling (#12253)
omit 44b5077 Move metrics configuration to new section - metrics (#12165)
omit f750045 Fix default values for Helm Chart (#12153)
omit 265d155 Enable Black - Python Auto Formmatter (#9550)
omit a31ccb7 Use PyUpgrade to use Python 3.6 features (#11447)
omit a70cdd1 Add Kubernetes cleanup-pods CLI command for Helm Chart (#11802)
omit ffcbf6f fix helm scheduler deployment / scheduler logs (#11685)
new f75166e fix helm scheduler deployment / scheduler logs (#11685)
new 5d026bd Add Kubernetes cleanup-pods CLI command for Helm Chart (#11802)
new 564eb95 Use PyUpgrade to use Python 3.6 features (#11447)
new 16eb0e3 Enable Black - Python Auto Formmatter (#9550)
new 3fd9b5e Fix default values for Helm Chart (#12153)
new 8bd2df2 Move metrics configuration to new section - metrics (#12165)
new 5edc9c3 Fix spelling (#12253)
new 5334277 Fix indentation for affinities in helm chart (#12288)
new 2b9f5d4 Mount airflow.cfg to pod_template_file (#12311)
new 6767ef6 Fix helm unit test for pod_template_file (#12345)
new c4b9dbe Proposal: remove -serviceaccount suffix from KSA names in helm chart (#10892)
new 466db7f Chart: Flower deployment should use Flower image (#10701)
new ad3a259 use the correct claim name in the webserver (#9688)
new 465a3ae Run create-user-job as user with specified id (#10291)
new 771c4ec Add imagePullSecrets to the create user job (#9802)
new 65e3264 Wrong key in DAGs Persistent Volume Claim (#10627)
new 45e355c Update scheduler deployment - dags volume mount (#10630)
new f03a5ee Fix missing dash in flag for statsd container (#10691)
new 36bfa04 Add TargetQueryValue to KEDA Autoscaler (#9748)
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (eeb9134)
\
N -- N -- N refs/heads/v1-10-test (36bfa04)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 19 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
chart/README.md | 10 ++++++++--
chart/templates/_helpers.yaml | 4 ++--
chart/templates/cleanup/cleanup-cronjob.yaml | 2 +-
chart/templates/cleanup/cleanup-serviceaccount.yaml | 2 +-
chart/templates/create-user-job.yaml | 6 ++++++
chart/templates/dags-persistent-volume-claim.yaml | 6 +++---
chart/templates/rbac/pod-cleanup-rolebinding.yaml | 2 +-
chart/templates/rbac/pod-launcher-rolebinding.yaml | 4 ++--
chart/templates/scheduler/scheduler-deployment.yaml | 8 ++++++--
chart/templates/scheduler/scheduler-serviceaccount.yaml | 2 +-
chart/templates/statsd/statsd-deployment.yaml | 2 +-
chart/templates/webserver/webserver-deployment.yaml | 2 +-
chart/templates/workers/worker-deployment.yaml | 2 +-
chart/templates/workers/worker-kedaautoscaler.yaml | 1 +
chart/templates/workers/worker-serviceaccount.yaml | 2 +-
chart/tests/test_basic_helm_chart.py | 2 +-
16 files changed, 37 insertions(+), 20 deletions(-)
[airflow] 05/19: Fix default values for Helm Chart (#12153)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 3fd9b5e45369147852bbb41903c9df3d4dba7da4
Author: Kaxil Naik <ka...@gmail.com>
AuthorDate: Sat Nov 7 11:14:58 2020 +0000
Fix default values for Helm Chart (#12153)
(cherry picked from commit 874e497ff8b81121308018d02137361f66ae3c36)
---
chart/README.md | 12 ++++++------
1 file changed, 6 insertions(+), 6 deletions(-)
diff --git a/chart/README.md b/chart/README.md
index df8ea5d..db59db7 100644
--- a/chart/README.md
+++ b/chart/README.md
@@ -142,17 +142,17 @@ The following tables lists the configurable parameters of the Airflow chart and
| `images.flower.repository` | Docker repository to pull image from. Update this to deploy a custom image | `~` |
| `images.flower.tag` | Docker image tag to pull image from. Update this to deploy a new custom image tag | `~` |
| `images.flower.pullPolicy` | PullPolicy for flower image | `IfNotPresent` |
-| `images.statsd.repository` | Docker repository to pull image from. Update this to deploy a custom image | `astronomerinc/ap-statsd-exporter` |
-| `images.statsd.tag` | Docker image tag to pull image from. Update this to deploy a new custom image tag | `~` |
+| `images.statsd.repository` | Docker repository to pull image from. Update this to deploy a custom image | `apache/airflow` |
+| `images.statsd.tag` | Docker image tag to pull image from. Update this to deploy a new custom image tag | `airflow-statsd-exporter-2020.09.05-v0.17.0` |
| `images.statsd.pullPolicy` | PullPolicy for statsd-exporter image | `IfNotPresent` |
| `images.redis.repository` | Docker repository to pull image from. Update this to deploy a custom image | `redis` |
| `images.redis.tag` | Docker image tag to pull image from. Update this to deploy a new custom image tag | `6-buster` |
| `images.redis.pullPolicy` | PullPolicy for redis image | `IfNotPresent` |
-| `images.pgbouncer.repository` | Docker repository to pull image from. Update this to deploy a custom image | `astronomerinc/ap-pgbouncer` |
-| `images.pgbouncer.tag` | Docker image tag to pull image from. Update this to deploy a new custom image tag | `~` |
+| `images.pgbouncer.repository` | Docker repository to pull image from. Update this to deploy a custom image | `apache/airflow` |
+| `images.pgbouncer.tag` | Docker image tag to pull image from. Update this to deploy a new custom image tag | `airflow-pgbouncer-2020.09.05-1.14.0` |
| `images.pgbouncer.pullPolicy` | PullPolicy for pgbouncer image | `IfNotPresent` |
-| `images.pgbouncerExporter.repository` | Docker repository to pull image from. Update this to deploy a custom image | `astronomerinc/ap-pgbouncer-exporter` |
-| `images.pgbouncerExporter.tag` | Docker image tag to pull image from. Update this to deploy a new custom image tag | `~` |
+| `images.pgbouncerExporter.repository` | Docker repository to pull image from. Update this to deploy a custom image | `apache/airflow` |
+| `images.pgbouncerExporter.tag` | Docker image tag to pull image from. Update this to deploy a new custom image tag | `airflow-pgbouncer-exporter-2020.09.25-0.5.0` |
| `images.pgbouncerExporter.pullPolicy` | PullPolicy for pgbouncer-exporter image | `IfNotPresent` |
| `env` | Environment variables key/values to mount into Airflow pods | `[]` |
| `secret` | Secret name/key pairs to mount into Airflow pods | `[]` |
[airflow] 18/19: Fix missing dash in flag for statsd container
(#10691)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit f03a5ee902f127235d89fd855ea34ae309774603
Author: Kamil Olszewski <34...@users.noreply.github.com>
AuthorDate: Wed Sep 2 20:43:00 2020 +0200
Fix missing dash in flag for statsd container (#10691)
Co-authored-by: Kamil Olszewski <ka...@polidea.com>
(cherry picked from commit 48ce4bdac42af6a0bbd33148c14ac0bfc3568ce2)
---
chart/templates/statsd/statsd-deployment.yaml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/chart/templates/statsd/statsd-deployment.yaml b/chart/templates/statsd/statsd-deployment.yaml
index 181c910..d836e81 100644
--- a/chart/templates/statsd/statsd-deployment.yaml
+++ b/chart/templates/statsd/statsd-deployment.yaml
@@ -65,7 +65,7 @@ spec:
image: {{ template "statsd_image" . }}
imagePullPolicy: {{ .Values.images.statsd.pullPolicy }}
args:
- - "-statsd.mapping-config=/etc/statsd-exporter/mappings.yml"
+ - "--statsd.mapping-config=/etc/statsd-exporter/mappings.yml"
resources:
{{ toYaml .Values.statsd.resources | indent 12 }}
ports:
[airflow] 09/19: Mount airflow.cfg to pod_template_file (#12311)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 2b9f5d481b96e07fe421bbf0b5b7783651952eb4
Author: Daniel Imberman <da...@gmail.com>
AuthorDate: Thu Nov 12 20:58:13 2020 -0800
Mount airflow.cfg to pod_template_file (#12311)
* Mount airflow.cfg to pod_template_file
k8sexecutor workers were launching without an airflow.cfg,
this was preventing logs from being sent to distributed logging systems.
* consistent naming
(cherry picked from commit ce06be4752708b35b074832b0f227177cad055fd)
---
chart/files/pod-template-file.kubernetes-helm-yaml | 15 +++++++++++----
chart/tests/test_pod_template_file.py | 21 +++++++++++++++++++++
2 files changed, 32 insertions(+), 4 deletions(-)
diff --git a/chart/files/pod-template-file.kubernetes-helm-yaml b/chart/files/pod-template-file.kubernetes-helm-yaml
index b4ec9a5..5c4fb92 100644
--- a/chart/files/pod-template-file.kubernetes-helm-yaml
+++ b/chart/files/pod-template-file.kubernetes-helm-yaml
@@ -40,6 +40,16 @@ spec:
volumeMounts:
- mountPath: {{ template "airflow_logs" . }}
name: airflow-logs
+ - name: config
+ mountPath: {{ template "airflow_config_path" . }}
+ subPath: airflow.cfg
+ readOnly: true
+{{- if .Values.scheduler.airflowLocalSettings }}
+ - name: config
+ mountPath: {{ template "airflow_local_setting_path" . }}
+ subPath: airflow_local_settings.py
+ readOnly: true
+{{- end }}
{{- if .Values.dags.gitSync.knownHosts }}
- mountPath: /etc/git-secret/known_hosts
name: {{ .Values.dags.gitSync.knownHosts }}
@@ -92,7 +102,4 @@ spec:
{{- end }}
- configMap:
name: {{ include "airflow_config" . }}
- name: airflow-config
- - configMap:
- name: {{ include "airflow_config" . }}
- name: airflow-local-settings
+ name: config
diff --git a/chart/tests/test_pod_template_file.py b/chart/tests/test_pod_template_file.py
index 3c61733..fb0dd45 100644
--- a/chart/tests/test_pod_template_file.py
+++ b/chart/tests/test_pod_template_file.py
@@ -185,6 +185,27 @@ class PodTemplateFileTest(unittest.TestCase):
self.assertEqual("dummy_image:latest", jmespath.search("spec.containers[0].image", docs[0]))
self.assertEqual("base", jmespath.search("spec.containers[0].name", docs[0]))
+ def test_mount_airflow_cfg(self):
+ docs = render_chart(
+ values={},
+ show_only=["templates/pod-template-file.yaml"],
+ )
+
+ self.assertRegex(docs[0]["kind"], "Pod")
+ self.assertDictEqual(
+ {'configMap': {'name': 'RELEASE-NAME-airflow-config'}, 'name': 'airflow-config'},
+ jmespath.search("spec.volumes[1]", docs[0]),
+ )
+ self.assertDictEqual(
+ {
+ 'name': 'config',
+ 'mountPath': '/opt/airflow/airflow.cfg',
+ 'subPath': 'airflow.cfg',
+ 'readOnly': True,
+ },
+ jmespath.search("spec.containers[0].volumeMounts[1]", docs[0]),
+ )
+
def test_should_create_valid_affinity_and_node_selector(self):
docs = render_chart(
values={
[airflow] 15/19: Add imagePullSecrets to the create user job (#9802)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 771c4ecc8727d6a779cd203b4338dc3d858e0388
Author: Vicken Simonian <vs...@gmail.com>
AuthorDate: Thu Jul 16 06:50:22 2020 -0700
Add imagePullSecrets to the create user job (#9802)
So that it can pull the specified image from a private registry.
(cherry picked from commit 473f506336b359f6d993b3cd5f8260585afbcb62)
---
chart/templates/create-user-job.yaml | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/chart/templates/create-user-job.yaml b/chart/templates/create-user-job.yaml
index 656059b..27a0363 100644
--- a/chart/templates/create-user-job.yaml
+++ b/chart/templates/create-user-job.yaml
@@ -53,6 +53,10 @@ spec:
{{ toYaml .Values.affinity | indent 8 }}
tolerations:
{{ toYaml .Values.tolerations | indent 8 }}
+ {{- if or .Values.registry.secretName .Values.registry.connection }}
+ imagePullSecrets:
+ - name: {{ template "registry_secret" . }}
+ {{- end }}
containers:
- name: create-user
image: {{ template "airflow_image" . }}
[airflow] 03/19: Use PyUpgrade to use Python 3.6 features (#11447)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 564eb957c5111ae466a9f288a4505a55500692a5
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Sun Nov 15 03:41:17 2020 +0100
Use PyUpgrade to use Python 3.6 features (#11447)
Use features like `f-strings` instead of format across the code-base.
More details: https://github.com/asottile/pyupgrade
(cherry-picked from commit 8c42cf1b00c90f0d7f11b8a3a455381de8e003c5)
---
chart/tests/test_chart_quality.py | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/chart/tests/test_chart_quality.py b/chart/tests/test_chart_quality.py
index 3898941..32237cb 100644
--- a/chart/tests/test_chart_quality.py
+++ b/chart/tests/test_chart_quality.py
@@ -28,9 +28,9 @@ CHART_FOLDER = os.path.dirname(os.path.dirname(__file__))
class ChartQualityTest(unittest.TestCase):
def test_values_validate_schema(self):
- with open(os.path.join(CHART_FOLDER, "values.yaml"), "r") as f:
+ with open(os.path.join(CHART_FOLDER, "values.yaml")) as f:
values = yaml.safe_load(f)
- with open(os.path.join(CHART_FOLDER, "values.schema.json"), "r") as f:
+ with open(os.path.join(CHART_FOLDER, "values.schema.json")) as f:
schema = json.load(f)
# Add extra restrictions just for the tests to make sure
[airflow] 17/19: Update scheduler deployment - dags volume mount
(#10630)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 45e355c260017780b919dc90878db80eabb4eb76
Author: gardnerdev <ra...@gmail.com>
AuthorDate: Fri Aug 28 21:57:34 2020 +0200
Update scheduler deployment - dags volume mount (#10630)
(cherry picked from commit e495be9fd93d0f700f953fe5749260850ecd9b11)
---
chart/templates/scheduler/scheduler-deployment.yaml | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/chart/templates/scheduler/scheduler-deployment.yaml b/chart/templates/scheduler/scheduler-deployment.yaml
index 619a16c1..9a928a6 100644
--- a/chart/templates/scheduler/scheduler-deployment.yaml
+++ b/chart/templates/scheduler/scheduler-deployment.yaml
@@ -147,6 +147,10 @@ spec:
subPath: airflow_local_settings.py
readOnly: true
{{- end }}
+{{- if .Values.dags.persistence.enabled }}
+ - name: dags
+ mountPath: {{ template "airflow_dags_mount_path" . }}
+{{- end }}
{{- if .Values.dags.gitSync.enabled }}
- name: dags
mountPath: {{ template "airflow_dags_mount_path" . }}
[airflow] 07/19: Fix spelling (#12253)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 5edc9c3ae53c7d92341370e0c5bc0910f9903b12
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Sun Nov 15 03:48:38 2020 +0100
Fix spelling (#12253)
Author: John Bampton <jb...@users.noreply.github.com>
(cherry-picked from commit 4f9439dec13d3118d5423bac246064dea7a95002)
---
chart/README.md | 2 +-
chart/tests/test_celery_kubernetes_executor.py | 2 +-
chart/tests/test_git_sync_webserver.py | 4 ++--
chart/tests/test_git_sync_worker.py | 4 ++--
4 files changed, 6 insertions(+), 6 deletions(-)
diff --git a/chart/README.md b/chart/README.md
index db59db7..1ae5d6c 100644
--- a/chart/README.md
+++ b/chart/README.md
@@ -298,7 +298,7 @@ helm list -n airflow
```
Run `kubectl port-forward svc/airflow-webserver 8080:8080 -n airflow`
-to port-forward the Airflow UI to http://localhost:8080/ to cofirm Airflow is working.
+to port-forward the Airflow UI to http://localhost:8080/ to confirm Airflow is working.
**Build a Docker image from your DAGs:**
diff --git a/chart/tests/test_celery_kubernetes_executor.py b/chart/tests/test_celery_kubernetes_executor.py
index fb21929..6c54e80 100644
--- a/chart/tests/test_celery_kubernetes_executor.py
+++ b/chart/tests/test_celery_kubernetes_executor.py
@@ -23,7 +23,7 @@ from tests.helm_template_generator import render_chart
class CeleryKubernetesExecutorTest(unittest.TestCase):
- def test_should_create_a_worker_deployment_with_the_celery_executo(self):
+ def test_should_create_a_worker_deployment_with_the_celery_executor(self):
docs = render_chart(
values={
"executor": "CeleryExecutor",
diff --git a/chart/tests/test_git_sync_webserver.py b/chart/tests/test_git_sync_webserver.py
index 75ec51b..09c9aa3 100644
--- a/chart/tests/test_git_sync_webserver.py
+++ b/chart/tests/test_git_sync_webserver.py
@@ -23,7 +23,7 @@ from tests.helm_template_generator import render_chart
class GitSyncWebserverTest(unittest.TestCase):
- def test_should_add_dags_volume_to_the_webserver_if_git_sync_and_peristence_is_enabled(self):
+ def test_should_add_dags_volume_to_the_webserver_if_git_sync_and_persistence_is_enabled(self):
docs = render_chart(
values={"dags": {"gitSync": {"enabled": True}, "persistence": {"enabled": True}}},
show_only=["templates/webserver/webserver-deployment.yaml"],
@@ -31,7 +31,7 @@ class GitSyncWebserverTest(unittest.TestCase):
self.assertEqual("dags", jmespath.search("spec.template.spec.volumes[1].name", docs[0]))
- def test_should_add_dags_volume_to_the_webserver_if_git_sync_is_enabled_and_peristence_is_disabled(self):
+ def test_should_add_dags_volume_to_the_webserver_if_git_sync_is_enabled_and_persistence_is_disabled(self):
docs = render_chart(
values={"dags": {"gitSync": {"enabled": True}, "persistence": {"enabled": False}}},
show_only=["templates/webserver/webserver-deployment.yaml"],
diff --git a/chart/tests/test_git_sync_worker.py b/chart/tests/test_git_sync_worker.py
index e5036d7..a56b0dc 100644
--- a/chart/tests/test_git_sync_worker.py
+++ b/chart/tests/test_git_sync_worker.py
@@ -23,7 +23,7 @@ from tests.helm_template_generator import render_chart
class GitSyncWorkerTest(unittest.TestCase):
- def test_should_add_dags_volume_to_the_worker_if_git_sync_and_peristence_is_enabled(self):
+ def test_should_add_dags_volume_to_the_worker_if_git_sync_and_persistence_is_enabled(self):
docs = render_chart(
values={
"executor": "CeleryExecutor",
@@ -35,7 +35,7 @@ class GitSyncWorkerTest(unittest.TestCase):
self.assertEqual("config", jmespath.search("spec.template.spec.volumes[0].name", docs[0]))
self.assertEqual("dags", jmespath.search("spec.template.spec.volumes[1].name", docs[0]))
- def test_should_add_dags_volume_to_the_worker_if_git_sync_is_enabled_and_peristence_is_disabled(self):
+ def test_should_add_dags_volume_to_the_worker_if_git_sync_is_enabled_and_persistence_is_disabled(self):
docs = render_chart(
values={
"executor": "CeleryExecutor",
[airflow] 06/19: Move metrics configuration to new section -
metrics (#12165)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 8bd2df205017851bd402535fbdcbd66c5c73dbd0
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Sun Nov 15 03:46:54 2020 +0100
Move metrics configuration to new section - metrics (#12165)
* Move metrics configuration to new section
* fixup! Move metrics configuration to new section
* fixup! fixup! Move metrics configuration to new section
* Apply suggestions from code review
Co-authored-by: Xiaodong DENG <xd...@gmail.com>
* fixup! Apply suggestions from code review
Co-authored-by: Xiaodong DENG <xd...@gmail.com>
(cherry-picked from commit 833ba453de9490bce742ebf4b648eec48caf2037)
---
chart/values.yaml | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/chart/values.yaml b/chart/values.yaml
index 1e3b81f..0f5b313 100644
--- a/chart/values.yaml
+++ b/chart/values.yaml
@@ -600,6 +600,11 @@ config:
auth_backend: airflow.api.auth.backend.deny_all
logging:
logging_level: DEBUG
+ metrics:
+ statsd_on: '{{ ternary "True" "False" .Values.statsd.enabled }}'
+ statsd_port: 9125
+ statsd_prefix: airflow
+ statsd_host: '{{ printf "%s-statsd" .Release.Name }}'
webserver:
enable_proxy_fix: 'True'
expose_config: 'True'
@@ -610,6 +615,7 @@ config:
scheduler:
scheduler_heartbeat_sec: 5
+ # For Airflow 1.10, backward compatibility
statsd_on: '{{ ternary "True" "False" .Values.statsd.enabled }}'
statsd_port: 9125
statsd_prefix: airflow
[airflow] 02/19: Add Kubernetes cleanup-pods CLI command for Helm
Chart (#11802)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 5d026bd8a7b8e41355b993089b0bee655cd22795
Author: Kaxil Naik <ka...@gmail.com>
AuthorDate: Tue Nov 3 15:28:51 2020 +0000
Add Kubernetes cleanup-pods CLI command for Helm Chart (#11802)
closes: https://github.com/apache/airflow/issues/11146
(cherry picked from commit 980c7252c0f28c251e9f87d736cd88d6027f3da3)
---
chart/templates/cleanup/cleanup-cronjob.yaml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/chart/templates/cleanup/cleanup-cronjob.yaml b/chart/templates/cleanup/cleanup-cronjob.yaml
index 3f44976..3ad3d4f 100644
--- a/chart/templates/cleanup/cleanup-cronjob.yaml
+++ b/chart/templates/cleanup/cleanup-cronjob.yaml
@@ -61,7 +61,7 @@ spec:
image: {{ template "default_airflow_image" . }}
imagePullPolicy: {{ .Values.images.airflow.pullPolicy }}
# Don't use entry point here, we don't need to wait on pg-bouncer etc being available.
- command: ["airflow-cleanup-pods", "--namespace={{ .Release.Namespace }}"]
+ args: ["kubernetes", "cleanup-pods", "--namespace={{ .Release.Namespace }}"]
env:
{{- include "standard_airflow_environment" . | indent 12 }}
{{- end }}
[airflow] 04/19: Enable Black - Python Auto Formmatter (#9550)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 16eb0e380b57eac6259bf69eb91ab9739123e8bf
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Sun Nov 15 03:42:47 2020 +0100
Enable Black - Python Auto Formmatter (#9550)
(cherry-picked from commit 4e8f9cc8d02b29c325b8a5a76b4837671bdf5f68)
---
chart/tests/test_celery_kubernetes_executor.py | 1 +
chart/tests/test_celery_kubernetes_pod_launcher_role.py | 1 +
chart/tests/test_chart_quality.py | 3 +--
chart/tests/test_dags_persistent_volume_claim.py | 1 +
chart/tests/test_flower_authorization.py | 1 +
chart/tests/test_git_sync_scheduler.py | 1 +
chart/tests/test_git_sync_webserver.py | 1 +
chart/tests/test_git_sync_worker.py | 1 +
chart/tests/test_migrate_database_job.py | 1 +
chart/tests/test_pod_template_file.py | 3 ++-
chart/tests/test_scheduler.py | 1 +
chart/tests/test_worker.py | 1 +
12 files changed, 13 insertions(+), 3 deletions(-)
diff --git a/chart/tests/test_celery_kubernetes_executor.py b/chart/tests/test_celery_kubernetes_executor.py
index 57c3980..fb21929 100644
--- a/chart/tests/test_celery_kubernetes_executor.py
+++ b/chart/tests/test_celery_kubernetes_executor.py
@@ -18,6 +18,7 @@
import unittest
import jmespath
+
from tests.helm_template_generator import render_chart
diff --git a/chart/tests/test_celery_kubernetes_pod_launcher_role.py b/chart/tests/test_celery_kubernetes_pod_launcher_role.py
index 535be11..952bc39 100644
--- a/chart/tests/test_celery_kubernetes_pod_launcher_role.py
+++ b/chart/tests/test_celery_kubernetes_pod_launcher_role.py
@@ -18,6 +18,7 @@
import unittest
import jmespath
+
from tests.helm_template_generator import render_chart
diff --git a/chart/tests/test_chart_quality.py b/chart/tests/test_chart_quality.py
index 32237cb..5f17165 100644
--- a/chart/tests/test_chart_quality.py
+++ b/chart/tests/test_chart_quality.py
@@ -18,11 +18,10 @@
import json
import os
import unittest
-import yaml
+import yaml
from jsonschema import validate
-
CHART_FOLDER = os.path.dirname(os.path.dirname(__file__))
diff --git a/chart/tests/test_dags_persistent_volume_claim.py b/chart/tests/test_dags_persistent_volume_claim.py
index 069a0cd..946c40f 100644
--- a/chart/tests/test_dags_persistent_volume_claim.py
+++ b/chart/tests/test_dags_persistent_volume_claim.py
@@ -18,6 +18,7 @@
import unittest
import jmespath
+
from tests.helm_template_generator import render_chart
diff --git a/chart/tests/test_flower_authorization.py b/chart/tests/test_flower_authorization.py
index f0cc5b0..0520ddd 100644
--- a/chart/tests/test_flower_authorization.py
+++ b/chart/tests/test_flower_authorization.py
@@ -18,6 +18,7 @@
import unittest
import jmespath
+
from tests.helm_template_generator import render_chart
diff --git a/chart/tests/test_git_sync_scheduler.py b/chart/tests/test_git_sync_scheduler.py
index a01c0f2..068f36c 100644
--- a/chart/tests/test_git_sync_scheduler.py
+++ b/chart/tests/test_git_sync_scheduler.py
@@ -18,6 +18,7 @@
import unittest
import jmespath
+
from tests.helm_template_generator import render_chart
diff --git a/chart/tests/test_git_sync_webserver.py b/chart/tests/test_git_sync_webserver.py
index 30c3f33..75ec51b 100644
--- a/chart/tests/test_git_sync_webserver.py
+++ b/chart/tests/test_git_sync_webserver.py
@@ -18,6 +18,7 @@
import unittest
import jmespath
+
from tests.helm_template_generator import render_chart
diff --git a/chart/tests/test_git_sync_worker.py b/chart/tests/test_git_sync_worker.py
index a70d311..e5036d7 100644
--- a/chart/tests/test_git_sync_worker.py
+++ b/chart/tests/test_git_sync_worker.py
@@ -18,6 +18,7 @@
import unittest
import jmespath
+
from tests.helm_template_generator import render_chart
diff --git a/chart/tests/test_migrate_database_job.py b/chart/tests/test_migrate_database_job.py
index 0524315..4b92aca 100644
--- a/chart/tests/test_migrate_database_job.py
+++ b/chart/tests/test_migrate_database_job.py
@@ -18,6 +18,7 @@
import unittest
import jmespath
+
from tests.helm_template_generator import render_chart
diff --git a/chart/tests/test_pod_template_file.py b/chart/tests/test_pod_template_file.py
index 0673e08..d9334de 100644
--- a/chart/tests/test_pod_template_file.py
+++ b/chart/tests/test_pod_template_file.py
@@ -17,10 +17,11 @@
import unittest
from os import remove
-from os.path import realpath, dirname
+from os.path import dirname, realpath
from shutil import copyfile
import jmespath
+
from tests.helm_template_generator import render_chart
ROOT_FOLDER = realpath(dirname(realpath(__file__)) + "/..")
diff --git a/chart/tests/test_scheduler.py b/chart/tests/test_scheduler.py
index 9769848..eb5225e 100644
--- a/chart/tests/test_scheduler.py
+++ b/chart/tests/test_scheduler.py
@@ -18,6 +18,7 @@
import unittest
import jmespath
+
from tests.helm_template_generator import render_chart
diff --git a/chart/tests/test_worker.py b/chart/tests/test_worker.py
index 2fc6d17..9b3515e 100644
--- a/chart/tests/test_worker.py
+++ b/chart/tests/test_worker.py
@@ -18,6 +18,7 @@
import unittest
import jmespath
+
from tests.helm_template_generator import render_chart
[airflow] 08/19: Fix indentation for affinities in helm chart
(#12288)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 53342779b7a4548fe9930dc4c52fa7cac9f1e198
Author: Daniel Imberman <da...@gmail.com>
AuthorDate: Wed Nov 11 09:23:20 2020 -0800
Fix indentation for affinities in helm chart (#12288)
This PR fixes a bug in the helm chart where custom affinities in
the pod_template_file cause the yaml to fail due to invalid spacing
(cherry picked from commit cbe4ef2c5e143c87c9edd1c54a4949bbdd7a8edd)
---
chart/files/pod-template-file.kubernetes-helm-yaml | 9 ++--
chart/tests/test_pod_template_file.py | 51 ++++++++++++++++++++++
2 files changed, 54 insertions(+), 6 deletions(-)
diff --git a/chart/files/pod-template-file.kubernetes-helm-yaml b/chart/files/pod-template-file.kubernetes-helm-yaml
index 8647060..b4ec9a5 100644
--- a/chart/files/pod-template-file.kubernetes-helm-yaml
+++ b/chart/files/pod-template-file.kubernetes-helm-yaml
@@ -66,12 +66,9 @@ spec:
restartPolicy: Never
securityContext:
runAsUser: {{ .Values.uid }}
- nodeSelector:
- {{ toYaml .Values.nodeSelector | indent 8 }}
- affinity:
- {{ toYaml .Values.affinity | indent 8 }}
- tolerations:
- {{ toYaml .Values.tolerations | indent 8 }}
+ nodeSelector: {{ toYaml .Values.nodeSelector | nindent 4 }}
+ affinity: {{ toYaml .Values.affinity | nindent 4 }}
+ tolerations: {{ toYaml .Values.tolerations | nindent 4 }}
serviceAccountName: '{{ .Release.Name }}-worker'
volumes:
{{- if .Values.dags.persistence.enabled }}
diff --git a/chart/tests/test_pod_template_file.py b/chart/tests/test_pod_template_file.py
index d9334de..3c61733 100644
--- a/chart/tests/test_pod_template_file.py
+++ b/chart/tests/test_pod_template_file.py
@@ -184,3 +184,54 @@ class PodTemplateFileTest(unittest.TestCase):
self.assertRegex(docs[0]["kind"], "Pod")
self.assertEqual("dummy_image:latest", jmespath.search("spec.containers[0].image", docs[0]))
self.assertEqual("base", jmespath.search("spec.containers[0].name", docs[0]))
+
+ def test_should_create_valid_affinity_and_node_selector(self):
+ docs = render_chart(
+ values={
+ "affinity": {
+ "nodeAffinity": {
+ "requiredDuringSchedulingIgnoredDuringExecution": {
+ "nodeSelectorTerms": [
+ {
+ "matchExpressions": [
+ {"key": "foo", "operator": "In", "values": ["true"]},
+ ]
+ }
+ ]
+ }
+ }
+ },
+ "tolerations": [
+ {"key": "dynamic-pods", "operator": "Equal", "value": "true", "effect": "NoSchedule"}
+ ],
+ "nodeSelector": {"diskType": "ssd"},
+ },
+ show_only=["templates/pod-template-file.yaml"],
+ )
+
+ self.assertRegex(docs[0]["kind"], "Pod")
+ self.assertEqual(
+ "foo",
+ jmespath.search(
+ "spec.affinity.nodeAffinity."
+ "requiredDuringSchedulingIgnoredDuringExecution."
+ "nodeSelectorTerms[0]."
+ "matchExpressions[0]."
+ "key",
+ docs[0],
+ ),
+ )
+ self.assertEqual(
+ "ssd",
+ jmespath.search(
+ "spec.nodeSelector.diskType",
+ docs[0],
+ ),
+ )
+ self.assertEqual(
+ "dynamic-pods",
+ jmespath.search(
+ "spec.tolerations[0].key",
+ docs[0],
+ ),
+ )
[airflow] 11/19: Proposal: remove -serviceaccount suffix from KSA
names in helm chart (#10892)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit c4b9dbe53aa585fc8184913412414655a3353fa2
Author: Jacob Ferriero <jf...@google.com>
AuthorDate: Tue Sep 15 03:39:39 2020 -0700
Proposal: remove -serviceaccount suffix from KSA names in helm chart (#10892)
* [WIP] remove -serviceaccount suffix in helm chart
It's quite annoying to have `-serviceaccount` in each service account name as this is a useless 15 characters that provides no additional information.
"why is this so frustrating to you Jake?"
GCP service accounts have 30 char name limit https://cloud.google.com/iam/docs/creating-managing-service-accounts#creating
For manageability / clarity I'd like to keep KSA and GSA names exactly the same when using workload identity which maps KSA<>GSA 1:1 https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity.
(cherry picked from commit 23768f694697be1b29ec00b8deeed5777d2538e8)
---
chart/README.md | 10 ++++++++--
chart/templates/cleanup/cleanup-cronjob.yaml | 2 +-
chart/templates/cleanup/cleanup-serviceaccount.yaml | 2 +-
chart/templates/rbac/pod-cleanup-rolebinding.yaml | 2 +-
chart/templates/rbac/pod-launcher-rolebinding.yaml | 4 ++--
chart/templates/scheduler/scheduler-deployment.yaml | 4 ++--
chart/templates/scheduler/scheduler-serviceaccount.yaml | 2 +-
chart/templates/workers/worker-deployment.yaml | 2 +-
chart/templates/workers/worker-serviceaccount.yaml | 2 +-
9 files changed, 18 insertions(+), 12 deletions(-)
diff --git a/chart/README.md b/chart/README.md
index 1ae5d6c..d56f114 100644
--- a/chart/README.md
+++ b/chart/README.md
@@ -35,6 +35,7 @@ cluster using the [Helm](https://helm.sh) package manager.
## Installing the Chart
To install this repository from source (using helm 3)
+
```bash
kubectl create namespace airflow
helm repo add stable https://charts.helm.sh/stable/
@@ -48,6 +49,7 @@ section lists the parameters that can be configured during installation.
> **Tip**: List all releases using `helm list`
## Upgrading the Chart
+
To upgrade the chart with the release name `airflow`:
```bash
@@ -90,6 +92,7 @@ helm upgrade airflow . \
```
## Mounting DAGS using Git-Sync side car without Persistence
+
This option will use an always running Git-Sync side car on every scheduler,webserver and worker pods. The Git-Sync side car containers will sync DAGs from a git repository every configured number of seconds. If you are using the KubernetesExecutor, Git-sync will run as an initContainer on your worker pods.
```bash
@@ -102,6 +105,7 @@ helm upgrade airflow . \
```
## Mounting DAGS from an externally populated PVC
+
In this approach, Airflow will read the DAGs from a PVC which has `ReadOnlyMany` or `ReadWriteMany` accessMode. You will have to ensure that the PVC is populated/updated with the required DAGs(this won't be handled by the chart). You can pass in the name of the volume claim to the chart
```bash
@@ -213,12 +217,14 @@ The following tables lists the configurable parameters of the Airflow chart and
| `webserver.resources.limits.memory` | Memory Limit of webserver | `~` |
| `webserver.resources.requests.cpu` | CPU Request of webserver | `~` |
| `webserver.resources.requests.memory` | Memory Request of webserver | `~` |
+| `webserver.service.annotations` | Annotations to be added to the webserver service | `{}` |
| `webserver.defaultUser` | Optional default airflow user information | `{}` |
-| `dags.persistence.*` | Dag persistence configuration | Please refer to `values.yaml` |
-| `dags.gitSync.*` | Git sync configuration | Please refer to `values.yaml` |
+| `dags.persistence.*` | Dag persistence configuration | Please refer to `values.yaml` |
+| `dags.gitSync.*` | Git sync configuration | Please refer to `values.yaml` |
| `multiNamespaceMode` | Whether the KubernetesExecutor can launch pods in multiple namespaces | `False` |
| `serviceAccountAnnottions.*` | Map of annotations for worker, webserver, scheduler kubernetes service accounts | {} |
+
Specify each parameter using the `--set key=value[,key=value]` argument to `helm install`. For example,
```bash
diff --git a/chart/templates/cleanup/cleanup-cronjob.yaml b/chart/templates/cleanup/cleanup-cronjob.yaml
index 3ad3d4f..fd253d2 100644
--- a/chart/templates/cleanup/cleanup-cronjob.yaml
+++ b/chart/templates/cleanup/cleanup-cronjob.yaml
@@ -51,7 +51,7 @@ spec:
{{ toYaml .Values.affinity | indent 12 }}
tolerations:
{{ toYaml .Values.tolerations | indent 12 }}
- serviceAccountName: {{ .Release.Name }}-cleanup-serviceaccount
+ serviceAccountName: {{ .Release.Name }}-cleanup
{{- if or .Values.registry.secretName .Values.registry.connection }}
imagePullSecrets:
- name: {{ template "registry_secret" . }}
diff --git a/chart/templates/cleanup/cleanup-serviceaccount.yaml b/chart/templates/cleanup/cleanup-serviceaccount.yaml
index 769cbdc..6ef3aec 100644
--- a/chart/templates/cleanup/cleanup-serviceaccount.yaml
+++ b/chart/templates/cleanup/cleanup-serviceaccount.yaml
@@ -22,7 +22,7 @@
kind: ServiceAccount
apiVersion: v1
metadata:
- name: {{ .Release.Name }}-cleanup-serviceaccount
+ name: {{ .Release.Name }}-cleanup
labels:
tier: airflow
release: {{ .Release.Name }}
diff --git a/chart/templates/rbac/pod-cleanup-rolebinding.yaml b/chart/templates/rbac/pod-cleanup-rolebinding.yaml
index 4c2dd25..0d09b87 100644
--- a/chart/templates/rbac/pod-cleanup-rolebinding.yaml
+++ b/chart/templates/rbac/pod-cleanup-rolebinding.yaml
@@ -37,6 +37,6 @@ roleRef:
name: {{ .Release.Name }}-cleanup-role
subjects:
- kind: ServiceAccount
- name: {{ .Release.Name }}-cleanup-serviceaccount
+ name: {{ .Release.Name }}-cleanup
namespace: {{ .Release.Namespace }}
{{- end }}
diff --git a/chart/templates/rbac/pod-launcher-rolebinding.yaml b/chart/templates/rbac/pod-launcher-rolebinding.yaml
index f258de7..849f32d 100644
--- a/chart/templates/rbac/pod-launcher-rolebinding.yaml
+++ b/chart/templates/rbac/pod-launcher-rolebinding.yaml
@@ -51,12 +51,12 @@ roleRef:
subjects:
{{- if $grantScheduler }}
- kind: ServiceAccount
- name: {{ .Release.Name }}-scheduler-serviceaccount
+ name: {{ .Release.Name }}-scheduler
namespace: {{ .Release.Namespace }}
{{- end }}
{{- if $grantWorker }}
- kind: ServiceAccount
- name: {{ .Release.Name }}-worker-serviceaccount
+ name: {{ .Release.Name }}-worker
namespace: {{ .Release.Namespace }}
{{- end }}
{{- end }}
diff --git a/chart/templates/scheduler/scheduler-deployment.yaml b/chart/templates/scheduler/scheduler-deployment.yaml
index 771dd1a..619a16c1 100644
--- a/chart/templates/scheduler/scheduler-deployment.yaml
+++ b/chart/templates/scheduler/scheduler-deployment.yaml
@@ -19,7 +19,7 @@
## Airflow Scheduler Deployment/StatefulSet
#################################
-# Are we using a local/sequenial executor?
+# Are we using a local/sequential executor?
{{- $local := or (eq .Values.executor "LocalExecutor") (eq .Values.executor "SequentialExecutor") }}
# Is persistence enabled on the _workers_?
# This is important because in $local mode, the scheduler assumes the role of the worker
@@ -81,7 +81,7 @@ spec:
{{ toYaml .Values.tolerations | indent 8 }}
restartPolicy: Always
terminationGracePeriodSeconds: 10
- serviceAccountName: {{ .Release.Name }}-scheduler-serviceaccount
+ serviceAccountName: {{ .Release.Name }}-scheduler
securityContext:
runAsUser: {{ .Values.uid }}
fsGroup: {{ .Values.gid }}
diff --git a/chart/templates/scheduler/scheduler-serviceaccount.yaml b/chart/templates/scheduler/scheduler-serviceaccount.yaml
index f12991d..2d61c76 100644
--- a/chart/templates/scheduler/scheduler-serviceaccount.yaml
+++ b/chart/templates/scheduler/scheduler-serviceaccount.yaml
@@ -22,7 +22,7 @@
kind: ServiceAccount
apiVersion: v1
metadata:
- name: {{ .Release.Name }}-scheduler-serviceaccount
+ name: {{ .Release.Name }}-scheduler
labels:
tier: airflow
release: {{ .Release.Name }}
diff --git a/chart/templates/workers/worker-deployment.yaml b/chart/templates/workers/worker-deployment.yaml
index 53998da..77d5fe2 100644
--- a/chart/templates/workers/worker-deployment.yaml
+++ b/chart/templates/workers/worker-deployment.yaml
@@ -72,7 +72,7 @@ spec:
{{ toYaml .Values.tolerations | indent 8 }}
terminationGracePeriodSeconds: {{ .Values.workers.terminationGracePeriodSeconds }}
restartPolicy: Always
- serviceAccountName: {{ .Release.Name }}-worker-serviceaccount
+ serviceAccountName: {{ .Release.Name }}-worker
securityContext:
runAsUser: {{ .Values.uid }}
fsGroup: {{ .Values.gid }}
diff --git a/chart/templates/workers/worker-serviceaccount.yaml b/chart/templates/workers/worker-serviceaccount.yaml
index f87751b..d7ed927 100644
--- a/chart/templates/workers/worker-serviceaccount.yaml
+++ b/chart/templates/workers/worker-serviceaccount.yaml
@@ -22,7 +22,7 @@
kind: ServiceAccount
apiVersion: v1
metadata:
- name: {{ .Release.Name }}-worker-serviceaccount
+ name: {{ .Release.Name }}-worker
labels:
tier: airflow
release: {{ .Release.Name }}
[airflow] 01/19: fix helm scheduler deployment / scheduler logs
(#11685)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit f75166e32ef3269c252bb36a7c1a8ac0689f09ce
Author: Florent Chehab <fc...@meilleursagents.com>
AuthorDate: Sat Oct 31 18:55:50 2020 +0100
fix helm scheduler deployment / scheduler logs (#11685)
Based on the airflow image entrypoint, we should use airflow commands directly.
The container exits otherwise.
(cherry picked from commit 069b1f71cf9fbeaea3af2dfdc14681996dd09f4b)
---
.../templates/scheduler/scheduler-deployment.yaml | 2 +-
chart/tests/test_basic_helm_chart.py | 31 ++++++++++++++++++++++
2 files changed, 32 insertions(+), 1 deletion(-)
diff --git a/chart/templates/scheduler/scheduler-deployment.yaml b/chart/templates/scheduler/scheduler-deployment.yaml
index 1ed1c3a..771dd1a 100644
--- a/chart/templates/scheduler/scheduler-deployment.yaml
+++ b/chart/templates/scheduler/scheduler-deployment.yaml
@@ -169,7 +169,7 @@ spec:
- name: scheduler-logs
image: {{ template "airflow_image" . }}
imagePullPolicy: {{ .Values.images.airflow.pullPolicy }}
- args: ["airflow", "serve_logs"]
+ args: ["serve_logs"]
ports:
- name: worker-logs
containerPort: {{ .Values.ports.workerLogs }}
diff --git a/chart/tests/test_basic_helm_chart.py b/chart/tests/test_basic_helm_chart.py
index 767a073..a9cfa16 100644
--- a/chart/tests/test_basic_helm_chart.py
+++ b/chart/tests/test_basic_helm_chart.py
@@ -16,6 +16,7 @@
# under the License.
import unittest
+from typing import Any, Dict, List, Union
import jmespath
@@ -84,3 +85,33 @@ class TestBaseChartTest(unittest.TestCase):
]
self.assertNotIn(('Job', 'TEST-BASIC-create-user'), list_of_kind_names_tuples)
self.assertEqual(OBJECT_COUNT_IN_BASIC_DEPLOYMENT - 1, len(k8s_objects))
+
+ def test_chart_is_consistent_with_official_airflow_image(self):
+ def get_k8s_objs_with_image(obj: Union[List[Any], Dict[str, Any]]) -> List[Dict[str, Any]]:
+ """
+ Recursive helper to retrieve all the k8s objects that have an "image" key
+ inside k8s obj or list of k8s obj
+ """
+ out: List[Dict[str, Any]] = []
+ if isinstance(obj, list):
+ for item in obj:
+ out += get_k8s_objs_with_image(item)
+ if isinstance(obj, dict):
+ if "image" in obj:
+ out += [obj]
+ # include sub objs, just in case
+ for val in obj.values():
+ out += get_k8s_objs_with_image(val)
+ return out
+
+ image_repo = "test-airflow-repo/airflow"
+ k8s_objects = render_chart("TEST-BASIC", {"defaultAirflowRepository": image_repo})
+
+ objs_with_image = get_k8s_objs_with_image(k8s_objects)
+ for obj in objs_with_image:
+ image: str = obj["image"] # pylint: disable=invalid-sequence-index
+ if image.startswith(image_repo):
+ # Make sure that a command is not specified
+ self.assertNotIn("command", obj)
+ # Make sure that the first arg is never airflow
+ self.assertNotEqual(obj["args"][0], "airflow") # pylint: disable=invalid-sequence-index
[airflow] 10/19: Fix helm unit test for pod_template_file (#12345)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 6767ef67bf1955d9df9bfadad87a631c28667b05
Author: Daniel Imberman <da...@gmail.com>
AuthorDate: Fri Nov 13 10:05:24 2020 -0800
Fix helm unit test for pod_template_file (#12345)
Fixes bug in unittest that is causing master to fail.
(cherry picked from commit 64978e323f0d6951a079f1dbe7d6b80127ffc19d)
---
chart/tests/test_pod_template_file.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/chart/tests/test_pod_template_file.py b/chart/tests/test_pod_template_file.py
index fb0dd45..768638f 100644
--- a/chart/tests/test_pod_template_file.py
+++ b/chart/tests/test_pod_template_file.py
@@ -193,7 +193,7 @@ class PodTemplateFileTest(unittest.TestCase):
self.assertRegex(docs[0]["kind"], "Pod")
self.assertDictEqual(
- {'configMap': {'name': 'RELEASE-NAME-airflow-config'}, 'name': 'airflow-config'},
+ {'configMap': {'name': 'RELEASE-NAME-airflow-config'}, 'name': 'config'},
jmespath.search("spec.volumes[1]", docs[0]),
)
self.assertDictEqual(
[airflow] 12/19: Chart: Flower deployment should use Flower image
(#10701)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 466db7f60be36017aa71e954bd2538ae1cbf6dbb
Author: Kaxil Naik <ka...@gmail.com>
AuthorDate: Wed Sep 2 20:34:03 2020 +0100
Chart: Flower deployment should use Flower image (#10701)
Co-authored-by: Steven Miller <sj...@gmail.com>
(cherry picked from commit e5785d4720d94ed3183bd46769ffdb54830b0d22)
---
chart/templates/_helpers.yaml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/chart/templates/_helpers.yaml b/chart/templates/_helpers.yaml
index 9914127..929861e 100644
--- a/chart/templates/_helpers.yaml
+++ b/chart/templates/_helpers.yaml
@@ -189,7 +189,7 @@
{{- end }}
{{ define "flower_image" -}}
-{{ printf "%s:%s" (.Values.images.airflow.repository | default .Values.defaultAirflowRepository) (.Values.images.airflow.tag | default .Values.defaultAirflowTag) }}
+{{ printf "%s:%s" (.Values.images.flower.repository | default .Values.defaultAirflowRepository) (.Values.images.flower.tag | default .Values.defaultAirflowTag) }}
{{- end }}
{{ define "statsd_image" -}}
[airflow] 16/19: Wrong key in DAGs Persistent Volume Claim (#10627)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 65e326406d26104f52b7e1dc2fc5980f42eea93b
Author: flvndh <17...@users.noreply.github.com>
AuthorDate: Fri Aug 28 16:52:05 2020 +0200
Wrong key in DAGs Persistent Volume Claim (#10627)
Co-authored-by: Flavien Dereume-Hancart <fl...@LL-PC0BE1K9-1.goiba.net>
(cherry picked from commit 5f403a87597fe10459a12bb8c1368fc8b65f9a7c)
---
chart/templates/dags-persistent-volume-claim.yaml | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/chart/templates/dags-persistent-volume-claim.yaml b/chart/templates/dags-persistent-volume-claim.yaml
index 53ea550..d3c95b2 100644
--- a/chart/templates/dags-persistent-volume-claim.yaml
+++ b/chart/templates/dags-persistent-volume-claim.yaml
@@ -31,11 +31,11 @@ spec:
resources:
requests:
storage: {{ .Values.dags.persistence.size | quote }}
- {{- if .Values.dags.persistence.storageClass }}
- {{- if (eq "-" .Values.dags.persistence.storageClass) }}
+ {{- if .Values.dags.persistence.storageClassName }}
+ {{- if (eq "-" .Values.dags.persistence.storageClassName) }}
storageClassName: ""
{{- else }}
- storageClassName: "{{ .Values.dags.persistence.storageClass }}"
+ storageClassName: "{{ .Values.dags.persistence.storageClassName }}"
{{- end }}
{{- end }}
{{- end }}
[airflow] 14/19: Run create-user-job as user with specified id
(#10291)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 465a3aef7da419af65ce308c11ecfd1e82a43256
Author: gardnerdev <ra...@gmail.com>
AuthorDate: Wed Aug 12 01:03:51 2020 +0200
Run create-user-job as user with specified id (#10291)
In secured cluster there is a need to run this job with specific user id
(cherry picked from commit 1266b29af79128c2364ef534e7a7de1a4ab6484f)
---
chart/templates/create-user-job.yaml | 2 ++
1 file changed, 2 insertions(+)
diff --git a/chart/templates/create-user-job.yaml b/chart/templates/create-user-job.yaml
index 0f76252..656059b 100644
--- a/chart/templates/create-user-job.yaml
+++ b/chart/templates/create-user-job.yaml
@@ -44,6 +44,8 @@ spec:
component: create-user-job
release: {{ .Release.Name }}
spec:
+ securityContext:
+ runAsUser: {{ .Values.uid }}
restartPolicy: OnFailure
nodeSelector:
{{ toYaml .Values.nodeSelector | indent 8 }}
[airflow] 13/19: use the correct claim name in the webserver (#9688)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit ad3a259b28a11305686c0c6e6785fcb25cb3e8ec
Author: Aneesh Joseph <ad...@coderplus.com>
AuthorDate: Tue Jul 7 22:38:35 2020 +0530
use the correct claim name in the webserver (#9688)
(cherry picked from commit 502b6ea65790ab0bfe35ab80d3a521de8df541d5)
---
chart/templates/_helpers.yaml | 2 +-
chart/templates/webserver/webserver-deployment.yaml | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/chart/templates/_helpers.yaml b/chart/templates/_helpers.yaml
index 929861e..059d64d 100644
--- a/chart/templates/_helpers.yaml
+++ b/chart/templates/_helpers.yaml
@@ -321,7 +321,7 @@ server_tls_key_file = /etc/pgbouncer/server.key
{{- end -}}
{{ define "airflow_dags_volume_claim" -}}
-{{- if and .Values.dags.persistence.enabled .Values.dags.persistence.existingClaim -}}
+{{- if .Values.dags.persistence.existingClaim -}}
{{ .Values.dags.persistence.existingClaim }}
{{- else -}}
{{ .Release.Name }}-dags
diff --git a/chart/templates/webserver/webserver-deployment.yaml b/chart/templates/webserver/webserver-deployment.yaml
index f6f781f..a3c42c0 100644
--- a/chart/templates/webserver/webserver-deployment.yaml
+++ b/chart/templates/webserver/webserver-deployment.yaml
@@ -146,7 +146,7 @@ spec:
{{- if .Values.dags.persistence.enabled }}
- name: dags
persistentVolumeClaim:
- claimName: {{ .Release.Name }}-dags
+ claimName: {{ template "airflow_dags_volume_claim" . }}
{{- else if .Values.dags.gitSync.enabled }}
- name: dags
emptyDir: {}
[airflow] 19/19: Add TargetQueryValue to KEDA Autoscaler (#9748)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 36bfa043f0575474763b9bd0bf0f200be3d18c69
Author: Daniel Imberman <da...@gmail.com>
AuthorDate: Fri Jul 10 07:48:17 2020 -0700
Add TargetQueryValue to KEDA Autoscaler (#9748)
Co-authored-by: Daniel Imberman <da...@astronomer.io>
(cherry picked from commit 0faa06ed9b010eb3d2c3a366d68649526f0bf2ad)
---
chart/templates/workers/worker-kedaautoscaler.yaml | 1 +
1 file changed, 1 insertion(+)
diff --git a/chart/templates/workers/worker-kedaautoscaler.yaml b/chart/templates/workers/worker-kedaautoscaler.yaml
index c733618..e135076 100644
--- a/chart/templates/workers/worker-kedaautoscaler.yaml
+++ b/chart/templates/workers/worker-kedaautoscaler.yaml
@@ -42,6 +42,7 @@ spec:
triggers:
- type: postgresql
metadata:
+ targetQueryValue: "1"
connection: AIRFLOW_CONN_AIRFLOW_DB
query: "SELECT ceil(COUNT(*)::decimal / 16) FROM task_instance WHERE state='running' OR state='queued'"
{{- end }}