You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/08/05 06:38:31 UTC
[GitHub] [airflow] dstandish opened a new pull request #17433: KEDA task count query should ignore k8s queue
dstandish opened a new pull request #17433:
URL: https://github.com/apache/airflow/pull/17433
CeleryKubernetesExecutor lets us use both celery and kubernetes executors.
KEDA lets us scale down to zero when there are no celery tasks running.
The existing KEDA query doesn't look at task `queue` so it wil scale up to 1 if there is
1 k8s task and 0 celery tasks.
We can prevent this from happening by ignoring the kubernetes queue in the KEDA query.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham closed pull request #17433: KEDA task count query should ignore k8s queue
Posted by GitBox <gi...@apache.org>.
jedcunningham closed pull request #17433:
URL: https://github.com/apache/airflow/pull/17433
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham merged pull request #17433: KEDA task count query should ignore k8s queue
Posted by GitBox <gi...@apache.org>.
jedcunningham merged pull request #17433:
URL: https://github.com/apache/airflow/pull/17433
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on a change in pull request #17433: KEDA task count query should ignore k8s queue
Posted by GitBox <gi...@apache.org>.
dstandish commented on a change in pull request #17433:
URL: https://github.com/apache/airflow/pull/17433#discussion_r683756891
##########
File path: chart/templates/workers/worker-kedaautoscaler.yaml
##########
@@ -49,5 +49,7 @@ spec:
query: >-
SELECT ceil(COUNT(*)::decimal / {{ .Values.config.celery.worker_concurrency }})
FROM task_instance
- WHERE state='running' OR state='queued'
+ WHERE (state='running' OR state='queued')
+{{ $k8s_queue := default (printf "kubernetes") .Values.config.celery_kubernetes_executor.kubernetes_queue -}}
+{{ eq .Values.executor "CeleryKubernetesExecutor" | ternary (printf "AND queue != '%s'" $k8s_queue) (print "") | indent 14 }}
Review comment:
looks good, thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #17433: KEDA task count query should ignore k8s queue
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #17433:
URL: https://github.com/apache/airflow/pull/17433#issuecomment-894383565
The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not modify the core of Airflow. If the committers decide that the full tests matrix is needed, they will add the label 'full tests needed'. Then you should rebase to the latest main or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on a change in pull request #17433: KEDA task count query should ignore k8s queue
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on a change in pull request #17433:
URL: https://github.com/apache/airflow/pull/17433#discussion_r683648189
##########
File path: chart/templates/workers/worker-kedaautoscaler.yaml
##########
@@ -49,5 +49,7 @@ spec:
query: >-
SELECT ceil(COUNT(*)::decimal / {{ .Values.config.celery.worker_concurrency }})
FROM task_instance
- WHERE state='running' OR state='queued'
+ WHERE (state='running' OR state='queued')
+{{ $k8s_queue := default (printf "kubernetes") .Values.config.celery_kubernetes_executor.kubernetes_queue -}}
+{{ eq .Values.executor "CeleryKubernetesExecutor" | ternary (printf "AND queue != '%s'" $k8s_queue) (print "") | indent 14 }}
Review comment:
```suggestion
WHERE (state='running' OR state='queued')
{{- if eq .Values.executor "CeleryKubernetesExecutor" }}
AND queue != '{{ .Values.config.celery_kubernetes_executor.kubernetes_queue }}'
{{- end }}
```
Do we need the default if we've set it in values already? I think this is easier to read.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org