[GitHub] [airflow] asosnovsky commented on a diff in pull request #30914: Added support for custom KEDA scaling

["asosnovsky (via GitHub)" <gi...@apache.org>]

asosnovsky commented on code in PR #30914:
URL: https://github.com/apache/airflow/pull/30914#discussion_r1185666839


##########
chart/values.yaml:
##########
@@ -704,6 +718,51 @@ scheduler:
     # Annotations to add to scheduler kubernetes service account.
     annotations: {}
 
+
+  # Allow KEDA autoscaling.
+  keda:
+    enabled: false
+    namespaceLabels: {}
+
+    # How often KEDA polls the airflow DB to report new scale requests to the HPA
+    pollingInterval: 5
+
+    # How many seconds KEDA will wait before scaling to zero.
+    # Note that HPA has a separate cooldown period for scale-downs
+    cooldownPeriod: 30
+
+    # Minimum number of workers created by keda
+    minReplicaCount: 0
+
+    # Maximum number of workers created by keda
+    maxReplicaCount: 10
+
+    # Specify HPA related options
+    advanced: {}
+    # horizontalPodAutoscalerConfig:
+    #   behavior:
+    #     scaleDown:
+    #       stabilizationWindowSeconds: 300
+    #       policies:
+    #         - type: Percent
+    #           value: 100
+    #           periodSeconds: 15
+
+    # Specify the KEDA Triggers for autoscaling
+    triggers: >-
+      - type: postgresql
+        metadata:
+          targetQueryValue: "1"
+          connectionFromEnv: AIRFLOW_CONN_AIRFLOW_DB
+          query: >-
+            SELECT ceil(COUNT(*)::decimal / {{ .Values.config.core.parallelism }})
+            FROM task_instance
+            WHERE (state='running' OR state='queued')
+            {{- if eq .Values.executor "CeleryKubernetesExecutor" }}
+            AND queue != '{{ .Values.config.celery_kubernetes_executor.kubernetes_queue }}'
+            {{- end }}

Review Comment:
   @dstandish perhaps this topic should be on a separate conversation? 
   our company atm is using a semi-hacked way of this (we basically set replicas to `~` and then manually defined the same definitions I've added here). I will add some test cases over the weekend (or monday) and after that it would be nice to merge this :)
   
   In the meantime maybe we should transfer this to a github issue on how to improve the queries? I feel like it should be ok, since my change allows people to set whatever query they want.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org