[GitHub] [airflow] kaxil commented on a change in pull request #8695: Make celery worker_prefetch_multiplier configurable

[GitBox <gi...@apache.org>]

kaxil commented on a change in pull request #8695:
URL: https://github.com/apache/airflow/pull/8695#discussion_r421143742



##########
File path: airflow/config_templates/default_airflow.cfg
##########
@@ -550,6 +550,15 @@ worker_concurrency = 8
 # Example: worker_autoscale = 16,12
 # worker_autoscale =
 
+# Due to an issue with Celery, worker_prefetch_multiplier must also be set to an int
+# greater than 1 for autoscaling to work.  Be aware that this causes the number of
+# records that a worker prefetchs to be greater than the available worker processes.
+# If you have multiple workers this can cause some jobs to be unnecessarily blocked if
+# one worker prefetches tasks and those tasks are blocked by long running work then another
+# worker with available process workers can't grab those tasks.
+# https://docs.celeryproject.org/en/stable/userguide/optimizing.html#prefetch-limits
+# worker_prefetch_multiplier =

Review comment:
       ```suggestion
   # worker_prefetch_multiplier = 1
   ```

##########
File path: airflow/config_templates/config.yml
##########
@@ -1130,6 +1130,19 @@
       type: string
       example: 16,12
       default: ~
+    - name: worker_prefetch_multiplier
+      description: |
+        Due to an issue with Celery, worker_prefetch_multiplier must also be set to an int
+        greater than 1 for autoscaling to work.  Be aware that this causes the number of
+        records that a worker prefetchs to be greater than the available worker processes.
+        If you have multiple workers this can cause some jobs to be unnecessarily blocked if
+        one worker prefetches tasks and those tasks are blocked by long running work then another
+        worker with available process workers can't grab those tasks.
+        https://docs.celeryproject.org/en/stable/userguide/optimizing.html#prefetch-limits
+      version_added: ~
+      type: int
+      example: ~
+      default: ~

Review comment:
       ```suggestion
         default: 1
   ```




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org