You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by ep...@apache.org on 2023/08/07 05:48:28 UTC
[airflow] branch main updated: Fix reload gunicorn workers (#32102)
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new f78a8363a6 Fix reload gunicorn workers (#32102)
f78a8363a6 is described below
commit f78a8363a6542a4b4c94642434424da7af7fcbbb
Author: Aleksandr Musorin <Sm...@mail.ru>
AuthorDate: Mon Aug 7 07:48:20 2023 +0200
Fix reload gunicorn workers (#32102)
* Toggle gunicorn --preload with reload_on_plugin_change
Since gunicorn can't reload a new code if starts with ``--preload``
setting, we need to check ``reload_on_plugin_change`` before set it up.
Gunicorn can't reload a new code because the code is preloaded into the
master process and worker are launched with ``fork``, they will still have
the old code.
* added warning message
* Improve warning message
Co-authored-by: Jed Cunningham <66...@users.noreply.github.com>
* Improve warning message
Co-authored-by: Jed Cunningham <66...@users.noreply.github.com>
* Improve warning message
Co-authored-by: Jed Cunningham <66...@users.noreply.github.com>
* Minor tweak of warning message
* Mention prod issue in config description
---------
Co-authored-by: Jed Cunningham <66...@users.noreply.github.com>
Co-authored-by: Tzu-ping Chung <ur...@gmail.com>
---
airflow/cli/commands/webserver_command.py | 16 +++++++++++-----
airflow/config_templates/config.yml | 3 ++-
2 files changed, 13 insertions(+), 6 deletions(-)
diff --git a/airflow/cli/commands/webserver_command.py b/airflow/cli/commands/webserver_command.py
index 5399f8ba64..17d48ad8da 100644
--- a/airflow/cli/commands/webserver_command.py
+++ b/airflow/cli/commands/webserver_command.py
@@ -422,11 +422,17 @@ def webserver(args):
run_args += ["airflow.www.app:cached_app()"]
- # To prevent different workers creating the web app and
- # all writing to the database at the same time, we use the --preload option.
- # With the preload option, the app is loaded before the workers are forked, and each worker will
- # then have a copy of the app
- run_args += ["--preload"]
+ if conf.getboolean("webserver", "reload_on_plugin_change", fallback=False):
+ log.warning(
+ "Setting reload_on_plugin_change = true prevents running Gunicorn with preloading. "
+ "This means the app cannot be loaded before workers are forked, and each worker has a "
+ "separate copy of the app. This may cause IntegrityError during webserver startup, and "
+ "should be avoided in production."
+ )
+ else:
+ # To prevent different workers creating the web app and
+ # all writing to the database at the same time, we use the --preload option.
+ run_args += ["--preload"]
gunicorn_master_proc: psutil.Process | subprocess.Popen
diff --git a/airflow/config_templates/config.yml b/airflow/config_templates/config.yml
index f868a6c6ea..f61f6baca0 100644
--- a/airflow/config_templates/config.yml
+++ b/airflow/config_templates/config.yml
@@ -1400,7 +1400,8 @@ webserver:
reload_on_plugin_change:
description: |
If set to True, Airflow will track files in plugins_folder directory. When it detects changes,
- then reload the gunicorn.
+ then reload the gunicorn. If set to True, gunicorn starts without preloading, which is slower, uses
+ more memory, and may cause race conditions. Avoid setting this to True in production.
version_added: 1.10.11
type: boolean
example: ~