You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "kaxil (via GitHub)" <gi...@apache.org> on 2026/01/22 14:49:04 UTC
[PR] Add gunicorn support for API server with rolling worker restarts [airflow]
kaxil opened a new pull request, #60940:
URL: https://github.com/apache/airflow/pull/60940
This PR adds an optional `gunicorn` server type for the API server, providing:
- **Memory sharing**: Gunicorn uses preload + fork, so workers share memory via copy-on-write (unlike uvicorn's multiprocess mode where each worker loads everything independently)
- **Rolling worker restarts**: GunicornMonitor performs zero-downtime worker recycling to prevent memory accumulation
- **Proper signal handling**: SIGTTOU kills oldest worker (FIFO), enabling true rolling restarts
## Usage
```bash
# Enable gunicorn mode
export AIRFLOW__API__SERVER_TYPE=gunicorn
export AIRFLOW__API__WORKER_REFRESH_INTERVAL=43200 # 12 hours
airflow api-server
```
## Configuration
New `[api]` configuration options:
- `server_type`: `uvicorn` (default) or `gunicorn`
- `worker_refresh_interval`: Seconds between worker refresh cycles (0 = disabled)
- `worker_refresh_batch_size`: Workers to refresh per cycle (default: 1)
- `reload_on_plugin_change`: Reload on plugin file changes (default: False)
## Architecture
```
┌─────────────────────────────────────────────────────────┐
│ airflow api-server (main process) │
│ ├── subprocess: gunicorn master (PID management) │
│ │ ├── worker 1 (UvicornWorker) │
│ │ ├── worker 2 (UvicornWorker) │
│ │ └── worker N (UvicornWorker) │
│ └── thread: GunicornMonitor (rolling restarts) │
└─────────────────────────────────────────────────────────┘
```
## Rolling Restart Flow
1. Spawn `batch_size` new workers (SIGTTIN)
2. Wait for new workers to be ready (process title check)
3. HTTP health check (`/api/v2/monitor/health`)
4. Kill `batch_size` old workers (SIGTTOU - kills oldest)
5. Repeat until all original workers replaced
## Gotchas vs Uvicorn mode
| Aspect | Gunicorn | Uvicorn |
|--------|----------|---------|
| Memory sharing | Yes (preload + fork COW) | No (independent workers) |
| Rolling restarts | Yes (GunicornMonitor) | No |
| Worker management | Master/worker architecture | Direct multiprocess |
| macOS support | Limited (setproctitle issues) | Full |
## Why Gunicorn is Optional
Gunicorn is an optional extra (`apache-airflow-core[gunicorn]`) rather than a required dependency because:
1. **Windows incompatibility**: Gunicorn is Unix-only and doesn't work on Windows
2. **Most users don't need it**: Default uvicorn mode is sufficient for:
- Development environments
- Single-worker deployments
- Short-lived containers (K8s pods that get recycled anyway)
3. **Dependency minimization**: Users who don't need rolling restarts or memory sharing shouldn't pay the dependency cost
**When to use gunicorn:**
- Long-running API server processes where memory accumulation is a concern
- Multi-worker deployments where memory sharing matters
- Production environments requiring zero-downtime worker recycling
## Known Limitations / Follow-up Items
### Log Format Inconsistency
Gunicorn subprocess uses its own logging format which differs from Airflow's structlog format:
```
# Gunicorn native logs:
[2026-01-22 14:14:03 +0000] [433] [INFO] Handling signal: ttin
# Airflow structlog logs:
2026-01-22T14:14:03.115354Z [info ] Rolling restart: spawning... [airflow.cli.commands.gunicorn_monitor]
```
This means API server logs will have mixed formats when using gunicorn mode. A follow-up PR should add `--logconfig` to gunicorn to match Airflow's format for consistent log parsing/export.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add gunicorn support for API server with rolling worker restarts [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on PR #60940:
URL: https://github.com/apache/airflow/pull/60940#issuecomment-4229144023
Just came across this - COOL!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add gunicorn support for API server with rolling worker restarts [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on code in PR #60940:
URL: https://github.com/apache/airflow/pull/60940#discussion_r2761011638
##########
airflow-core/newsfragments/60921.significant.rst:
##########
@@ -0,0 +1,51 @@
+Add gunicorn support for API server with zero-downtime worker recycling
+
+The API server now supports gunicorn as an alternative server with rolling worker restarts
+to prevent memory accumulation in long-running processes.
+
+**Key Benefits:**
+
+* **Rolling worker restarts**: New workers spawn and pass health checks before old workers
+ are killed, ensuring zero downtime during worker recycling.
+
+* **Memory sharing**: Gunicorn uses preload + fork, so workers share memory via
+ copy-on-write. This significantly reduces total memory usage compared to uvicorn's
+ multiprocess mode where each worker loads everything independently.
+
+* **Correct FIFO signal handling**: Gunicorn's SIGTTOU kills the oldest worker (FIFO),
+ not the newest (LIFO), which is correct for rolling restarts.
+
+**Configuration:**
+
+.. code-block:: ini
+
+ [api]
+ # Use gunicorn instead of uvicorn
+ server_type = gunicorn
+
+ # Enable rolling worker restarts every 12 hours
+ worker_refresh_interval = 43200
+
+ # Restart workers one at a time
+ worker_refresh_batch_size = 1
+
+Or via environment variables:
+
+.. code-block:: bash
+
+ export AIRFLOW__API__SERVER_TYPE=gunicorn
+ export AIRFLOW__API__WORKER_REFRESH_INTERVAL=43200
+
+**Requirements:**
+
+Install the gunicorn extra: ``pip install 'apache-airflow-core[gunicorn]'``
+
+**Note on uvicorn (default):**
+
+The default uvicorn mode does not support rolling worker restarts because:
+
+1. With workers=1, there is no master process to send signals to
+2. uvicorn's SIGTTOU kills the newest worker (LIFO), defeating rolling restart purposes
+3. Each uvicorn worker loads everything independently with no memory sharing
+
Review Comment:
Probably, although airflow-core/docs/administration-and-deployment/web-stack.rst might be a better place and our helm chart. So a separate PR for that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add gunicorn support for API server with rolling worker restarts [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #60940:
URL: https://github.com/apache/airflow/pull/60940#discussion_r2741116337
##########
airflow-core/src/airflow/cli/commands/gunicorn_monitor.py:
##########
Review Comment:
This shouldn't live under `cli/commands` , `src/airflow/api_fastapi/gunicorn_monitor.py` would be better I think
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add gunicorn support for API server with rolling worker restarts [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on code in PR #60940:
URL: https://github.com/apache/airflow/pull/60940#discussion_r2753284864
##########
airflow-core/newsfragments/60921.significant.rst:
##########
@@ -0,0 +1,51 @@
+Add gunicorn support for API server with zero-downtime worker recycling
+
+The API server now supports gunicorn as an alternative server with rolling worker restarts
+to prevent memory accumulation in long-running processes.
+
+**Key Benefits:**
+
+* **Rolling worker restarts**: New workers spawn and pass health checks before old workers
+ are killed, ensuring zero downtime during worker recycling.
+
+* **Memory sharing**: Gunicorn uses preload + fork, so workers share memory via
+ copy-on-write. This significantly reduces total memory usage compared to uvicorn's
+ multiprocess mode where each worker loads everything independently.
+
+* **Correct FIFO signal handling**: Gunicorn's SIGTTOU kills the oldest worker (FIFO),
+ not the newest (LIFO), which is correct for rolling restarts.
+
+**Configuration:**
+
+.. code-block:: ini
+
+ [api]
+ # Use gunicorn instead of uvicorn
+ server_type = gunicorn
+
+ # Enable rolling worker restarts every 12 hours
+ worker_refresh_interval = 43200
+
+ # Restart workers one at a time
+ worker_refresh_batch_size = 1
+
+Or via environment variables:
+
+.. code-block:: bash
+
+ export AIRFLOW__API__SERVER_TYPE=gunicorn
+ export AIRFLOW__API__WORKER_REFRESH_INTERVAL=43200
+
+**Requirements:**
+
+Install the gunicorn extra: ``pip install 'apache-airflow-core[gunicorn]'``
+
+**Note on uvicorn (default):**
+
+The default uvicorn mode does not support rolling worker restarts because:
+
+1. With workers=1, there is no master process to send signals to
+2. uvicorn's SIGTTOU kills the newest worker (LIFO), defeating rolling restart purposes
+3. Each uvicorn worker loads everything independently with no memory sharing
+
Review Comment:
Do we need to mention the case when using `uvicorn` in k8s?
e.g. setup a CornJob that periodically run `kubectl rollout restart deployment/airflow-api-server`
Or, since the api-server will be OOM killed and k8s will create a new pod automatically, so we don't need to mention the above case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add gunicorn support for API server with rolling worker restarts [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil merged PR #60940:
URL: https://github.com/apache/airflow/pull/60940
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add gunicorn support for API server with rolling worker restarts [airflow]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #60940:
URL: https://github.com/apache/airflow/pull/60940#discussion_r2737878846
##########
airflow-core/src/airflow/cli/commands/api_server_command.py:
##########
@@ -50,22 +52,163 @@
# more info here: https://github.com/benoitc/gunicorn/issues/1877#issuecomment-1911136399
-@enable_memray_trace(component=MemrayTraceComponents.api)
-def _run_api_server(args, apps: str, num_workers: int, worker_timeout: int, proxy_headers: bool):
- """Run the API server."""
+def _build_gunicorn_command(
+ host: str,
+ port: int,
+ num_workers: int,
+ worker_timeout: int,
+ ssl_cert: str | None,
+ ssl_key: str | None,
+ log_level: str,
+ access_log_enabled: bool,
+ proxy_headers: bool,
+) -> list[str]:
+ """
+ Build the gunicorn command line arguments.
+
+ Uses uvicorn.workers.UvicornWorker as the worker class to run the ASGI app.
+ """
+ cmd = [
+ "gunicorn",
+ "airflow.api_fastapi.main:app",
+ "--worker-class",
+ "uvicorn.workers.UvicornWorker",
+ "--config",
+ "python:airflow.api_fastapi.gunicorn_config",
+ "--bind",
+ f"{host}:{port}",
+ "--workers",
+ str(num_workers),
+ "--timeout",
+ str(worker_timeout),
+ "--graceful-timeout",
+ str(worker_timeout),
+ "--keep-alive",
+ str(worker_timeout),
+ "--log-level",
+ log_level,
+ # Preload app to share memory across workers via copy-on-write
+ "--preload",
+ ]
+
+ if ssl_cert and ssl_key:
+ cmd.extend(["--certfile", ssl_cert, "--keyfile", ssl_key])
+
+ # Configure access logging - gunicorn doesn't log access by default
+ if access_log_enabled:
+ cmd.extend(["--access-logfile", "-"]) # Log to stdout
+
+ if proxy_headers:
+ cmd.extend(["--forwarded-allow-ips", "*"])
+
+ return cmd
+
+
+def _run_api_server_with_gunicorn(
+ args,
+ apps: str,
+ num_workers: int,
+ worker_timeout: int,
+ proxy_headers: bool,
+) -> None:
+ """
+ Run the API server using gunicorn with uvicorn workers.
+
+ Gunicorn provides proper master/worker process management with:
+ - SIGTTIN to spawn new workers
+ - SIGTTOU to kill oldest worker (FIFO order - correct for rolling restarts)
+ - Memory sharing via preload + fork copy-on-write
+
+ Runs GunicornMonitor in the main thread (like AF2's webserver pattern).
+ """
+ ssl_cert, ssl_key = _get_ssl_cert_and_key_filepaths(args)
+
+ log_level = conf.get("logging", "uvicorn_logging_level", fallback="info").lower()
+ access_log_enabled = log_level not in ("error", "critical", "fatal")
+
+ cmd = _build_gunicorn_command(
+ host=args.host,
+ port=args.port,
+ num_workers=num_workers,
+ worker_timeout=worker_timeout,
+ ssl_cert=ssl_cert,
+ ssl_key=ssl_key,
+ log_level=log_level,
+ access_log_enabled=access_log_enabled,
+ proxy_headers=proxy_headers,
+ )
+
log.info(
textwrap.dedent(
f"""\
- Running the uvicorn with:
+ Running gunicorn with:
Apps: {apps}
Workers: {num_workers}
Host: {args.host}:{args.port}
Timeout: {worker_timeout}
- Logfiles: {args.log_file or "-"}
+ Command: {" ".join(cmd)}
================================================================="""
)
)
Review Comment:
This log message duplicates the one at lines 266-277. Consider removing this duplicate since it's logged again in the main _run_api_server function.
##########
airflow-core/docs/extra-packages-ref.rst:
##########
@@ -126,6 +126,8 @@ other packages that can be used by airflow or some of its providers.
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------------+
| graphviz | ``pip install 'apache-airflow[graphviz]'`` | Graphviz renderer for converting Dag to graphical output |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------------+
+| gunicorn | ``pip install 'apache-airflow[gunicorn]'`` | periodic worker recycling of ``uvicorn`` workers on the API-server |
Review Comment:
The description is slightly misleading. Gunicorn doesn't recycle 'uvicorn workers' - it uses UvicornWorker as the worker class. Consider: 'Gunicorn server with rolling worker restarts for the API server' or 'Worker recycling support for the API server'.
```suggestion
| gunicorn | ``pip install 'apache-airflow[gunicorn]'`` | Gunicorn server with rolling worker restarts for the API server |
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add gunicorn support for API server with rolling worker restarts [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on code in PR #60940:
URL: https://github.com/apache/airflow/pull/60940#discussion_r2762495742
##########
airflow-core/newsfragments/60921.significant.rst:
##########
@@ -0,0 +1,51 @@
+Add gunicorn support for API server with zero-downtime worker recycling
+
+The API server now supports gunicorn as an alternative server with rolling worker restarts
+to prevent memory accumulation in long-running processes.
+
+**Key Benefits:**
+
+* **Rolling worker restarts**: New workers spawn and pass health checks before old workers
+ are killed, ensuring zero downtime during worker recycling.
+
+* **Memory sharing**: Gunicorn uses preload + fork, so workers share memory via
+ copy-on-write. This significantly reduces total memory usage compared to uvicorn's
+ multiprocess mode where each worker loads everything independently.
+
+* **Correct FIFO signal handling**: Gunicorn's SIGTTOU kills the oldest worker (FIFO),
+ not the newest (LIFO), which is correct for rolling restarts.
+
+**Configuration:**
+
+.. code-block:: ini
+
+ [api]
+ # Use gunicorn instead of uvicorn
+ server_type = gunicorn
+
+ # Enable rolling worker restarts every 12 hours
+ worker_refresh_interval = 43200
+
+ # Restart workers one at a time
+ worker_refresh_batch_size = 1
+
+Or via environment variables:
+
+.. code-block:: bash
+
+ export AIRFLOW__API__SERVER_TYPE=gunicorn
+ export AIRFLOW__API__WORKER_REFRESH_INTERVAL=43200
+
+**Requirements:**
+
+Install the gunicorn extra: ``pip install 'apache-airflow-core[gunicorn]'``
+
+**Note on uvicorn (default):**
+
+The default uvicorn mode does not support rolling worker restarts because:
+
+1. With workers=1, there is no master process to send signals to
+2. uvicorn's SIGTTOU kills the newest worker (LIFO), defeating rolling restart purposes
+3. Each uvicorn worker loads everything independently with no memory sharing
+
Review Comment:
Make sense to me, I just tacked them in #61430.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add gunicorn support for API server with rolling worker restarts [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #60940:
URL: https://github.com/apache/airflow/pull/60940#issuecomment-3846398177
### Backport failed to create: v3-1-test. View the failure log <a href='https://github.com/apache/airflow/actions/runs/21666510742'> Run details </a>
<table>
<tr>
<th>Status</th>
<th>Branch</th>
<th>Result</th>
</tr>
<tr>
<td>❌</td>
<td>v3-1-test</td>
<td><a href="https://github.com/apache/airflow/commit/b906080fe82cd72c9024dd62cf1b64c15f403ed8"><img src='https://img.shields.io/badge/Commit-b906080-red' alt='Commit Link'></a></td>
</tr>
</table>
You can attempt to backport this manually by running:
```bash
cherry_picker b906080 v3-1-test
```
This should apply the commit to the v3-1-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.
After you have resolved the conflicts, you can continue the backport process by running:
```bash
cherry_picker --continue
```
If you don't have cherry-picker installed, see the [installation guide](https://github.com/apache/airflow/blob/main/dev/README_AIRFLOW3_DEV.md#how-to-backport-pr-with-cherry-picker-cli).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org