You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "ron-gaist (via GitHub)" <gi...@apache.org> on 2026/01/08 12:17:29 UTC
[I] Edge worker sometimes fails on startup with 405: Method not allowed [airflow]
ron-gaist opened a new issue, #60261:
URL: https://github.com/apache/airflow/issues/60261
### Apache Airflow Provider(s)
edge3
### Versions of Apache Airflow Providers
apache-airflow-providers-celery==3.13.0
apache-airflow-providers-common-compat==1.8.3
apache-airflow-providers-common-io==1.6.4
apache-airflow-providers-common-sql=1.28.2
apache-airflow-providers-fab==3.0.1
apache-airflow-providers-standard==1.9.1
apache-airflow-providers-postgres==6.4.0
apache-airflow-providers-edge3==1.4.0
### Apache Airflow version
3.1.2
### Operating System
Debian GNU/Linux 12 (bookworm)
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
Setup scale: (yes we know this is very large scale)
8 api servers x 64 worker processes each
9 Edge workers attempting to connect continuously (and continuously failing)
etc... (rest are not relevant)
### What happened
Upon edge worker startup we get the following error:
> 405 Client Error: Method not allowed for url: http://<AIRFLOW-API-SERVER-URL>/edge_worker/v1/worker/<WORKER-NAME>
Possible relevant values in the helm chart values.yaml:
```
apiSecretKey: <VALUE>
jwtSecret: <VALUE>
config:
edge:
api_enabled: 'True'
api_url: ''http://<AIRFLOW-API-SERVER-URL>/edge_worker/v1/rcpapi'
```
This behavior seems to come and go - workers can fail on startup and after some time start up normally.
### What you think should happen instead
_No response_
### How to reproduce
Reproducing this seems to be tricky. Even for us this behavior sometimes exists and other times not. The best information we can give for reproducing this seems to be the airflow config.
### Anything else
`405 Method not allowed` seems to suggest that the API servers did not start with the necessary portion of the edge plugin.
This is very odd, and is the kind of behavior we'd expect if the config lacked `edge: api_enabled: 'True', api_url: '...'`
However we double and triple checked that these configurations are set in the api servers (and on the edge worker too, for what it's worth)
**Additional note (could be related):**
Upon startup of each api-server we get an error related to edge plugin import:
```
[error] Failed to import plugin edge_executor [airflow.plugins_manager] loc=plugins_manager.py:268
....
psycop2.errors.LockNotAvailable: canceling statement due to lock timeout
```
We checked the lock timeout on our postgres db and it is set to 0 (unlimited)
We aren't pointing to this as the *root cause* because previously edge has worked for us while this plugin has been similarly not imported. But this might point in the right direction
We know this isn't related to the *amount* of api_servers (we tried with 1) but it might be related to the amount of worker processes - we don't know for sure.
Thank you for the time considering our issue!!
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Edge worker sometimes fails on startup with 405: Method not allowed [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl closed issue #60261: Edge worker sometimes fails on startup with 405: Method not allowed
URL: https://github.com/apache/airflow/issues/60261
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Edge worker sometimes fails on startup with 405: Method not allowed [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on issue #60261:
URL: https://github.com/apache/airflow/issues/60261#issuecomment-3740421815
Not, if you want to produce HTTP 405 error for testing:
- Just start breeze with EdgeExecutor
- CTRL+C the API Server panel
- in the API server panel call `unset AIRFLOW__EDGE__API_ENABLED`
- Restart API Server
Then Edge Worker will fail in HTTP 405 and also does not start-up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Edge worker sometimes fails on startup with 405: Method not allowed [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on issue #60261:
URL: https://github.com/apache/airflow/issues/60261#issuecomment-3746665997
> Also, in my setup, I have 8 api-server replicas deployed, each running 64 workers.
Note from performance and best practices how FastAPI runs it was seen that 1 worker and rather more replicas are better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Edge worker sometimes fails on startup with 405: Method not allowed [airflow]
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on issue #60261:
URL: https://github.com/apache/airflow/issues/60261#issuecomment-3763609223
It seems that this bug report has been shifted to a feature request, so I've applied the correct labels :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Edge worker sometimes fails on startup with 405: Method not allowed [airflow]
Posted by "arjav1528 (via GitHub)" <gi...@apache.org>.
arjav1528 commented on issue #60261:
URL: https://github.com/apache/airflow/issues/60261#issuecomment-3734306406
I'd like to submit a PR on this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Edge worker sometimes fails on startup with 405: Method not allowed [airflow]
Posted by "ron-gaist (via GitHub)" <gi...@apache.org>.
ron-gaist commented on issue #60261:
URL: https://github.com/apache/airflow/issues/60261#issuecomment-3743324009
here is the full traceback of the `LOCK_TIMEOUT` error:
`2026-01-12T09:55:24.930491Z [error ] Failed to import plugin edge_executor [airflow.plugins_manager] loc=plugins_manager.py:261
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1910, in _execute_context
self.dialect.do_execute(
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
cursor.execute(statement, parameters)
psycopg2.errors.LockNotAvailable: canceling statement due to lock timeout
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/utils/db.py", line 1362, in create_global_lock
conn.execute(text("SELECT pg_advisory_lock(:id)"), {"id": lock.value})
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/future/engine.py", line 286, in execute
return self._execute_20(
^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1710, in _execute_20
return meth(self, args_10style, kwargs_10style, execution_options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection
return connection._execute_clauseelement(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1577, in _execute_clauseelement
ret = self._execute_context(
^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1953, in _execute_context
self._handle_dbapi_exception(
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 2134, in _handle_dbapi_exception
util.raise_(
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
raise exception
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1910, in _execute_context
self.dialect.do_execute(
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (psycopg2.errors.LockNotAvailable) canceling statement due to lock timeout
[SQL: SELECT pg_advisory_lock(%(id)s)]
[parameters: {'id': 1}]
(Background on this error at: https://sqlalche.me/e/14/e3q8)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1910, in _execute_context
self.dialect.do_execute(
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
cursor.execute(statement, parameters)
psycopg2.errors.InFailedSqlTransaction: current transaction is aborted, commands ignored until end of transaction block
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/plugins_manager.py", line 253, in load_entrypoint_plugins
plugin_class = entry_point.load()
^^^^^^^^^^^^^^^^^^
File "/usr/python/lib/python3.12/importlib/metadata/__init__.py", line 205, in load
module = import_module(match.group("module"))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/python/lib/python3.12/importlib/__init__.py", line 90, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 999, in exec_module
File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/edge3/plugins/edge_executor_plugin.py", line 245, in <module>
class EdgeExecutorPlugin(AirflowPlugin):
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/edge3/plugins/edge_executor_plugin.py", line 251, in EdgeExecutorPlugin
fastapi_apps = [_get_api_endpoint()]
^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/utils/session.py", line 100, in wrapper
return func(*args, session=session, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/edge3/plugins/edge_executor_plugin.py", line 38, in _get_api_endpoint
with create_global_lock(session=session, lock=DBLocks.MIGRATIONS):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/python/lib/python3.12/contextlib.py", line 137, in __enter__
return next(self.gen)
^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/utils/db.py", line 1373, in create_global_lock
conn.execute(text("SET LOCK TIMEOUT TO DEFAULT"))
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/future/engine.py", line 286, in execute
return self._execute_20(
^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1710, in _execute_20
return meth(self, args_10style, kwargs_10style, execution_options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection
return connection._execute_clauseelement(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1577, in _execute_clauseelement
ret = self._execute_context(
^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1953, in _execute_context
self._handle_dbapi_exception(
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 2134, in _handle_dbapi_exception
util.raise_(
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
raise exception
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1910, in _execute_context
self.dialect.do_execute(
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
cursor.execute(statement, parameters)
qlalchemy.exc.InternalError: (psycopg2.errors.InFailedSqlTransaction) current transaction is aborted, commands ignored until end of transaction block
SQL: SET LOCK TIMEOUT TO DEFAULT]
Background on this error at: https://sqlalche.me/e/14/2j85)`
Also, in my setup, I have 8 `api-server` replicas deployed, each running 64 workers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Edge worker sometimes fails on startup with 405: Method not allowed [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on issue #60261:
URL: https://github.com/apache/airflow/issues/60261#issuecomment-3740333928
So before raising a fix we need to triage the root cause to fix it on the "right level". If you receive the lock timeout error I suspect the API extension for Edge is not starting up, means it is not active.
This means if the Edge worker calls the backend then there is just the API endpoint missing. FastAPI responds with a HTTP 405 signalling that the "method not allowed" == there is no API endpoint. Unfortunately HTTP 405 seems to be the default instead of a HTTP 404 (Not Found).
So probably we need to things:
1. Improved Error Handling - respond on HTTP 405 to log the same error as if HTTP 404 is hit
2. Understand the root cause of the DB Locking issue. Might also be a permission limit in Airflow. More logs are needed. Then the startup of UI Plugin should be improved to either fix the problem or if un-critical (I assume/remember the locking is only used when migration is needed - assuming no migration needed) then skip the problem and continue with a warning.
Before raising a fix, I see you use Edge 1.4.0 - can you please upgrade to Edge 2.0.1 and also to most recent patch version of Airflow and reproduce?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Edge worker sometimes fails on startup with 405: Method not allowed [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on issue #60261:
URL: https://github.com/apache/airflow/issues/60261#issuecomment-3746664106
Thanks for the details. Yeah never tested 8 API servers starting at the same time - whereas I'd have assumed migration does not lock for long and might queue for 8x 1s ... interesting that it runs longer and in timeout.
So a workaround then is not starting all 8 API server at the same time? Does this help?
Yes and agree - then the best approach would be to change the current migration logic from hard coding in the plugin to a real DB migration job. It was only made because at point of first migration the utility was not released as Airflow 3 was not out and later was not possible as Airflow 2 was lagging that feature. As Airflow 2.x support is dropped now, migration could change to "real" DB migration tooling for providers like in FAB.
As all versions since Airflow 3 support in edge have not changed DB, actually the full migration can be removed and initial setup for alembic could be added. So any of the volunteers, would be cool to fix this!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Edge worker sometimes fails on startup with 405: Method not allowed [airflow]
Posted by "suii2210 (via GitHub)" <gi...@apache.org>.
suii2210 commented on issue #60261:
URL: https://github.com/apache/airflow/issues/60261#issuecomment-3730397075
I’d like to work on this issue. I can investigate the edge worker startup failures and work toward a fix or mitigation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Edge worker sometimes fails on startup with 405: Method not allowed [airflow]
Posted by "Subham-KRLX (via GitHub)" <gi...@apache.org>.
Subham-KRLX commented on issue #60261:
URL: https://github.com/apache/airflow/issues/60261#issuecomment-3724961530
@ron-gaist i would like to fix this issue can you please assign this issue to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Edge worker sometimes fails on startup with 405: Method not allowed [airflow]
Posted by "ron-gaist (via GitHub)" <gi...@apache.org>.
ron-gaist commented on issue #60261:
URL: https://github.com/apache/airflow/issues/60261#issuecomment-3743566962
After investigating further, I found in the traceback that the failure happens in the `_get_api_endpoint` function in the file `airflow.providers.edge3.plugins.edge_executor_plugin`. When I looked at that function, I saw it tries to acquire a lock and run migrations during each `api-server` process startup (we had 8 `api-server` replicas with 64 workers each).
With multiple `api-server` processes starting at the same time, most of them time out while waiting on `pg_advisory_lock`. As a result, many `api-server` processes fail to load the Edge API endpoints, which is what causes the 405 error.
Based on my current investigation, it seems the solution is to move the migrations to Alembic and run them only once via the `migrateDatabaseJob`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Edge worker sometimes fails on startup with 405: Method not allowed [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on issue #60261:
URL: https://github.com/apache/airflow/issues/60261#issuecomment-3957800573
I assume the bug is fixed by https://github.com/apache/airflow/pull/61155 as the long lasting lock is removed by migrating to new DB Manager. Thanks @dheerajturaga for making this!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org