You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "arjav1528 (via GitHub)" <gi...@apache.org> on 2026/01/15 16:06:07 UTC
[PR] fix(edge3): Resolve edge worker 405 error by deferring table creation to FastAPI startup [airflow]
arjav1528 opened a new pull request, #60596:
URL: https://github.com/apache/airflow/pull/60596
**Closes:** #60261
## Summary
This PR fixes the intermittent `405 Method Not Allowed` error that edge workers encounter during startup in high-concurrency environments (e.g., 8 API servers × 64 worker processes).
## Root Cause
The edge executor plugin was acquiring a global database lock (`DBLocks.MIGRATIONS`) **at class definition time** (plugin import) to create tables. When many API server processes started simultaneously, they all competed for the same lock, causing:
1. `psycopg2.errors.LockNotAvailable` errors on lock timeout
2. Plugin import failures → edge worker endpoints not registered
3. Edge workers receiving `405 Method Not Allowed` for `/edge_worker/v1/worker/`
## Solution
- Moved table creation from plugin import time to a **FastAPI startup event**
- Plugin import no longer blocks on database operations
- Added retry mechanism (5 attempts) for lock acquisition
- Exponential backoff with jitter to avoid thundering herd
- Proper logging for debugging
- Check if tables exist **before** attempting to acquire lock
- Skip lock entirely during normal operation (tables already exist)
- Double-check pattern after lock acquisition to handle race conditions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(edge3): Resolve edge worker 405 error by deferring table creation to FastAPI startup [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on PR #60596:
URL: https://github.com/apache/airflow/pull/60596#issuecomment-3770098664
Still draft and a couple of errors... so WIP but you can compare your code with this here for example: https://github.com/apache/airflow/pull/60664/changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(edge3): Resolve edge worker 405 error by deferring table creation to FastAPI startup [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl closed pull request #60596: fix(edge3): Resolve edge worker 405 error by deferring table creation to FastAPI startup
URL: https://github.com/apache/airflow/pull/60596
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(edge3): Resolve edge worker 405 error by deferring table creation to FastAPI startup [airflow]
Posted by "arjav1528 (via GitHub)" <gi...@apache.org>.
arjav1528 commented on PR #60596:
URL: https://github.com/apache/airflow/pull/60596#issuecomment-3770092119
@jscheffl @dheerajturaga I have applied the suggestewd changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(edge3): Resolve edge worker 405 error by deferring table creation to FastAPI startup [airflow]
Posted by "arjav1528 (via GitHub)" <gi...@apache.org>.
arjav1528 commented on PR #60596:
URL: https://github.com/apache/airflow/pull/60596#issuecomment-3792218452
> Soryy... no. You did not.
can you check now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(edge3): Resolve edge worker 405 error by deferring table creation to FastAPI startup [airflow]
Posted by "dheerajturaga (via GitHub)" <gi...@apache.org>.
dheerajturaga commented on PR #60596:
URL: https://github.com/apache/airflow/pull/60596#issuecomment-3770103701
I have separated the airflow-core changes to support the future implementation of EdgeDBManager here #60752
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(edge3): Resolve edge worker 405 error by deferring table creation to FastAPI startup [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on PR #60596:
URL: https://github.com/apache/airflow/pull/60596#issuecomment-3797289936
Sorry, I hope I am not wrong but following the discussion and also reminding that feedback was not listened... this smells a bit like AI slop. Closing therefore.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(edge3): Resolve edge worker 405 error by deferring table creation to FastAPI startup [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on code in PR #60596:
URL: https://github.com/apache/airflow/pull/60596#discussion_r2696048460
##########
scripts/ci/prek/check_shared_distributions_usage.py:
##########
Review Comment:
These changes are unrelated. Might be a good general improvement for typing... but please separate this to another PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(edge3): Resolve edge worker 405 error by deferring table creation to FastAPI startup [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on PR #60596:
URL: https://github.com/apache/airflow/pull/60596#issuecomment-3770094773
Soryy... no. You did not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org