You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "jun-roh (via GitHub)" <gi...@apache.org> on 2025/12/16 01:36:43 UTC
[D] Airflow 3.1.3 – API Server returns 500 Internal Server Error on /login once per day (session invalidation / PendingRollbackError) [airflow]
GitHub user jun-roh created a discussion: Airflow 3.1.3 – API Server returns 500 Internal Server Error on /login once per day (session invalidation / PendingRollbackError)
Hello,
We are setting up an environment using Apache Airflow 3.1.3.
We are encountering an issue where the login session expires once per day, and after that, accessing the login page results in a 500 Internal Server Error from the API server.
This happens consistently and requires restarting the API server to recover.
Below are the error logs, configuration, and our custom AuthManager implementation.
Any help or guidance would be greatly appreciated.
### **Environment**
• Airflow version: 3.1.3
• Python version: 3.12
• Executor: CeleryExecutor
• Database: MySQL (RDS)
• Auth Manager: Custom SafeFabAuthManager (based on FAB)
• Deployment: Docker Compose
• API Server: airflow api-server --apps all --workers 1
### **Issue Summary**
• Login works initially
• After ~24 hours:
• Login session becomes invalid
• /auth/login/ returns 500 Internal Server Error
• Stack trace shows sqlalchemy.exc.PendingRollbackError
• Restarting airflow-apiserver resolves the issue temporarily
### **Error Log (Excerpt)**
```
sqlalchemy.exc.PendingRollbackError: Can't reconnect until invalid transaction is rolled back.
```
The error occurs during:
```
/login/ [GET]
airflow.providers.fab.auth_manager.security_manager.override.load_user
```
### **Docker Compose (airflow-apiserver)**
```
airflow-apiserver:
command: >
airflow api-server --host 0.0.0.0 --port 8080 --workers 1 --apps all
```
Database pooling configuration:
```
AIRFLOW__DATABASE__SQL_ALCHEMY_ENGINE_OPTIONS={
"pool_pre_ping": true,
"pool_recycle": 280,
"pool_size": 100,
"max_overflow": 20,
"pool_timeout": 30,
"pool_reset_on_return": "rollback"
}
```
### **Custom Auth Manager**
To mitigate the issue, we implemented a custom SafeFabAuthManager that wraps DB calls and forcibly resets the session when a PendingRollbackError occurs.
```
class SafeFabSecurityManager(FabAirflowSecurityManagerOverride):
def _safe_db_call(self, func, *args, **kwargs):
try:
return func(*args, **kwargs)
except Exception as e:
if isinstance(e, (SQLAlchemyError, PendingRollbackError)):
session = self._get_session()
if session:
session.rollback()
session.close()
return func(*args, **kwargs)
raise
```
This reduces the frequency of failures but does not fully eliminate the issue.
### **Questions**
1. Is this a known issue in Airflow 3.1.x related to FAB / API Server session handling?
2. Is the API Server expected to manage SQLAlchemy sessions differently compared to the Webserver?
3. Are there recommended settings for:
• pool_recycle
• pool_reset_on_return
• session scoping in FAB auth?
4. Would using NullPool or disabling connection reuse for the API server be recommended?
### **Expected Behavior**
• Login page should gracefully redirect to re-authentication
• No 500 errors on /login
• Expired sessions should not poison the SQLAlchemy connection pool
GitHub link: https://github.com/apache/airflow/discussions/59487
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Airflow 3.1.3 – API Server returns 500 Internal Server Error on /login once per day (session invalidation / PendingRollbackError) [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
GitHub user potiuk added a comment to the discussion: Airflow 3.1.3 – API Server returns 500 Internal Server Error on /login once per day (session invalidation / PendingRollbackError)
I believe there were some workarounds posted in the meantime
GitHub link: https://github.com/apache/airflow/discussions/59487#discussioncomment-15268664
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Airflow 3.1.3 – API Server returns 500 Internal Server Error on /login once per day (session invalidation / PendingRollbackError) [airflow]
Posted by "dsuarez (via GitHub)" <gi...@apache.org>.
GitHub user dsuarez edited a comment on the discussion: Airflow 3.1.3 – API Server returns 500 Internal Server Error on /login once per day (session invalidation / PendingRollbackError)
I am also having this problem with Airflow 3.1.5. After a while (probably under the circumstances described), Airflow API Server starts returning Internal Server Error.
I am using apache/airflow:3.1.5 docker image. Is there an updated image without this problem?
GitHub link: https://github.com/apache/airflow/discussions/59487#discussioncomment-15893877
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Airflow 3.1.3 – API Server returns 500 Internal Server Error on /login once per day (session invalidation / PendingRollbackError) [airflow]
Posted by "raoakmal55 (via GitHub)" <gi...@apache.org>.
GitHub user raoakmal55 added a comment to the discussion: Airflow 3.1.3 – API Server returns 500 Internal Server Error on /login once per day (session invalidation / PendingRollbackError)
I am also facing this issue and i have tried to add pgbouncer as well but no luck so far. By Adding those parameters we can just reducued the frequency of the issue but its still there. Here are my configs
```yaml
extraEnv: |
- name: AIRFLOW__DATABASE__SQL_ALCHEMY_POOL_PRE_PING
value: "True"
- name: AIRFLOW__DATABASE__SQL_ALCHEMY_POOL_RESET_ON_RETURN
value: "rollback"
- name: AIRFLOW__DATABASE__SQL_ALCHEMY_POOL_RECYCLE
value: "300"
- name: AIRFLOW__DATABASE__SQL_ALCHEMY_POOL_TIMEOUT
value: "30"
- name: AIRFLOW__DATABASE__SQL_ALCHEMY_ENGINE_ARGS
value: '{"pool_pre_ping": true}'
```
GitHub link: https://github.com/apache/airflow/discussions/59487#discussioncomment-16570238
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Airflow 3.1.3 – API Server returns 500 Internal Server Error on /login once per day (session invalidation / PendingRollbackError) [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
GitHub user potiuk added a comment to the discussion: Airflow 3.1.3 – API Server returns 500 Internal Server Error on /login once per day (session invalidation / PendingRollbackError)
Also make sure to upgrade to latest Fab provider.
GitHub link: https://github.com/apache/airflow/discussions/59487#discussioncomment-15264564
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Airflow 3.1.3 – API Server returns 500 Internal Server Error on /login once per day (session invalidation / PendingRollbackError) [airflow]
Posted by "raoakmal55 (via GitHub)" <gi...@apache.org>.
GitHub user raoakmal55 added a comment to the discussion: Airflow 3.1.3 – API Server returns 500 Internal Server Error on /login once per day (session invalidation / PendingRollbackError)
@jun-roh Are you able to fix that issue after upgrading the airflow to 3.1.5. If yes can you please share the pool settings that you used. Thanks alot
GitHub link: https://github.com/apache/airflow/discussions/59487#discussioncomment-16570275
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Airflow 3.1.3 – API Server returns 500 Internal Server Error on /login once per day (session invalidation / PendingRollbackError) [airflow]
Posted by "arodrber-ds (via GitHub)" <gi...@apache.org>.
GitHub user arodrber-ds added a comment to the discussion: Airflow 3.1.3 – API Server returns 500 Internal Server Error on /login once per day (session invalidation / PendingRollbackError)
I was also having this error, so I fixed it by setting to this version: .
So far, the problem hasn't occurred again.apache-airflow-providers-fab==3.0.2
GitHub link: https://github.com/apache/airflow/discussions/59487#discussioncomment-15798886
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Airflow 3.1.3 – API Server returns 500 Internal Server Error on /login once per day (session invalidation / PendingRollbackError) [airflow]
Posted by "jun-roh (via GitHub)" <gi...@apache.org>.
GitHub user jun-roh added a comment to the discussion: Airflow 3.1.3 – API Server returns 500 Internal Server Error on /login once per day (session invalidation / PendingRollbackError)
We are currently building our Airflow image based on
apache/airflow:3.1.5-python3.13.
If we configure Airflow with the following setting:
```
AIRFLOW__CORE__AUTH_MANAGER: airflow.providers.fab.auth_manager.FabAuthManager
```
does this mean that we can rely entirely on the default FAB-based authentication manager?
In that case, would it be correct to assume that we no longer need to implement a custom AuthManager?
More specifically, does the default FabAuthManager fully handle authentication and session management for both the Webserver and the API Server in Airflow 3.1.5?
Thank you very much for your guidance.
GitHub link: https://github.com/apache/airflow/discussions/59487#discussioncomment-15264618
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Airflow 3.1.3 – API Server returns 500 Internal Server Error on /login once per day (session invalidation / PendingRollbackError) [airflow]
Posted by "dsuarez (via GitHub)" <gi...@apache.org>.
GitHub user dsuarez added a comment to the discussion: Airflow 3.1.3 – API Server returns 500 Internal Server Error on /login once per day (session invalidation / PendingRollbackError)
I am also having this problem with Airfloe 3.1.5. After a while (probably under the circunstances described), Airflow API Server starts returning Internal Server Error.
I am using apache/airflow:3.1.5 image. Is there an updated image without this problem?
GitHub link: https://github.com/apache/airflow/discussions/59487#discussioncomment-15893877
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Airflow 3.1.3 – API Server returns 500 Internal Server Error on /login once per day (session invalidation / PendingRollbackError) [airflow]
Posted by "elangovankrishna (via GitHub)" <gi...@apache.org>.
GitHub user elangovankrishna added a comment to the discussion: Airflow 3.1.3 – API Server returns 500 Internal Server Error on /login once per day (session invalidation / PendingRollbackError)
We recently upgraded to Airflow 3.1.5 and ran in the frequent 500 Internal Server Error from the API server, this was more prevalent on the PROD env because of more number of users, this is what i have implemented and it seems to be working
We were specifically running into this issue
issue 500 Internal Server Error with the message mismatching_state: CSRF Warning! State not equal in request and response. alongside the sqlalchemy.exc.PendingRollbackError in our Airflow Prod and Dev environments.
# Database Connection Pool Resilience to stop PendingRollbackErrors / CSRF state
- name: AIRFLOW__DATABASE__SQL_ALCHEMY_POOL_PRE_PING
value: "True"
- name: AIRFLOW__DATABASE__SQL_ALCHEMY_POOL_RECYCLE
value: "1800"
GitHub link: https://github.com/apache/airflow/discussions/59487#discussioncomment-16490887
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Airflow 3.1.3 – API Server returns 500 Internal Server Error on /login once per day (session invalidation / PendingRollbackError) [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
GitHub user potiuk added a comment to the discussion: Airflow 3.1.3 – API Server returns 500 Internal Server Error on /login once per day (session invalidation / PendingRollbackError)
I don't sure what you mean by "fully handle" - but I think there are some open issues for API auth wiht Fab Auth Manager - you might need to look up through issues and find out.
GitHub link: https://github.com/apache/airflow/discussions/59487#discussioncomment-15268661
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Airflow 3.1.3 – API Server returns 500 Internal Server Error on /login once per day (session invalidation / PendingRollbackError) [airflow]
Posted by "dsuarez (via GitHub)" <gi...@apache.org>.
GitHub user dsuarez edited a comment on the discussion: Airflow 3.1.3 – API Server returns 500 Internal Server Error on /login once per day (session invalidation / PendingRollbackError)
I am also having this problem with Airflow 3.1.5. After a while (probably under the circunstances described), Airflow API Server starts returning Internal Server Error.
I am using apache/airflow:3.1.5 docker image. Is there an updated image without this problem?
GitHub link: https://github.com/apache/airflow/discussions/59487#discussioncomment-15893877
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Airflow 3.1.3 – API Server returns 500 Internal Server Error on /login once per day (session invalidation / PendingRollbackError) [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
GitHub user potiuk added a comment to the discussion: Airflow 3.1.3 – API Server returns 500 Internal Server Error on /login once per day (session invalidation / PendingRollbackError)
Yes. this has been fixed in 3.1.5 - look th the changelog
GitHub link: https://github.com/apache/airflow/discussions/59487#discussioncomment-15264557
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Airflow 3.1.3 – API Server returns 500 Internal Server Error on /login once per day (session invalidation / PendingRollbackError) [airflow]
Posted by "giuseppevavalaqi (via GitHub)" <gi...@apache.org>.
GitHub user giuseppevavalaqi added a comment to the discussion: Airflow 3.1.3 – API Server returns 500 Internal Server Error on /login once per day (session invalidation / PendingRollbackError)
Same thing for me.
GitHub link: https://github.com/apache/airflow/discussions/59487#discussioncomment-15894623
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org