You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/03/01 13:29:18 UTC
[GitHub] [airflow] HannoGao opened a new issue #21895: Cannot get (set) airflow variables within the scheduler
HannoGao opened a new issue #21895:
URL: https://github.com/apache/airflow/issues/21895
### Apache Airflow version
2.2.4 (latest released)
### What happened
```
from airflow.models import Variable
var = Variable.get(key_var)
```
When this is called during a special function during the normal scheduling job e.g. from somewhere in scheduler_job e.g. here [scheduler_job](https://github.com/apache/airflow/blob/737c0efff929710c0a5abbaa8de3575a55297aa5/airflow/jobs/scheduler_job.py#L413)
gives the following error:
in [variable.py](https://github.com/apache/airflow/blob/6b0ca646ec849af91fe8a10d3d5656cafa3ed4bd/airflow/models/variable.py#L256)
```
[2022-03-01 12:11:05,768] {variable.py:271} ERROR - Unable to retrieve variable from secrets backend (MetastoreBackend). Checking subsequent secrets backend.
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/airflow/models/variable.py", line 267, in get_variable_from_secrets
var_val = secrets_backend.get_variable(key=key)
File "/usr/local/lib/python3.8/dist-packages/airflow/utils/session.py", line 70, in wrapper
return func(*args, session=session, **kwargs)
File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__
next(self.gen)
File "/usr/local/lib/python3.8/dist-packages/airflow/utils/session.py", line 32, in create_session
session.commit()
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1046, in commit
self.transaction.commit()
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 504, in commit
self._prepare_impl()
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 472, in _prepare_impl
self.session.dispatch.before_commit(self.session)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/event/attr.py", line 322, in __call__
fn(*args, **kw)
File "/usr/local/lib/python3.8/dist-packages/airflow/utils/sqlalchemy.py", line 265, in _validate_commit
raise RuntimeError("UNEXPECTED COMMIT - THIS WILL BREAK HA LOCKS!")
RuntimeError: UNEXPECTED COMMIT - THIS WILL BREAK HA LOCKS!
```
This is not the case, when calling the same functionality via the RestAPI.
Getting variables during the scheduling was possibile e.g. in Airflow 1.
### What you expected to happen
RuntimeError("UNEXPECTED COMMIT - THIS WILL BREAK HA LOCKS!") in sqlalchemy. Probably due to the fact, the function is called during scheduling.
### How to reproduce
Set and Get a "test Variable in the scheduler and trigger a dag.
### Operating System
container with ubuntu 20.04
### Versions of Apache Airflow Providers
_No response_
### Deployment
Other Docker-based deployment
### Deployment details
microk8s version 1.18/stable, container for scheduler, webserver and postgres
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #21895: Cannot get (set) airflow variables within the scheduler
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #21895:
URL: https://github.com/apache/airflow/issues/21895
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #21895: Cannot get (set) airflow variables within the scheduler
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #21895:
URL: https://github.com/apache/airflow/issues/21895#issuecomment-1055537825
This is as excepcted. You SHOULD NOT modify scheduler code. VERY BAD things happen if you start doing it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #21895: Cannot get (set) airflow variables within the scheduler
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #21895:
URL: https://github.com/apache/airflow/issues/21895#issuecomment-1055444314
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org