You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/09/19 14:07:50 UTC
[GitHub] [airflow] khalidmammadov opened a new pull request #18359: Session propagation for task_failed_deps command
khalidmammadov opened a new pull request #18359:
URL: https://github.com/apache/airflow/pull/18359
When failed dependencies checked as in below example, the command fails most of the time and some time succeeds. This is due to race condition for the session object.
It happens as dependency checker tries to access **ti.execution_date** attribute which is **lazily** loaded and when load is required the session is not available anymore as it was created temporarily for the duration _get_ti call.
This PR fixes it by creating session beforehand and propagating to downstream. This helps avoiding of creating multiple sessions and making sure session is available when this lazy attribute is accessed.
I have also took opportunity to fix a typo for "**create_if_necssary**" -> "**create_if_necessary**"
Thanks for reviewing!
root@c10237c871db:/opt/airflow/airflow/cli/commands# airflow tasks failed-deps my_dag print_date manual__2021-09-11T00:00:00+00:00
[2021-09-19 13:16:44,512] {dagbag.py:492} INFO - Filling up the DagBag from /files/dags
Traceback (most recent call last):
File "/usr/local/bin/airflow", line 33, in <module>
sys.exit(load_entry_point('apache-airflow', 'console_scripts', 'airflow')())
File "/opt/airflow/airflow/__main__.py", line 40, in main
args.func(args)
File "/opt/airflow/airflow/cli/cli_parser.py", line 48, in command
return func(*args, **kwargs)
File "/opt/airflow/airflow/utils/cli.py", line 92, in wrapper
return f(*args, **kwargs)
File "/opt/airflow/airflow/cli/commands/task_command.py", line 312, in task_failed_deps
failed_deps = list(ti.get_failed_dep_statuses(dep_context=dep_context))
File "/opt/airflow/airflow/models/taskinstance.py", line 1042, in get_failed_dep_statuses
for dep_status in dep.get_dep_statuses(self, session, dep_context):
File "/opt/airflow/airflow/ti_deps/deps/base_ti_dep.py", line 101, in get_dep_statuses
yield from self._get_dep_statuses(ti, session, dep_context)
File "/opt/airflow/airflow/ti_deps/deps/exec_date_after_start_date_dep.py", line 31, in _get_dep_statuses
if ti.task.start_date and ti.execution_date < ti.task.start_date:
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/ext/associationproxy.py", line 193, in __get__
return inst.get(obj)
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/ext/associationproxy.py", line 546, in get
target = getattr(obj, self.target_collection)
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/attributes.py", line 294, in __get__
return self.impl.get(instance_state(instance), dict_)
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/attributes.py", line 730, in get
value = self.callable_(state, passive)
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/strategies.py", line 720, in _load_for_state
% (orm_util.state_str(state), self.key)
sqlalchemy.orm.exc.DetachedInstanceError: Parent instance <TaskInstance at 0x7fad6e3acc88> is not bound to a Session; lazy load operation of attribute 'dag_run' cannot proceed (Background on this error at: http://sqlalche.me/e/13/bhk3)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] khalidmammadov commented on pull request #18359: Session propagation for task_failed_deps command
Posted by GitBox <gi...@apache.org>.
khalidmammadov commented on pull request #18359:
URL: https://github.com/apache/airflow/pull/18359#issuecomment-926675458
It is on latest master version:
Branch name: main
Docker image: ghcr.io/apache/airflow/main/ci/python3.6
Airflow source version: 2.2.0.dev0
Python version: 3.6
Backend: sqlite
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on pull request #18359: Session propagation for task_failed_deps command
Posted by GitBox <gi...@apache.org>.
uranusjr commented on pull request #18359:
URL: https://github.com/apache/airflow/pull/18359#issuecomment-926244425
Could you clarify your Airflow setup and version?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil closed pull request #18359: Session propagation for task_failed_deps command
Posted by GitBox <gi...@apache.org>.
kaxil closed pull request #18359:
URL: https://github.com/apache/airflow/pull/18359
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on pull request #18359: Session propagation for task_failed_deps command
Posted by GitBox <gi...@apache.org>.
kaxil commented on pull request #18359:
URL: https://github.com/apache/airflow/pull/18359#issuecomment-936994977
closed by https://github.com/apache/airflow/pull/18499
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on pull request #18359: Session propagation for task_failed_deps command
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on pull request #18359:
URL: https://github.com/apache/airflow/pull/18359#issuecomment-922479753
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
Here are some useful points:
- Pay attention to the quality of your code (flake8, mypy and type annotations). Our [pre-commits]( https://github.com/apache/airflow/blob/main/STATIC_CODE_CHECKS.rst#prerequisites-for-pre-commit-hooks) will help you with that.
- In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/main/docs/apache-airflow/howto/custom-operator.rst) Consider adding an example DAG that shows how users should use it.
- Consider using [Breeze environment](https://github.com/apache/airflow/blob/main/BREEZE.rst) for testing locally, itโs a heavy docker but it ships with a working Airflow and a lot of integrations.
- Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
- Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
- Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#coding-style-and-best-practices).
Apache Airflow is a community-driven project and together we are making it better ๐.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org