You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/05/06 12:46:10 UTC
[GitHub] [airflow] ranamit112 opened a new issue #15698: task_instance_mutation_hook not called by scheduler when importing airflow.models.taskinstance
ranamit112 opened a new issue #15698:
URL: https://github.com/apache/airflow/issues/15698
<!--
Welcome to Apache Airflow! For a smooth issue process, try to answer the following questions.
Don't worry if they're not all applicable; just try to include what you can :-)
If you need to include code snippets or logs, please put them in fenced code
blocks. If they're super-long, please use the details tag like
<details><summary>super-long log</summary> lots of stuff </details>
Please delete these comment blocks before submitting the issue.
-->
<!--
IMPORTANT!!!
PLEASE CHECK "SIMILAR TO X EXISTING ISSUES" OPTION IF VISIBLE
NEXT TO "SUBMIT NEW ISSUE" BUTTON!!!
PLEASE CHECK IF THIS ISSUE HAS BEEN REPORTED PREVIOUSLY USING SEARCH!!!
Please complete the next sections or the issue will be closed.
These questions are the first thing we need to know to understand the context.
-->
**Apache Airflow version**: 1.10.12 (also tested with 1.10.15, 2.0.2 but less extensively)
**Kubernetes version (if you are using kubernetes)** (use `kubectl version`): None
**Environment**: Linux / Docker
- **Cloud provider or hardware configuration**: None
- **OS** (e.g. from /etc/os-release): Red Hat Enterprise Linux Server 7.9 (Maipo)
- **Kernel** (e.g. `uname -a`): Linux d7b9410c0f25 4.19.104-microsoft-standard #1 SMP Wed Feb 19 06:37:35 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
- **Install tools**:
- **Others**: Tested on both real Linux (Red Hat) and a docker inside a windows machine.
**What happened**: Custom `task_instance_mutation_hook` not called by scheduler even though airflow loads the `airflow_local_settings` module.
**What you expected to happen**:
`task_instance_mutation_hook` called before every task instance run.
I think the way `airflow.models.dagrun` loads `task_instance_mutation_hook` from `airflow_local_settings` does not work when `airflow_local_settings` imports `airflow.models.taskinstance` or `airflow.models.dagrun`.
**How to reproduce it**:
1. Added `airflow_local_settings.py` to \{AIRFLOW_HOME\}\config
```python
import logging
from airflow.models.taskinstance import TaskInstance
def task_instance_mutation_hook(ti: TaskInstance):
logging.getLogger("").warning("HERE IN task_instance_mutation_hook log")
print("HERE IN task_instance_mutation_hook")
ti.queue = "X"
```
2. See output `[2021-05-06 11:13:04,076] {settings.py:392} INFO - Loaded airflow_local_settings from /usr/local/airflow/config/airflow_local_settings.py.`
3. function is never called - log/print is not written and queue does not update.
4. Additionally, example code to reproduce code issue:
```python
import airflow
import airflow.models.dagrun
import inspect
print(inspect.getfile(airflow.settings.task_instance_mutation_hook))
print(inspect.getfile(airflow.models.dagrun.task_instance_mutation_hook))
```
outputs
```
/usr/local/airflow/config/airflow_local_settings.py
/opt/bb/lib/python3.7/site-packages/airflow/settings.py
```
5. when removing `from airflow.models.taskinstance import TaskInstance` from airflow_local_settings.py everything works as expected.
**Anything else we need to know**:
BTW, do the logs printed from `task_instance_mutation_hook` go anywhere? Even after I remove the import and the queue is update, I can't see anything in the logs files or in the scheduler console.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] MatrixManAtYrService edited a comment on issue #15698: task_instance_mutation_hook not called by scheduler when importing airflow.models.taskinstance
Posted by GitBox <gi...@apache.org>.
MatrixManAtYrService edited a comment on issue #15698:
URL: https://github.com/apache/airflow/issues/15698#issuecomment-842810436
If I do:
```python3
def task_instance_mutation_hook(ti):
print("MUTATION HOOK CALLED")
```
and run `airflow dags test mydagname $(date "+%Y-%m-%d")`, the message appears in stdout:
```
[2021-05-18 04:03:43,472] {dagbag.py:487} INFO - Filling up the DagBag from /usr/local/airflow/dags
MUTATION_HOOK_CALLED
INFO - Adding to queue: ['<TaskInstance: mydagname.mytaskname ...
```
If I add the import and the type hint as shown above, the message stops showing up.
Edit: Oops, I had a version before the fix, so I haven't tested the fix--just replicated the bug.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] MatrixManAtYrService commented on issue #15698: task_instance_mutation_hook not called by scheduler when importing airflow.models.taskinstance
Posted by GitBox <gi...@apache.org>.
MatrixManAtYrService commented on issue #15698:
URL: https://github.com/apache/airflow/issues/15698#issuecomment-842810436
If I do:
```python3
def task_instance_mutation_hook(ti):
print("MUTATION HOOK CALLED")
```
and run `airflow dags test mydagname $(date "+%Y-%m-%d")`, the message appears in stdout:
```
[2021-05-18 04:03:43,472] {dagbag.py:487} INFO - Filling up the DagBag from /usr/local/airflow/dags
MUTATION_HOOK_CALLED
INFO - Adding to queue: ['<TaskInstance: mydagname.mytaskname ...
```
If I add the import and the type hint as shown above, the message stops showing up. I still need to check whether the image I'm testing with includes the fix--but I think I'm replicating it at least.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb closed issue #15698: task_instance_mutation_hook not called by scheduler when importing airflow.models.taskinstance
Posted by GitBox <gi...@apache.org>.
ashb closed issue #15698:
URL: https://github.com/apache/airflow/issues/15698
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org