You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/08/17 15:20:33 UTC
[GitHub] [airflow] khmelevskiy opened a new issue, #25768: Dag could not be found with AIRFLOW__CORE__LAZY_LOAD_PLUGINS:false and using partial/expand
khmelevskiy opened a new issue, #25768:
URL: https://github.com/apache/airflow/issues/25768
### Apache Airflow version
2.3.3
### What happened
Hi!
We are setting up the [datahub] plugin (https://datahubproject.io/docs/lineage/airflow/) and for it to work you need to set the config:
`AIRFLOW__CORE__LAZY_LOAD_PLUGINS: false`
when installing this config, dags with partial/expand stopped working for us.
Error from airflow worker: `Failed to execute task Dag 'repeated_mapping' could not be found; either it does not exist or it failed to parse.`
Example from your [site](https://airflow.apache.org/docs/apache-airflow/2.3.3/concepts/dynamic-task-mapping.html#repeated-mapping):
```
import pendulum
from airflow.decorators import task, dag
from datetime import timedelta
@dag(
start_date=pendulum.today("UTC").subtract(days=1),
schedule_interval="10 * * * *",
default_args={
"owner": "etl",
"retries": 3,
"depends_on_past": False,
"retry_delay": timedelta(minutes=3),
},
max_active_runs=1,
catchup=False,
tags=["test"],
)
def repeated_mapping():
@task
def add_one(x: int):
return x + 1
first = add_one.expand(x=[1, 2, 3])
second = add_one.expand(x=first)
repeated_mapping_dag = repeated_mapping()
```
please help us solve the problem
### What you think should happen instead
_No response_
### How to reproduce
_No response_
### Operating System
FROM --platform=linux/amd64 apache/airflow:2.3.3-python3.9
### Versions of Apache Airflow Providers
apache-airflow-providers-amazon==2.4.0
apache-airflow-providers-celery==3.0.0
apache-airflow-providers-cncf-kubernetes==4.1.0
apache-airflow-providers-common-sql==1.0.0
apache-airflow-providers-docker==3.0.0
apache-airflow-providers-elasticsearch==4.0.0
apache-airflow-providers-ftp==3.0.0
apache-airflow-providers-google==8.0.0
apache-airflow-providers-grpc==3.0.0
apache-airflow-providers-hashicorp==3.0.0
apache-airflow-providers-http==3.0.0
apache-airflow-providers-imap==3.0.0
apache-airflow-providers-jdbc==2.0.1
apache-airflow-providers-microsoft-azure==4.0.0
apache-airflow-providers-microsoft-mssql==3.2.0
apache-airflow-providers-mysql==2.2.3
apache-airflow-providers-odbc==3.0.0
apache-airflow-providers-postgres==5.0.0
apache-airflow-providers-redis==3.0.0
apache-airflow-providers-sendgrid==3.0.0
apache-airflow-providers-sftp==3.0.0
apache-airflow-providers-slack==5.0.0
apache-airflow-providers-sqlite==3.0.0
apache-airflow-providers-ssh==3.0.0
### Deployment
Other 3rd-party Helm chart
### Deployment details
[Helm Chart](https://github.com/airflow-helm/charts/tree/main/charts/airflow)
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] khmelevskiy commented on issue #25768: “DAG could not be found” in worker when using task-mapping with lazy_loading_plugins=false
Posted by GitBox <gi...@apache.org>.
khmelevskiy commented on issue #25768:
URL: https://github.com/apache/airflow/issues/25768#issuecomment-1323197327
I correctly understood that no one is going to correct this error?
@uranusjr @eladkal ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] khmelevskiy closed issue #25768: “DAG could not be found” in worker when using task-mapping with lazy_loading_plugins=false
Posted by "khmelevskiy (via GitHub)" <gi...@apache.org>.
khmelevskiy closed issue #25768: “DAG could not be found” in worker when using task-mapping with lazy_loading_plugins=false
URL: https://github.com/apache/airflow/issues/25768
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #25768: “DAG could not be found” in worker when using task-mapping with lazy_loading_plugins=false
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #25768:
URL: https://github.com/apache/airflow/issues/25768#issuecomment-1323251622
As usual in open-source if someone fixes it, it will be fixed.
What I suggest (I see that issue only now) that there is very little information in the issue. I believe you misdiagnosed the impact of ally load plugin, i suspect that the two plugins you mentioned are causing the issue - not Airflow itself. Causing third party plugins is always a risk because they can manipulate Python codein a way that make them incompatible with Airflow new features (and it's the responsibility of the 3rd-party plugins developers to keep their code updated..
I presume you already raised issue with them ? Could you point us to the issues you raised ? What did they tell? Maybe they already released a version that is compatible with dynamic task mapping? If you did not do it yet - then i think you should start with them (and likely they want to know there is a problem, so raising it to them is a very good thing for them and community. You might even link that issue when you report it to them
Anyway - maybe we can help here with diagnosing this as well, but definitely more info and input will be needed. More specifically you need to show more information - stack traces and specific circumstances where it happens - and likely the same information you will have to pass to the plugin developers. This is the best way you can help us to help you (and the third party plugin developers) to maybe narrow down the reason why their plugins are causing the problems - but ultimately it should be them to analyse that and fix them, not is, atost we can try to help to see where it comes from. Reporting those details is also important - maybe while looking at it (and getting some information from the plugin developers - they might be able to propose some changes that will make it easier for them to fix the problem.
To summarize - if you want to continue using the plugins - providing more details here and subsequent opening issues with the plugin developers is likely the best thing you can do to speed up analysis, help and possible fixes (likely from the plugin developers).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] khmelevskiy commented on issue #25768: Dag could not be found with AIRFLOW__CORE__LAZY_LOAD_PLUGINS:false and using partial/expand
Posted by GitBox <gi...@apache.org>.
khmelevskiy commented on issue #25768:
URL: https://github.com/apache/airflow/issues/25768#issuecomment-1219037453
we also found that the problem appears after installing `acryl-datahub-airflow-plugin`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #25768: Dag could not be found with AIRFLOW__CORE__LAZY_LOAD_PLUGINS:false and using partial/expand
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #25768:
URL: https://github.com/apache/airflow/issues/25768#issuecomment-1218154515
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org