You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@airflow.apache.org by "krisnaru (via GitHub)" <gi...@apache.org> on 2024/02/23 04:53:17 UTC

[I] Unable to import dags in dag_bag.py [airflow]

krisnaru opened a new issue, #37647:
URL: https://github.com/apache/airflow/issues/37647

   ### Apache Airflow version
   
   2.8.1
   
   ### If "Other Airflow 2 version" selected, which one?
   
   _No response_
   
   ### What happened?
   
   Unable to load DAGs using dag_bag.py also scheduler does not report any errors in logs.
   
   My code organization:
   - root/project1/dag1.py
   - root/project2/dag2.py
   - root/airflow/dag_bag.py
   
   dag_bag.py
   `
   from project1.dag1 import load_dag()
   from project1.dag2 import load_dag()
   dag1=load_dag()
   dag2=load_dag()
   `
   settings
   `
   PYTHONPATH: root
   dags_folder: root/airflow/
   `
   If i move the dag code to dags_folder, then dags show up. Is there a design limitation on DAG loading, scheduler does not load DAGs using import paths?
   
   
   ### What you think should happen instead?
   
   DAGs should be loaded using dag_bag.py successfully
   
   ### How to reproduce
   
   1. Create code structure according to below hierarchy
   - root/project1/dag1.py
   - root/project2/dag2.py
   - root/airflow/dag_bag.py
   
   2. And start scheduler
   
   ### Operating System
   
   Debian GNU/Linu
   
   ### Versions of Apache Airflow Providers
   
   Not required for this issue
   
   ### Deployment
   
   Official Apache Airflow Helm Chart
   
   ### Deployment details
   
   Kubernetes deployment
   
   ### Anything else?
   
   I can provide sample code and repro steps if needed
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Unable to import dags in dag_bag.py [airflow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.

github-actions[bot] commented on issue #37647:
URL: https://github.com/apache/airflow/issues/37647#issuecomment-2002198631

   This issue has been closed because it has not received response from the issue author.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Unable to import dags in dag_bag.py [airflow]

Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.

boring-cyborg[bot] commented on issue #37647:
URL: https://github.com/apache/airflow/issues/37647#issuecomment-1960730563

   Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Unable to import dags in dag_bag.py [airflow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.

github-actions[bot] closed issue #37647: Unable to import dags in dag_bag.py
URL: https://github.com/apache/airflow/issues/37647


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Unable to import dags in dag_bag.py [airflow]

Posted by "jscheffl (via GitHub)" <gi...@apache.org>.

jscheffl commented on issue #37647:
URL: https://github.com/apache/airflow/issues/37647#issuecomment-1962743732

   I assume the reported problem is a mis-understanding. DAGs are not loaded by the user via dag_bag but loaded by the scheduler as defined in the Dags-Folder (e.g. via `AIRFLOW__CORE__DAGS_FOLDER` or as configured as described in https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#dags-folder).
   Dags are usually NOT added to python path and you don't load them yourself - the Scheduler will iterate over the DAG folder and load automatically.
   
   The process is also described here: https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/dags.html#loading-dags
   
   If you have a different problem and I mis-understood the bug report, please describe where you try to load the DAGs. As describe your configuration, how you started your scheduler and add logs.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Unable to import dags in dag_bag.py [airflow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.

github-actions[bot] commented on issue #37647:
URL: https://github.com/apache/airflow/issues/37647#issuecomment-1987019646

   This issue has been automatically marked as stale because it has been open for 14 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Unable to import dags in dag_bag.py [airflow]

Posted by "waldoppper (via GitHub)" <gi...@apache.org>.

waldoppper commented on issue #37647:
URL: https://github.com/apache/airflow/issues/37647#issuecomment-2010138894

   For posterity's sake:
   
   In this case, the root cause was [this line](https://github.com/apache/airflow/blob/01a819a42daa7990c30ab9776208b3dcb9f3a28b/airflow/utils/file.py#L358), which filtered out `root/airflow/dag_bag.py` because it did not contain the string "airflow". 
   
   This [was documented](https://airflow.apache.org/docs/apache-airflow/2.8.3/core-concepts/dags.html) - so this can be considered WAD.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org