[jira] [Comment Edited] (AIRFLOW-6011) Airflow treats local dependencies as DAG files if they contain the string "DAG"

["Aaron Liblong (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AIRFLOW-6011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16976872#comment-16976872 ] 

Aaron Liblong edited comment on AIRFLOW-6011 at 11/18/19 8:49 PM:
------------------------------------------------------------------

That documentation on packaged DAGs says the following:
{quote} 
{code:java}
my_dag1.py
my_dag2.py
package1/__init__.py
package1/functions.py
{code}
 
{quote}
{quote}Airflow will scan the zip file and try to load my_dag1.py and my_dag2.py. It will not go into subdirectories as these are considered to be potential packages.
{quote}
My issue is that Airflow _did_ look in the subdirectory, and specifically for files containing the character sequence "DAG" (regardless of whether it was in the context of using the actual {{DAG}} class, or just in a comment).


was (Author: liblonga):
That documentation on packaged DAGs says the following:
{quote} 
{code:java}
my_dag1.py
my_dag2.py
package1/__init__.py
package1/functions.py
{code}
 
{quote}
{quote}Airflow will scan the zip file and try to load my_dag1.py and my_dag2.py. It will not go into subdirectories as these are considered to be potential packages.
{quote}
My issue is that Airflow _did_ look in the subdirectory, and specifically for files containing the character sequence "DAG".

> Airflow treats local dependencies as DAG files if they contain the string "DAG"
> -------------------------------------------------------------------------------
>
>                 Key: AIRFLOW-6011
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-6011
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: DAG, dependencies
>    Affects Versions: 1.10.2
>            Reporter: Aaron Liblong
>            Priority: Major
>
> Our Airflow instance uses what I believe is a common pattern of having a directory for DAG dependencies within the main {{dags}} directory itself. E.g.:
> {code:java}
> dags
> | - dag1.py
> | - dag2.py
> | - dependencies
> | | - dependency1.py
> | | - dependency2.py
> {code}
>  
> Then {{dag1.py}} contains something like {{import dependency1}}.
> The dependencies are normally not treated like DAG files themselves, as expected. However, when they contain the string "DAG", even in a comment, they _will_ be treated as DAG files. I'm guessing this is the result of Airflow doing a recursive grep for that exact (case-sensitive) string. This is not intuitive behaviour for users, and is hard to debug, since users will reasonably expect non-directive comments not to have an effect on how their code will being treated.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)