You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Cedric Hourcade (JIRA)" <ji...@apache.org> on 2017/10/18 08:35:00 UTC

[jira] [Created] (AIRFLOW-1729) Ignore whole directories in .airflowignore

Cedric Hourcade created AIRFLOW-1729:
----------------------------------------

             Summary: Ignore whole directories in .airflowignore
                 Key: AIRFLOW-1729
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1729
             Project: Apache Airflow
          Issue Type: Improvement
          Components: core
    Affects Versions: Airflow 2.0
            Reporter: Cedric Hourcade
            Priority: Minor


The .airflowignore file allows to prevent scanning files for DAG. But even if we blacklist fulldirectory the {{os.walk}} will still go through them no matter how deep they are and skip files one by one, which can be an issue when you keep around big .git or virtualvenv directories.

I suggest to add something like:
{code}
dirs[:] = [d for d in dirs if not any([re.findall(p, os.path.join(root, d)) for p in patterns])]
{code}
to prune the directories here: https://github.com/apache/incubator-airflow/blob/cfc2f73c445074e1e09d6ef6a056cd2b33a945da/airflow/utils/dag_processing.py#L208-L209 and in {{list_py_file_paths}}




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)