You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "chen xianxin (Jira)" <ji...@apache.org> on 2019/10/14 03:42:00 UTC
[jira] [Closed] (AIRFLOW-5639) DagFileProcessor: parse dag files
every time, which consume lots of resources and is unnecessary
[ https://issues.apache.org/jira/browse/AIRFLOW-5639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
chen xianxin closed AIRFLOW-5639.
---------------------------------
Resolution: Not A Problem
Importing dag files provide task-dependencies and other information that DB has not, so importing dag files is necessary every time, unless dagbag will be saved in some place such as DB, memory.
> DagFileProcessor: parse dag files every time, which consume lots of resources and is unnecessary
> ------------------------------------------------------------------------------------------------
>
> Key: AIRFLOW-5639
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5639
> Project: Apache Airflow
> Issue Type: Improvement
> Components: scheduler
> Affects Versions: 1.10.5
> Reporter: chen xianxin
> Priority: Major
>
>
> h2. Code
> [https://github.com/apache/airflow/blob/v1-10-stable/airflow/models/dagbag.py#L166-L170]
> h2. Problem description
> *self.file_last_changed* doesn't work, beacause it's set to empty when Dagbag is initialized. so even no changes were made to the file, it will also be imported. I have sure this problem by print log.
> Generate bagdags from files take about 50% time of dag files processing, if only generate bagdags when files are changed, lots of resources will be saved.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)