[jira] [Commented] (AIRFLOW-774) dagbag_size/collect_dags/dagbag_import_errors stats incorrect

["Ace Haidrey (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AIRFLOW-774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16111775#comment-16111775 ] 

Ace Haidrey commented on AIRFLOW-774:
-------------------------------------

Does anyone have an idea of how involved a fix to this would be, or the way to go about it. Currently our team is running to the same issues of having errors regarding stats collection. Counts are off, and more.

> dagbag_size/collect_dags/dagbag_import_errors stats incorrect
> -------------------------------------------------------------
>
>                 Key: AIRFLOW-774
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-774
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: logging
>            Reporter: Dan Davydov
>
> After the multiprocessor change was made (dag folders are processed in parallel), the number of dags reported by airflow is for each of these subprocesses which is inaccurate, and potentially orders of magnitude less than the actual number of dags. These individual processes stats should be aggregated. The collect_dags/dagbag_import_errors stats should also be fixed (time it takes to parse the dags).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)