[GitHub] [airflow] marengaz commented on a change in pull request #5350: [AIRFLOW-4716] Instrument dag loading time duration

[GitBox <gi...@apache.org>]

marengaz commented on a change in pull request #5350: [AIRFLOW-4716] Instrument dag loading time duration
URL: https://github.com/apache/airflow/pull/5350#discussion_r315774306
 
 

 ##########
 File path: airflow/models/dagbag.py
 ##########
 @@ -391,6 +396,14 @@ def collect_dags(
             'dagbag_import_errors', len(self.import_errors), 1)
         self.dagbag_stats = sorted(
             stats, key=lambda x: x.duration, reverse=True)
+        for file_stat in self.dagbag_stats:
+            dag_ids = dags_by_name[file_stat.dags]
+            if file_stat.dag_num >= 1:
+                # if we found multiple dags per file, the stat is 'dag_id1 _ dag_id2'
+                dag_names = '_'.join(dag_ids)
 
 Review comment:
   hey @XD-DENG / @feng-tao / @milton0825 
   about the `dag_names = '_'.join(dag_ids)`; we have some files generating 100s of dags, which makes this metric quite unusable.
   - could you advise how does statsd handles such a long metric name (we are actually using a [statsd to prometheus exporter](https://github.com/prometheus/statsd_exporter) so im not that familiar with statsd)?
   - is it not possible to get the name of the file where the dags are generated instead?
   - the description in docs/metrics.rst doesn't quite match the current implementation

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services