[GitHub] [airflow] ms32035 commented on a change in pull request #13199: Create dag dependencies view

[GitBox <gi...@apache.org>]

ms32035 commented on a change in pull request #13199:
URL: https://github.com/apache/airflow/pull/13199#discussion_r546445867



##########
File path: airflow/www/views.py
##########
@@ -3772,3 +3772,102 @@ def autocomplete(self, session=None):
         payload = [row[0] for row in dag_ids_query.union(owners_query).limit(10).all()]
 
         return wwwutils.json_response(payload)
+
+
+class DagDependenciesView(AirflowBaseView):
+    """View to show dependencies between DAGs"""
+
+    refresh_interval = conf.getint(
+        "dag_dependencies_plugin",
+        "refresh_interval",
+        fallback=conf.getint("scheduler", "dag_dir_list_interval"),
+    )
+    last_refresh = datetime.utcnow() - timedelta(seconds=refresh_interval)
+    nodes = []
+    edges = []
+
+    @expose('/dag-dependencies')
+    @auth.has_access(
+        [
+            (permissions.ACTION_CAN_READ, permissions.RESOURCE_DAG),
+        ]
+    )
+    @gzipped
+    @action_logging
+    def list(self):
+        """Display DAG dependencies"""
+        title = "DAG Dependencies"
+
+        if datetime.utcnow() > self.last_refresh + timedelta(seconds=self.refresh_interval):
+            self._calculate_graph()
+            self.last_refresh = datetime.utcnow()
+
+        return self.render_template(
+            "airflow/dag_dependencies.html",
+            title=title,
+            nodes=self.nodes,
+            edges=self.edges,
+            last_refresh=self.last_refresh.strftime("%Y-%m-%d %H:%M:%S"),
+            arrange=conf.get("webserver", "dag_orientation"),
+            width=request.args.get("width", "100%"),
+            height=request.args.get("height", "800"),
+        )
+
+    def _calculate_graph(self):
+
+        current_app.dag_bag.collect_dags_from_db()

Review comment:
       Not quite sure how I could optimize it? It's already optimized in that it only refreshes after an interval which is equal to the scheduler recalculation interval. All dags are required to calculate the dependencies. Few ideas I have, which are not optimization but might help:
   
   - set last_refresh_time at the start of the refresh, so that a second refresh triggered before first one finishes does not cause a recalculation in parallel
   - remove dags without dependencies from rendering
   - add a global configuration option to disable the view. I guess some deployments may be too large for such a global view.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org