[GitHub] Fokko commented on issue #4396: [WIP][AIRFLOW-3585] - Add edges to database

[GitBox <gi...@apache.org>]

Fokko commented on issue #4396: [WIP][AIRFLOW-3585] - Add edges to database
URL: https://github.com/apache/airflow/pull/4396#issuecomment-456720871
 
 
   Thanks @ashb for providing feedback. I think it would be a good idea to store this per DagRun. Versioned versions of the graph would introduce another layer of complexity, and this is an attempt to reduce the complexity of the logic to bring everything to one single place (the database).
   Having good indices on the tables would be okay in my opinion. I think the number of DagRuns would typically be in the order of thousands, maybe up to a million, the number of Edges would be a couple of million at most then.
   
   Also parsing huge DAGs involves a lot of complexity and processing power, so the idea is to do this only once, and then store it in the database using the right keys.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services