You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2018/12/18 06:43:02 UTC
[GitHub] Fokko commented on a change in pull request #4328: AIRFLOW-3530 -
No re-parsing of dag files
Fokko commented on a change in pull request #4328: AIRFLOW-3530 - No re-parsing of dag files
URL: https://github.com/apache/incubator-airflow/pull/4328#discussion_r242421319
##########
File path: airflow/utils/dag_processing.py
##########
@@ -44,101 +45,13 @@
# To avoid circular imports
import airflow.models
from airflow import configuration as conf
-from airflow.dag.base_dag import BaseDag, BaseDagBag
-from airflow.exceptions import AirflowException
from airflow.settings import logging_class_path
from airflow.utils import timezone
from airflow.utils.db import provide_session
from airflow.utils.log.logging_mixin import LoggingMixin
from airflow.utils.state import State
-class SimpleDag(BaseDag):
Review comment:
The SimpleDag is almost an exact copy of the dag, with the addition that it is serializable. I had a discussion with Peter, and I fully agree with him that the additional complexity of having this class hierarchy (`SimpleDag`, `BaseDag`, `Dag`) isn't worth the upsides that it brings.
Firstly, my question would be, can we make the `Dag` in such a way that we can serialize this one as well. Secondly, why do we serialize in general? Afaik it was because of performance reasons (the webserver shows the dag based on the pickled instance), but I'm in doubt if we don't parse all the dags all the time if we still need this mechanism.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services