You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/07/20 06:16:57 UTC
[GitHub] [airflow] dstandish opened a new pull request, #25175: Add datasets to dag dependencies view
dstandish opened a new pull request, #25175:
URL: https://github.com/apache/airflow/pull/25175
<img width="628" alt="image" src="https://user-images.githubusercontent.com/15932138/179910251-b06f2ef4-eca2-4479-8c89-0b7da4975646.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on a diff in pull request #25175: Add datasets to dag dependencies view
Posted by GitBox <gi...@apache.org>.
dstandish commented on code in PR #25175:
URL: https://github.com/apache/airflow/pull/25175#discussion_r925771460
##########
airflow/serialization/serialized_objects.py:
##########
@@ -839,7 +882,21 @@ def deserialize_operator(cls, encoded_op: Dict[str, Any]) -> Operator:
@classmethod
def detect_dependencies(cls, op: Operator) -> Optional['DagDependency']:
"""Detects between DAG dependencies for the operator."""
- return cls.dependency_detector.detect_task_dependencies(op)
+ dependency_detector = DependencyDetector()
+ custom_dependency_detector = conf.getimport('scheduler', 'dependency_detector')
+ deps = set()
+ if not (custom_dependency_detector is None or type(dependency_detector) is DependencyDetector):
+ warnings.warn(
+ "Use of a custom dependency detector is deprecated. "
+ "Support will be removed in a future release.",
+ DeprecationWarning,
+ )
+ dep = custom_dependency_detector.detect_task_dependencies(op)
+ if dep:
+ deps.add(dep)
+ deps.update(dependency_detector.detect_task_dependencies(op))
Review Comment:
_really_, maybe we don't even need to maintain backcompat because it's just UI, and there's no promise that the UI doesn't change -- e.g. when we removed tree view and replaced with grid. so since in theory we can freely remove this view and replace it with something else, it seems it would follow that we can also remove the pluggability?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on a diff in pull request #25175: Add datasets to dag dependencies view
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on code in PR #25175:
URL: https://github.com/apache/airflow/pull/25175#discussion_r925695366
##########
airflow/models/dagbag.py:
##########
@@ -623,11 +623,12 @@ def _serialize_dag_capturing_errors(dag, session):
)
self.log.debug("Calling the DAG.bulk_sync_to_db method")
try:
+ DAG.bulk_write_to_db(self.dags.values(), session=session)
Review Comment:
Why are we flipping this order? I feel like I remember it was intentionally this way, but I can't remember why.
##########
airflow/serialization/serialized_objects.py:
##########
@@ -839,7 +882,21 @@ def deserialize_operator(cls, encoded_op: Dict[str, Any]) -> Operator:
@classmethod
def detect_dependencies(cls, op: Operator) -> Optional['DagDependency']:
"""Detects between DAG dependencies for the operator."""
- return cls.dependency_detector.detect_task_dependencies(op)
+ dependency_detector = DependencyDetector()
+ custom_dependency_detector = conf.getimport('scheduler', 'dependency_detector')
+ deps = set()
+ if not (custom_dependency_detector is None or type(dependency_detector) is DependencyDetector):
+ warnings.warn(
+ "Use of a custom dependency detector is deprecated. "
+ "Support will be removed in a future release.",
+ DeprecationWarning,
+ )
+ dep = custom_dependency_detector.detect_task_dependencies(op)
+ if dep:
+ deps.add(dep)
+ deps.update(dependency_detector.detect_task_dependencies(op))
Review Comment:
Just checking, we should be calling both the custom and default one?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] bbovenzi commented on a diff in pull request #25175: Add datasets to dag dependencies view
Posted by GitBox <gi...@apache.org>.
bbovenzi commented on code in PR #25175:
URL: https://github.com/apache/airflow/pull/25175#discussion_r926820799
##########
airflow/www/views.py:
##########
@@ -5330,6 +5330,14 @@ def _calculate_graph(self):
@staticmethod
def _node_dict(node_id, label, node_class):
+ def trunc_uri(val):
+ if len(val) > 25:
+ return val[0:23] + '...'
Review Comment:
As per another conversation, let's just show the whole URI for now so a user has the whole context and worry about shortening it later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish merged pull request #25175: Add datasets to dag dependencies view
Posted by GitBox <gi...@apache.org>.
dstandish merged PR #25175:
URL: https://github.com/apache/airflow/pull/25175
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on a diff in pull request #25175: Add datasets to dag dependencies view
Posted by GitBox <gi...@apache.org>.
dstandish commented on code in PR #25175:
URL: https://github.com/apache/airflow/pull/25175#discussion_r925765811
##########
airflow/models/dagbag.py:
##########
@@ -623,11 +623,12 @@ def _serialize_dag_capturing_errors(dag, session):
)
self.log.debug("Calling the DAG.bulk_sync_to_db method")
try:
+ DAG.bulk_write_to_db(self.dags.values(), session=session)
Review Comment:
if the order needs to be preserved, then we'll have to come up with something else. we could abbreviate the URI i suppose? then we would not need the id... wdyt?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] bbovenzi commented on a diff in pull request #25175: Add datasets to dag dependencies view
Posted by GitBox <gi...@apache.org>.
bbovenzi commented on code in PR #25175:
URL: https://github.com/apache/airflow/pull/25175#discussion_r926415884
##########
airflow/www/views.py:
##########
@@ -5330,6 +5330,14 @@ def _calculate_graph(self):
@staticmethod
def _node_dict(node_id, label, node_class):
+ def trunc_uri(val):
+ if len(val) > 25:
+ return val[0:23] + '...'
Review Comment:
Is there somewhere that we pass the full URI so we can show it in a tooltip?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on a diff in pull request #25175: Add datasets to dag dependencies view
Posted by GitBox <gi...@apache.org>.
dstandish commented on code in PR #25175:
URL: https://github.com/apache/airflow/pull/25175#discussion_r925763651
##########
airflow/models/dagbag.py:
##########
@@ -623,11 +623,12 @@ def _serialize_dag_capturing_errors(dag, session):
)
self.log.debug("Calling the DAG.bulk_sync_to_db method")
try:
+ DAG.bulk_write_to_db(self.dags.values(), session=session)
Review Comment:
yeah, the reason is, the datasets are written to DB in bulk_write_to_db. we need the datasets to be in the DB so we can look up the ID when serializing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on a diff in pull request #25175: Add datasets to dag dependencies view
Posted by GitBox <gi...@apache.org>.
dstandish commented on code in PR #25175:
URL: https://github.com/apache/airflow/pull/25175#discussion_r925768619
##########
airflow/serialization/serialized_objects.py:
##########
@@ -839,7 +882,21 @@ def deserialize_operator(cls, encoded_op: Dict[str, Any]) -> Operator:
@classmethod
def detect_dependencies(cls, op: Operator) -> Optional['DagDependency']:
"""Detects between DAG dependencies for the operator."""
- return cls.dependency_detector.detect_task_dependencies(op)
+ dependency_detector = DependencyDetector()
+ custom_dependency_detector = conf.getimport('scheduler', 'dependency_detector')
+ deps = set()
+ if not (custom_dependency_detector is None or type(dependency_detector) is DependencyDetector):
+ warnings.warn(
+ "Use of a custom dependency detector is deprecated. "
+ "Support will be removed in a future release.",
+ DeprecationWarning,
+ )
+ dep = custom_dependency_detector.detect_task_dependencies(op)
+ if dep:
+ deps.add(dep)
+ deps.update(dependency_detector.detect_task_dependencies(op))
Review Comment:
right, we're storing the deps in a set in order to dedupe. custom "detector" can only return a single value, we just add it to what we get from the current detector.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on a diff in pull request #25175: Add datasets to dag dependencies view
Posted by GitBox <gi...@apache.org>.
dstandish commented on code in PR #25175:
URL: https://github.com/apache/airflow/pull/25175#discussion_r926809274
##########
airflow/www/views.py:
##########
@@ -5330,6 +5330,14 @@ def _calculate_graph(self):
@staticmethod
def _node_dict(node_id, label, node_class):
+ def trunc_uri(val):
+ if len(val) > 25:
+ return val[0:23] + '...'
Review Comment:
Yes, it is the node I'd, but I'm not sure if that makes it into dom. I tried looking but didn't see?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on a diff in pull request #25175: Add datasets to dag dependencies view
Posted by GitBox <gi...@apache.org>.
dstandish commented on code in PR #25175:
URL: https://github.com/apache/airflow/pull/25175#discussion_r926843361
##########
airflow/www/views.py:
##########
@@ -5330,6 +5330,14 @@ def _calculate_graph(self):
@staticmethod
def _node_dict(node_id, label, node_class):
+ def trunc_uri(val):
+ if len(val) > 25:
+ return val[0:23] + '...'
Review Comment:
ok, sure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on a diff in pull request #25175: Add datasets to dag dependencies view
Posted by GitBox <gi...@apache.org>.
dstandish commented on code in PR #25175:
URL: https://github.com/apache/airflow/pull/25175#discussion_r926861103
##########
airflow/www/views.py:
##########
@@ -5330,6 +5330,14 @@ def _calculate_graph(self):
@staticmethod
def _node_dict(node_id, label, node_class):
+ def trunc_uri(val):
+ if len(val) > 25:
+ return val[0:23] + '...'
Review Comment:
yeah it seems we don't have tooltip code in the dag dependencies js
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org