You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Lee-W (via GitHub)" <gi...@apache.org> on 2024/07/18 16:49:07 UTC

[PR] Extend dataset dependencies [airflow]

Lee-W opened a new pull request, #40868:
URL: https://github.com/apache/airflow/pull/40868

   <!--
    Licensed to the Apache Software Foundation (ASF) under one
    or more contributor license agreements.  See the NOTICE file
    distributed with this work for additional information
    regarding copyright ownership.  The ASF licenses this file
    to you under the Apache License, Version 2.0 (the
    "License"); you may not use this file except in compliance
    with the License.  You may obtain a copy of the License at
   
      http://www.apache.org/licenses/LICENSE-2.0
   
    Unless required by applicable law or agreed to in writing,
    software distributed under the License is distributed on an
    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    KIND, either express or implied.  See the License for the
    specific language governing permissions and limitations
    under the License.
    -->
   
   <!--
   Thank you for contributing! Please make sure that your code changes
   are covered with tests. And in case of new features or big changes
   remember to adjust the documentation.
   
   Feel free to ping committers for the review!
   
   In case of an existing issue, reference it using one of the following:
   
   closes: #ISSUE
   related: #ISSUE
   
   How to write a good git commit message:
   http://chris.beams.io/posts/git-commit/
   -->
   
   
   
   <!-- Please keep an empty line above the dashes. -->
   ---
   **^ Add meaningful description above**
   Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information.
   In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Extend dataset dependencies [airflow]

Posted by "bbovenzi (via GitHub)" <gi...@apache.org>.
bbovenzi commented on code in PR #40868:
URL: https://github.com/apache/airflow/pull/40868#discussion_r1688140724


##########
airflow/datasets/__init__.py:
##########
@@ -271,6 +306,20 @@ def iter_datasets(self) -> Iterator[tuple[str, Dataset]]:
                 yield k, v
                 seen.add(k)
 
+    def iter_dag_deps(self, *, source: str, target: str) -> Iterator[DagDependency]:
+        """
+        Iterate dataset, dataset aliases and their resolved datasets  as dag dependency.
+
+        :meta private:
+        """
+        dag_deps: set[DagDependency] = set()

Review Comment:
   The above is fine for now. But yes, I think we should show the conditions for triggering a DAG in the graph. That is separate from showing dataset aliases at all.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Extend dataset dependencies [airflow]

Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #40868:
URL: https://github.com/apache/airflow/pull/40868#discussion_r1687820653


##########
airflow/datasets/__init__.py:
##########
@@ -306,6 +350,9 @@ def __init__(self, name: str) -> None:
         self.name = name
         self.objects = expand_alias_to_datasets(name)
 
+    def __repr__(self) -> str:
+        return f"_DatasetAliasCondition({', '.join(map(str, self.objects))})"

Review Comment:
   This does not seem right? At least the class name certainly isn’t.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Extend dataset dependencies [airflow]

Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #40868:
URL: https://github.com/apache/airflow/pull/40868#discussion_r1687433284


##########
airflow/datasets/__init__.py:
##########
@@ -271,6 +306,20 @@ def iter_datasets(self) -> Iterator[tuple[str, Dataset]]:
                 yield k, v
                 seen.add(k)
 
+    def iter_dag_deps(self, *, source: str, target: str) -> Iterator[DagDependency]:
+        """
+        Iterate dataset, dataset aliases and their resolved datasets  as dag dependency.
+
+        :meta private:
+        """
+        dag_deps: set[DagDependency] = set()

Review Comment:
   Can you elaborate?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Extend dataset dependencies [airflow]

Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on code in PR #40868:
URL: https://github.com/apache/airflow/pull/40868#discussion_r1687498781


##########
airflow/datasets/__init__.py:
##########
@@ -271,6 +306,20 @@ def iter_datasets(self) -> Iterator[tuple[str, Dataset]]:
                 yield k, v
                 seen.add(k)
 
+    def iter_dag_deps(self, *, source: str, target: str) -> Iterator[DagDependency]:
+        """
+        Iterate dataset, dataset aliases and their resolved datasets  as dag dependency.
+
+        :meta private:
+        """
+        dag_deps: set[DagDependency] = set()

Review Comment:
   AFAIK, we do not have `DatasetAll` and `DatasetAny` in the graph. so `schedule=Dataset("1") & Dataset("1") | Dataset("1")` would be something similar to the above comment.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Extend dataset dependencies [airflow]

Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #40868:
URL: https://github.com/apache/airflow/pull/40868#discussion_r1687821599


##########
airflow/datasets/__init__.py:
##########
@@ -314,6 +361,38 @@ def as_expression(self) -> Any:
         """
         return {"alias": self.name}
 
+    def iter_dag_dependencies(self, *, source: str = "", target: str = "") -> Iterator[DagDependency]:
+        """
+        Iterate a dataset alias and its resolved datasets  as dag dependency.
+
+        :meta private:
+        """
+        if self.objects:
+            for obj in self.objects:
+                dataset = cast(Dataset, obj)

Review Comment:
   I don’t think this is right? We can have nesting conditions, in those cases `obj` would not be a Dataset.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Extend dataset dependencies [airflow]

Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on code in PR #40868:
URL: https://github.com/apache/airflow/pull/40868#discussion_r1688140014


##########
airflow/datasets/__init__.py:
##########
@@ -271,6 +306,20 @@ def iter_datasets(self) -> Iterator[tuple[str, Dataset]]:
                 yield k, v
                 seen.add(k)
 
+    def iter_dag_deps(self, *, source: str, target: str) -> Iterator[DagDependency]:
+        """
+        Iterate dataset, dataset aliases and their resolved datasets  as dag dependency.
+
+        :meta private:
+        """
+        dag_deps: set[DagDependency] = set()

Review Comment:
   Got your point! Yep, it's no longer in `iter_dag_dependencies`. But it still happens in `airflow/serialization/serialized_objects.py`. So I think it's still not ` strictly in the UI layer (i.e. somewhere in API or frontend),` But in the "current" (before this PR) version, we're using `iter_datasets` to [detect_dag_dependencies](https://github.com/apache/airflow/blob/9ec9eb79a0cc845d86e7380c73269d2ee1d3c210/airflow/serialization/serialized_objects.py#L963). ~~On the other hand, the [outlet part](https://github.com/apache/airflow/blob/9ec9eb79a0cc845d86e7380c73269d2ee1d3c210/airflow/serialization/serialized_objects.py#L946-L955) did not deduplicate.~~ I was wrong, [outlet](https://github.com/apache/airflow/blob/0eaa6117bf0310be798ad2be5c82905bd6fa30b5/airflow/serialization/serialized_objects.py#L1362) was also de-duplicated
   
   I think what you suggest is better. (let the API layer do the deduplication job, I'll probably do it [here](https://github.com/apache/airflow/blob/9ec9eb79a0cc845d86e7380c73269d2ee1d3c210/airflow/www/views.py#L3498)), but would like to check whether changing this behavior (which affect serialized DAG) is actually ok. Thanks!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Extend dataset dependencies [airflow]

Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on code in PR #40868:
URL: https://github.com/apache/airflow/pull/40868#discussion_r1687578288


##########
airflow/datasets/__init__.py:
##########
@@ -271,6 +306,20 @@ def iter_datasets(self) -> Iterator[tuple[str, Dataset]]:
                 yield k, v
                 seen.add(k)
 
+    def iter_dag_deps(self, *, source: str, target: str) -> Iterator[DagDependency]:
+        """
+        Iterate dataset, dataset aliases and their resolved datasets  as dag dependency.
+
+        :meta private:
+        """
+        dag_deps: set[DagDependency] = set()

Review Comment:
   > say ("a" | "b") & ("a" | "c"). This cannot be simplified, but we only want one line to a instead of two. 
   
   I guess I'm a bit confused. What do you expect the UI to look like in this situation? It would look something like the following now.
   
   ```mermaid
   flowchart LR
       a --> DAG
       b --> DAG
       c --> DAG
   ```
   
   If we don't duplicate (for cases that can't actually be dedup), I would probably want something like this. 
   
   ```mermaid
   flowchart LR
       a --> Or1
       b --> Or1
       a --> Or2
       c --> Or2
   
       Or1 --> And
       Or2 --> And
   
       And --> DAG
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Extend dataset dependencies [airflow]

Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on code in PR #40868:
URL: https://github.com/apache/airflow/pull/40868#discussion_r1688101731


##########
airflow/datasets/__init__.py:
##########
@@ -306,6 +350,9 @@ def __init__(self, name: str) -> None:
         self.name = name
         self.objects = expand_alias_to_datasets(name)
 
+    def __repr__(self) -> str:
+        return f"_DatasetAliasCondition({', '.join(map(str, self.objects))})"

Review Comment:
   This is actually expected. I'd like to use it to separate `DatasetAny` and `_DatasetAliasCondition`.



##########
airflow/datasets/__init__.py:
##########
@@ -314,6 +361,38 @@ def as_expression(self) -> Any:
         """
         return {"alias": self.name}
 
+    def iter_dag_dependencies(self, *, source: str = "", target: str = "") -> Iterator[DagDependency]:
+        """
+        Iterate a dataset alias and its resolved datasets  as dag dependency.
+
+        :meta private:
+        """
+        if self.objects:
+            for obj in self.objects:
+                dataset = cast(Dataset, obj)

Review Comment:
   No nested condition would happen in `_DatasetAliasCondition` (at least in the current design) . This condition will only expand a `DatasetAlias` to `DatasetAny( resolved datasets )`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Extend dataset dependencies [airflow]

Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on code in PR #40868:
URL: https://github.com/apache/airflow/pull/40868#discussion_r1687497644


##########
airflow/datasets/__init__.py:
##########
@@ -271,6 +306,20 @@ def iter_datasets(self) -> Iterator[tuple[str, Dataset]]:
                 yield k, v
                 seen.add(k)
 
+    def iter_dag_deps(self, *, source: str, target: str) -> Iterator[DagDependency]:
+        """
+        Iterate dataset, dataset aliases and their resolved datasets  as dag dependency.
+
+        :meta private:
+        """
+        dag_deps: set[DagDependency] = set()

Review Comment:
   Sure.
   
   currently, a DAG with `schedule=[Dataset("1"), Dataset("1")]` will generate the following graph.
   
   
   ```mermaid
   flowchart LR
       dataset1 --> DAG1
   ```
   
   By not deduplicate, my understanding is we want to have something like this. Then we'll have to mark that the frist `Dataset("1")` and the second `Dataset("1")` are different. It's not undoable. Just not sure whether it makes sense.
   
   ```mermaid
   flowchart LR
       dataset1 --> DAG1
       dataset1 --> DAG1
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Extend dataset dependencies [airflow]

Posted by "phanikumv (via GitHub)" <gi...@apache.org>.
phanikumv merged PR #40868:
URL: https://github.com/apache/airflow/pull/40868


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Extend dataset dependencies [airflow]

Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on code in PR #40868:
URL: https://github.com/apache/airflow/pull/40868#discussion_r1687552200


##########
airflow/datasets/__init__.py:
##########
@@ -271,6 +306,20 @@ def iter_datasets(self) -> Iterator[tuple[str, Dataset]]:
                 yield k, v
                 seen.add(k)
 
+    def iter_dag_deps(self, *, source: str, target: str) -> Iterator[DagDependency]:
+        """
+        Iterate dataset, dataset aliases and their resolved datasets  as dag dependency.
+
+        :meta private:
+        """
+        dag_deps: set[DagDependency] = set()

Review Comment:
   What I'm thinking now is whether there's a situation we really don't want deduplication. Or is this more like a performance issue?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Extend dataset dependencies [airflow]

Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on code in PR #40868:
URL: https://github.com/apache/airflow/pull/40868#discussion_r1687578288


##########
airflow/datasets/__init__.py:
##########
@@ -271,6 +306,20 @@ def iter_datasets(self) -> Iterator[tuple[str, Dataset]]:
                 yield k, v
                 seen.add(k)
 
+    def iter_dag_deps(self, *, source: str, target: str) -> Iterator[DagDependency]:
+        """
+        Iterate dataset, dataset aliases and their resolved datasets  as dag dependency.
+
+        :meta private:
+        """
+        dag_deps: set[DagDependency] = set()

Review Comment:
   > say ("a" | "b") & ("a" | "c"). This cannot be simplified, but we only want one line to a instead of two. 
   
   I guess I'm a bit confused. What do you expect the UI to look like in this situation? It would look something like that.
   
   ```mermaid
   flowchart LR
       a --> DAG
       b --> DAG
       c --> DAG
   ```
   
   If we don't duplicate (for cases that can't actually be dedup), I would probably want something like this. 
   
   ```mermaid
   flowchart LR
       a --> Or1
       b --> Or1
       a --> Or2
       c --> Or2
   
       Or1 --> And
       Or2 --> And
   
       And --> DAG
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Extend dataset dependencies [airflow]

Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on PR #40868:
URL: https://github.com/apache/airflow/pull/40868#issuecomment-2237068975

   <img width="771" alt="image" src="https://github.com/user-attachments/assets/6317a114-abaa-478d-bc4f-408b2adafd62">
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Extend dataset dependencies [airflow]

Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on PR #40868:
URL: https://github.com/apache/airflow/pull/40868#issuecomment-2244103645

   Thanks @bbovenzi . I think this one is mostly just for the webserver. I'm planning on merging it in half a day or so. if anyone wants to take a deeper look. Please let me know 🙂 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Extend dataset dependencies [airflow]

Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on code in PR #40868:
URL: https://github.com/apache/airflow/pull/40868#discussion_r1689473013


##########
airflow/datasets/__init__.py:
##########
@@ -271,6 +306,20 @@ def iter_datasets(self) -> Iterator[tuple[str, Dataset]]:
                 yield k, v
                 seen.add(k)
 
+    def iter_dag_deps(self, *, source: str, target: str) -> Iterator[DagDependency]:
+        """
+        Iterate dataset, dataset aliases and their resolved datasets  as dag dependency.
+
+        :meta private:
+        """
+        dag_deps: set[DagDependency] = set()

Review Comment:
   @uranusjr I've changed the serialization behavior and moved the deduplication to the API layer. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Extend dataset dependencies [airflow]

Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on code in PR #40868:
URL: https://github.com/apache/airflow/pull/40868#discussion_r1688140014


##########
airflow/datasets/__init__.py:
##########
@@ -271,6 +306,20 @@ def iter_datasets(self) -> Iterator[tuple[str, Dataset]]:
                 yield k, v
                 seen.add(k)
 
+    def iter_dag_deps(self, *, source: str, target: str) -> Iterator[DagDependency]:
+        """
+        Iterate dataset, dataset aliases and their resolved datasets  as dag dependency.
+
+        :meta private:
+        """
+        dag_deps: set[DagDependency] = set()

Review Comment:
   Got your point! Yep, it's no longer in `iter_dag_dependencies`. But it still happens in `airflow/serialization/serialized_objects.py`. So I think it's still not ` strictly in the UI layer (i.e. somewhere in API or frontend),` But in the "current" (before this PR) version, we're using `iter_datasets` to [detect_dag_dependencies](https://github.com/apache/airflow/blob/9ec9eb79a0cc845d86e7380c73269d2ee1d3c210/airflow/serialization/serialized_objects.py#L963). On the other hand, the [outlet part](https://github.com/apache/airflow/blob/9ec9eb79a0cc845d86e7380c73269d2ee1d3c210/airflow/serialization/serialized_objects.py#L946-L955) did not deduplicate.
   
   I think what you suggest is better. (let the API layer do the deduplication job, I'll probably do it [here](https://github.com/apache/airflow/blob/9ec9eb79a0cc845d86e7380c73269d2ee1d3c210/airflow/www/views.py#L3498)), but would like to check whether changing this behavior (which affect serialized DAG) is actually ok. Thanks!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Extend dataset dependencies [airflow]

Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #40868:
URL: https://github.com/apache/airflow/pull/40868#discussion_r1687819158


##########
airflow/datasets/__init__.py:
##########
@@ -271,6 +306,20 @@ def iter_datasets(self) -> Iterator[tuple[str, Dataset]]:
                 yield k, v
                 seen.add(k)
 
+    def iter_dag_deps(self, *, source: str, target: str) -> Iterator[DagDependency]:
+        """
+        Iterate dataset, dataset aliases and their resolved datasets  as dag dependency.
+
+        :meta private:
+        """
+        dag_deps: set[DagDependency] = set()

Review Comment:
   By “now” above you mean before this PR right? That’s what I would expect the graph to look like. So we still need some dedup logic when rendering the UI, regardless what we do in all/any. But I want those to be strictly in the UI layer (i.e. somewhere in API or frontend), not `iter_dag_dependencies`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Extend dataset dependencies [airflow]

Posted by "bbovenzi (via GitHub)" <gi...@apache.org>.
bbovenzi commented on PR #40868:
URL: https://github.com/apache/airflow/pull/40868#issuecomment-2246797443

   Looks to be working now!
   
   <img width="504" alt="Screenshot 2024-07-23 at 11 34 22 PM" src="https://github.com/user-attachments/assets/68c58e61-d8b7-4708-bbd2-5da0d20ebe65">
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Extend dataset dependencies [airflow]

Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #40868:
URL: https://github.com/apache/airflow/pull/40868#discussion_r1687531869


##########
airflow/serialization/dag_dependency.py:
##########


Review Comment:
   Fair enough, it’s an internal class so most places are fine as long as contributors can find it and it doesn’t introduce import cycle issues.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Extend dataset dependencies [airflow]

Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #40868:
URL: https://github.com/apache/airflow/pull/40868#discussion_r1687398867


##########
airflow/datasets/__init__.py:
##########
@@ -271,6 +306,20 @@ def iter_datasets(self) -> Iterator[tuple[str, Dataset]]:
                 yield k, v
                 seen.add(k)
 
+    def iter_dag_deps(self, *, source: str, target: str) -> Iterator[DagDependency]:
+        """
+        Iterate dataset, dataset aliases and their resolved datasets  as dag dependency.
+
+        :meta private:
+        """
+        dag_deps: set[DagDependency] = set()

Review Comment:
   Is the order of DAG dependencies significant? If not, I think we can deduplicate at where this function is called instead. If we have aliases nested in conditions (or just nested conditions), each nesting would have a set doing deduplication, which isn’t efficient IMO.
   
   Or, do we need to deduplicate dependencies at all…?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Extend dataset dependencies [airflow]

Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on code in PR #40868:
URL: https://github.com/apache/airflow/pull/40868#discussion_r1687400902


##########
airflow/datasets/__init__.py:
##########
@@ -185,6 +186,14 @@ def evaluate(self, statuses: dict[str, bool]) -> bool:
     def iter_datasets(self) -> Iterator[tuple[str, Dataset]]:
         raise NotImplementedError
 
+    def iter_dag_deps(self, *, source: str, target: str) -> Iterator[DagDependency]:

Review Comment:
   Nope, `iter_dag_dependencies` is good. Will update it



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Extend dataset dependencies [airflow]

Posted by "bbovenzi (via GitHub)" <gi...@apache.org>.
bbovenzi commented on PR #40868:
URL: https://github.com/apache/airflow/pull/40868#issuecomment-2246177199

   Actually, I am using this branch locally and I noticed that we only have dag<->dataset-alias relationships, but we don't have any dataset<->dataset-alias edges. Is that not possible?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Extend dataset dependencies [airflow]

Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on PR #40868:
URL: https://github.com/apache/airflow/pull/40868#issuecomment-2246668869

   > Actually, I am using this branch locally and I noticed that we only have dag<->dataset-alias relationships, but we don't have any dataset<->dataset-alias edges. Is that not possible?
   
   That's possible. We just need to confirm what the dag dependencies we actually want. What I'm doing now is something like
   
   ```mermaid
   flowchart LR
       dataset_1 --> dataset_alias_1
       dataset_2 --> dataset_alias_1
       dataset_3 --> dataset_alias_1
   
       dataset_alias_1 --> DAG_1
   
       DAG_1 --> dataset_alias_2
   
       dataset_alias_2 --> dataset_4
       dataset_alias_2 --> dataset_5
   ```
   
   This means `DAG_1` set `schedule=DatatsetAlias("datset_alias_1")` and `outlets=[DatasetAlias("dataset_alias_2")]`. `datset_alias_1` can be resolved in to dataset 1 ~ 3 while `dataset_alias_2` can be resolved into dataset 4 ~ 5. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Extend dataset dependencies [airflow]

Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on code in PR #40868:
URL: https://github.com/apache/airflow/pull/40868#discussion_r1687404008


##########
airflow/datasets/__init__.py:
##########
@@ -271,6 +306,20 @@ def iter_datasets(self) -> Iterator[tuple[str, Dataset]]:
                 yield k, v
                 seen.add(k)
 
+    def iter_dag_deps(self, *, source: str, target: str) -> Iterator[DagDependency]:
+        """
+        Iterate dataset, dataset aliases and their resolved datasets  as dag dependency.
+
+        :meta private:
+        """
+        dag_deps: set[DagDependency] = set()

Review Comment:
   oh, in the current dateset_dependecnies endpoint, I think it'll be filtered out anyway. so we can also defer the deduplication later as well



##########
airflow/serialization/dag_dependency.py:
##########


Review Comment:
   I feel it is still something mainly for `serialisation`. `iter_dag_deps` is mainly just for `serialisation` and later be used in the view.py. 
   
   I do not object to moving it to `utils`. I'm just not sure moving lots of things there is a good idea 🤔 



##########
airflow/datasets/__init__.py:
##########
@@ -271,6 +306,20 @@ def iter_datasets(self) -> Iterator[tuple[str, Dataset]]:
                 yield k, v
                 seen.add(k)
 
+    def iter_dag_deps(self, *, source: str, target: str) -> Iterator[DagDependency]:
+        """
+        Iterate dataset, dataset aliases and their resolved datasets  as dag dependency.
+
+        :meta private:
+        """
+        dag_deps: set[DagDependency] = set()

Review Comment:
   > Is the order of DAG dependencies significant? If not, I think we can deduplicate at where this function is called instead. 
   
   I think not. but I'll test out.
   
   > Or, do we need to deduplicate dependencies at all…?
   
   I would probably a +0? e.g., `scheudle=[Datset("test-1"), Dataset("test-1")]` should we generate 2 `Dataset("test-1")`? On the other hand, it's also what users want 🤔  I feel it's more like a UI thing and would like to know how @bbovenzi  think



##########
airflow/datasets/__init__.py:
##########
@@ -271,6 +306,20 @@ def iter_datasets(self) -> Iterator[tuple[str, Dataset]]:
                 yield k, v
                 seen.add(k)
 
+    def iter_dag_deps(self, *, source: str, target: str) -> Iterator[DagDependency]:
+        """
+        Iterate dataset, dataset aliases and their resolved datasets  as dag dependency.
+
+        :meta private:
+        """
+        dag_deps: set[DagDependency] = set()

Review Comment:
   I feel that not deduplicating might make it even more complicated.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Extend dataset dependencies [airflow]

Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #40868:
URL: https://github.com/apache/airflow/pull/40868#discussion_r1687396344


##########
airflow/datasets/__init__.py:
##########
@@ -185,6 +186,14 @@ def evaluate(self, statuses: dict[str, bool]) -> bool:
     def iter_datasets(self) -> Iterator[tuple[str, Dataset]]:
         raise NotImplementedError
 
+    def iter_dag_deps(self, *, source: str, target: str) -> Iterator[DagDependency]:

Review Comment:
   Any concerns just naming this `iter_dag_dependencies` to match the return value type name?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Extend dataset dependencies [airflow]

Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #40868:
URL: https://github.com/apache/airflow/pull/40868#discussion_r1687399369


##########
airflow/serialization/dag_dependency.py:
##########


Review Comment:
   Now that this is used for things other than serialisation, I wonder if the class should be moved outside of `serialization` entirely, maybe into `utils` or somewhere else.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Extend dataset dependencies [airflow]

Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #40868:
URL: https://github.com/apache/airflow/pull/40868#discussion_r1687564898


##########
airflow/datasets/__init__.py:
##########
@@ -271,6 +306,20 @@ def iter_datasets(self) -> Iterator[tuple[str, Dataset]]:
                 yield k, v
                 seen.add(k)
 
+    def iter_dag_deps(self, *, source: str, target: str) -> Iterator[DagDependency]:
+        """
+        Iterate dataset, dataset aliases and their resolved datasets  as dag dependency.
+
+        :meta private:
+        """
+        dag_deps: set[DagDependency] = set()

Review Comment:
   For DatasetAll and DatasetAny specifically, I don’t think there’s a situation we need to keep the original user input. I’d even argue it’s an oversight I did not implement dedup in `__and__` and `__or__` in the initial release.
   
   There are cases where we can’t dedup in all/any but need further cleanup in the UI though, say `("a" | "b") & ("a" | "c")`. This cannot be simplified, but we only want one line to `a` instead of two. But that’s a UI consideration that should only be done either in the API layer or frontend (React).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Extend dataset dependencies [airflow]

Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on code in PR #40868:
URL: https://github.com/apache/airflow/pull/40868#discussion_r1689054394


##########
airflow/datasets/__init__.py:
##########
@@ -271,6 +306,20 @@ def iter_datasets(self) -> Iterator[tuple[str, Dataset]]:
                 yield k, v
                 seen.add(k)
 
+    def iter_dag_deps(self, *, source: str, target: str) -> Iterator[DagDependency]:
+        """
+        Iterate dataset, dataset aliases and their resolved datasets  as dag dependency.
+
+        :meta private:
+        """
+        dag_deps: set[DagDependency] = set()

Review Comment:
   @uranusjr I just created https://github.com/apache/airflow/pull/40984 to verify that deduplication is happening in the main branch and would like to confirm it's ok to change that behavior in this branch.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Extend dataset dependencies [airflow]

Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on code in PR #40868:
URL: https://github.com/apache/airflow/pull/40868#discussion_r1687623687


##########
airflow/datasets/__init__.py:
##########
@@ -271,6 +306,20 @@ def iter_datasets(self) -> Iterator[tuple[str, Dataset]]:
                 yield k, v
                 seen.add(k)
 
+    def iter_dag_deps(self, *, source: str, target: str) -> Iterator[DagDependency]:
+        """
+        Iterate dataset, dataset aliases and their resolved datasets  as dag dependency.
+
+        :meta private:
+        """
+        dag_deps: set[DagDependency] = set()

Review Comment:
   *not related to the whether to deduplicate discussion*
   
   I've moved the deduplication out of `iter_dag_dependencies`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Extend dataset dependencies [airflow]

Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #40868:
URL: https://github.com/apache/airflow/pull/40868#discussion_r1687530984


##########
airflow/datasets/__init__.py:
##########
@@ -271,6 +306,20 @@ def iter_datasets(self) -> Iterator[tuple[str, Dataset]]:
                 yield k, v
                 seen.add(k)
 
+    def iter_dag_deps(self, *, source: str, target: str) -> Iterator[DagDependency]:
+        """
+        Iterate dataset, dataset aliases and their resolved datasets  as dag dependency.
+
+        :meta private:
+        """
+        dag_deps: set[DagDependency] = set()

Review Comment:
   In this particular situation, I wonder if we should just deduplicate in DatasetAll and DatasetAny. I anticipate there might be other places we should dedup, but at least for those there’s a simple math solution.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org