You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "mpeteuil (via GitHub)" <gi...@apache.org> on 2024/02/15 22:55:20 UTC
[PR] Make Datasets hashable [airflow]
mpeteuil opened a new pull request, #37465:
URL: https://github.com/apache/airflow/pull/37465
Currently DAGs accept a [`Collection["Dataset"]`](https://github.com/apache/airflow/blob/0c02ead4d8a527cbf0a916b6344f255c520e637f/airflow/models/dag.py#L171) as an option for the `schedule`, but that collection cannot be a `set` because Datasets are not a hashable type. The interesting thing is that [the `DatasetModel` is actually already hashable](https://github.com/apache/airflow/blob/dec78ab3f140f35e507de825327652ec24d03522/airflow/models/dataset.py#L93-L100), so this introduces a bit of duplication since it's mostly the same implementation. However, Airflow users are primarily interfacing with `Dataset`, not `DatasetModel` so I think it makes sense for `Dataset` to be hashable. I'm not sure how to square the duplication or what `__eq__` and `__hash__` provide for `DatasetModel` though.
There was discussion [on the original PR that created the `DatasetModel`](https://github.com/apache/airflow/pull/24613) about whether to create two classes or one. In that discussion @kaxil mentioned:
> I would slightly favour a separate `DatasetModel` and `Dataset` so `Dataset` becomes an extensible class, and `DatasetModel` just stores the info about the class. So users don't need to care about SQLAlchmey stuff when extending it.
That provides a bit of background on why they both exist for anyone reading this who is curious.
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
<!-- Please keep an empty line above the dashes. -->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make Datasets hashable [airflow]
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on code in PR #37465:
URL: https://github.com/apache/airflow/pull/37465#discussion_r1497817501
##########
airflow/datasets/__init__.py:
##########
@@ -46,3 +46,12 @@ def _check_uri(self, attr, uri: str):
def __fspath__(self):
return self.uri
+
+ def __eq__(self, other):
+ if isinstance(other, self.__class__):
+ return self.uri == other.uri
+ else:
+ return NotImplemented
+
+ def __hash__(self):
+ return hash(self.uri)
Review Comment:
Nice!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make Datasets hashable [airflow]
Posted by "mpeteuil (via GitHub)" <gi...@apache.org>.
mpeteuil commented on code in PR #37465:
URL: https://github.com/apache/airflow/pull/37465#discussion_r1497609428
##########
airflow/datasets/__init__.py:
##########
@@ -46,3 +46,12 @@ def _check_uri(self, attr, uri: str):
def __fspath__(self):
return self.uri
+
+ def __eq__(self, other):
+ if isinstance(other, self.__class__):
+ return self.uri == other.uri
+ else:
+ return NotImplemented
+
+ def __hash__(self):
+ return hash(self.uri)
Review Comment:
Thanks @Taragolis, I thought about that too. I think in this case because a `Dataset` URI uniquely identifies a `Dataset` using `extra` doesn't need to be a consideration. [There's even a unique index on the `DatasetModel`](https://github.com/apache/airflow/blob/dec78ab3f140f35e507de825327652ec24d03522/airflow/models/dataset.py#L74-L77) enforcing URI uniqueness.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make Datasets hashable [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #37465:
URL: https://github.com/apache/airflow/pull/37465#issuecomment-1947554480
LGTM. But I would prefer Others to also take a look - maybe there is a good reason for not having it hashable :). And thanks for the thorough research on the past !
BTW. Static checks need fixing on that one (I heartily recommend installing pre-commit and largely stop caring about thsoe failures).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make Datasets hashable [airflow]
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on code in PR #37465:
URL: https://github.com/apache/airflow/pull/37465#discussion_r1497500920
##########
airflow/datasets/__init__.py:
##########
@@ -46,3 +46,12 @@ def _check_uri(self, attr, uri: str):
def __fspath__(self):
return self.uri
+
+ def __eq__(self, other):
+ if isinstance(other, self.__class__):
+ return self.uri == other.uri
+ else:
+ return NotImplemented
+
+ def __hash__(self):
+ return hash(self.uri)
Review Comment:
Just wondering is also `extra` should be take into the account when calculate extra? If so it would be prettry difficult because dicts are unhashable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make Datasets hashable [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #37465:
URL: https://github.com/apache/airflow/pull/37465
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org