You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "uranusjr (via GitHub)" <gi...@apache.org> on 2024/01/10 12:23:33 UTC
[PR] Introduce DatasetTimetable [airflow]
uranusjr opened a new pull request, #36710:
URL: https://github.com/apache/airflow/pull/36710
This special timetable allows a DAG to be run against a time-based schedule and dataset events at the same time. The logic is nothing special---scheduled runs are created based on a time-based timetable, and dataset-triggered runs are created when dataset events happen. The two do not interact in any way.
I aim to maybe refactor this a bit so a DAG can take more complex combinations (e.g. `dataset1 OR dataset2 OR timetable`), but this is the simplest thing for the most common use case of “rerun DAG whenever things update, but also periodically once in a while”.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Introduce DatasetTimetable [airflow]
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #36710:
URL: https://github.com/apache/airflow/pull/36710#discussion_r1469261013
##########
airflow/timetables/datasets.py:
##########
@@ -0,0 +1,90 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+import collections.abc
+import typing
+
+import attrs
+
+from airflow.datasets import Dataset
+from airflow.exceptions import AirflowTimetableInvalid
+from airflow.timetables.simple import DatasetTriggeredTimetable
+from airflow.utils.types import DagRunType
+
+if typing.TYPE_CHECKING:
+ import pendulum
+
+ from airflow.timetables.base import DagRunInfo, DataInterval, TimeRestriction, Timetable
+
+
+class DatasetTimetable(DatasetTriggeredTimetable):
+ """Combine time-based scheduling with event-based scheduling."""
+
+ def __init__(self, time: Timetable, event: collections.abc.Collection[Dataset]) -> None:
Review Comment:
Or `datasets`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Introduce DatasetTimetable [airflow]
Posted by "pankajkoti (via GitHub)" <gi...@apache.org>.
pankajkoti commented on code in PR #36710:
URL: https://github.com/apache/airflow/pull/36710#discussion_r1466198807
##########
airflow/timetables/datasets.py:
##########
@@ -0,0 +1,90 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+import collections.abc
+import typing
+
+import attrs
+
+from airflow.datasets import Dataset
+from airflow.exceptions import AirflowTimetableInvalid
+from airflow.timetables.simple import DatasetTriggeredTimetable
+from airflow.utils.types import DagRunType
+
+if typing.TYPE_CHECKING:
+ import pendulum
+
+ from airflow.timetables.base import DagRunInfo, DataInterval, TimeRestriction, Timetable
+
+
+class DatasetTimetable(DatasetTriggeredTimetable):
+ """Combine time-based scheduling with event-based scheduling."""
+
+ def __init__(self, time: Timetable, event: collections.abc.Collection[Dataset]) -> None:
Review Comment:
I am also wondering here if the second param `event` here might be confused with the `EventsTimetable` that we have
https://github.com/apache/airflow/blob/c01daf811925816e9ae09b78c37b9ff8d87ce691/airflow/timetables/events.py#L33
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Introduce DatasetTimetable [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #36710:
URL: https://github.com/apache/airflow/pull/36710#discussion_r1466590749
##########
airflow/timetables/datasets.py:
##########
@@ -0,0 +1,90 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+import collections.abc
+import typing
+
+import attrs
+
+from airflow.datasets import Dataset
+from airflow.exceptions import AirflowTimetableInvalid
+from airflow.timetables.simple import DatasetTriggeredTimetable
+from airflow.utils.types import DagRunType
+
+if typing.TYPE_CHECKING:
+ import pendulum
+
+ from airflow.timetables.base import DagRunInfo, DataInterval, TimeRestriction, Timetable
+
+
+class DatasetTimetable(DatasetTriggeredTimetable):
Review Comment:
I am afraid that ship has sailed (TimeTable). It's original sin of using TimeTable where we have not imagined ther might be other triggers.
But we could potentially consider making an Alias and rename the base class as well - possibly that's worth considering. Unfortunately Trigger has alredy been taken for something else (this would be probably best name).
Maybe just `Schedule` - taking into account that the canonical name of the parameter is "schedule' and:
```
ScheduleArg = Union[ArgNotSet, ScheduleInterval, Timetable, Collection["Dataset"]]
```
While schedule is also time-bound, it's far less than TimeTable:
```python
DatasetOrTimeSchedule(CustomSchedule):
```
That would be nice if we do it this way I think,
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Introduce DatasetTimetable [airflow]
Posted by "sunank200 (via GitHub)" <gi...@apache.org>.
sunank200 commented on code in PR #36710:
URL: https://github.com/apache/airflow/pull/36710#discussion_r1466192120
##########
airflow/timetables/datasets.py:
##########
@@ -0,0 +1,90 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+import collections.abc
+import typing
+
+import attrs
+
+from airflow.datasets import Dataset
+from airflow.exceptions import AirflowTimetableInvalid
+from airflow.timetables.simple import DatasetTriggeredTimetable
+from airflow.utils.types import DagRunType
+
+if typing.TYPE_CHECKING:
+ import pendulum
+
+ from airflow.timetables.base import DagRunInfo, DataInterval, TimeRestriction, Timetable
+
+
+class DatasetTimetable(DatasetTriggeredTimetable):
Review Comment:
I liked `DatasetOrTimeTiggeredTimetable` better as the purpose is clear from this name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Introduce `DatasetOrTimeSchedule` [airflow]
Posted by "dstandish (via GitHub)" <gi...@apache.org>.
dstandish commented on code in PR #36710:
URL: https://github.com/apache/airflow/pull/36710#discussion_r1471754215
##########
airflow/timetables/datasets.py:
##########
@@ -0,0 +1,90 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+import collections.abc
+import typing
+
+import attrs
+
+from airflow.datasets import Dataset
+from airflow.exceptions import AirflowTimetableInvalid
+from airflow.timetables.simple import DatasetTriggeredTimetable as DatasetTriggeredSchedule
+from airflow.utils.types import DagRunType
+
+if typing.TYPE_CHECKING:
+ import pendulum
+
+ from airflow.timetables.base import DagRunInfo, DataInterval, TimeRestriction, Timetable
+
+
+class DatasetOrTimeSchedule(DatasetTriggeredSchedule):
+ """Combine time-based scheduling with event-based scheduling."""
+
+ def __init__(self, time: Timetable, datasets: collections.abc.Collection[Dataset]) -> None:
+ self.time = time
Review Comment:
i'm renaming `time` to `timetable`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Introduce `DatasetOrTimeSchedule` [airflow]
Posted by "phanikumv (via GitHub)" <gi...@apache.org>.
phanikumv merged PR #36710:
URL: https://github.com/apache/airflow/pull/36710
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Introduce DatasetTimetable [airflow]
Posted by "sunank200 (via GitHub)" <gi...@apache.org>.
sunank200 commented on code in PR #36710:
URL: https://github.com/apache/airflow/pull/36710#discussion_r1466204093
##########
airflow/timetables/datasets.py:
##########
@@ -0,0 +1,90 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+import collections.abc
+import typing
+
+import attrs
+
+from airflow.datasets import Dataset
+from airflow.exceptions import AirflowTimetableInvalid
+from airflow.timetables.simple import DatasetTriggeredTimetable
+from airflow.utils.types import DagRunType
+
+if typing.TYPE_CHECKING:
+ import pendulum
+
+ from airflow.timetables.base import DagRunInfo, DataInterval, TimeRestriction, Timetable
+
+
+class DatasetTimetable(DatasetTriggeredTimetable):
+ """Combine time-based scheduling with event-based scheduling."""
+
+ def __init__(self, time: Timetable, event: collections.abc.Collection[Dataset]) -> None:
Review Comment:
Should we just rename this to `dataset` instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Introduce DatasetTimetable [airflow]
Posted by "pankajkoti (via GitHub)" <gi...@apache.org>.
pankajkoti commented on code in PR #36710:
URL: https://github.com/apache/airflow/pull/36710#discussion_r1466196636
##########
airflow/timetables/datasets.py:
##########
@@ -0,0 +1,90 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+import collections.abc
+import typing
+
+import attrs
+
+from airflow.datasets import Dataset
+from airflow.exceptions import AirflowTimetableInvalid
+from airflow.timetables.simple import DatasetTriggeredTimetable
+from airflow.utils.types import DagRunType
+
+if typing.TYPE_CHECKING:
+ import pendulum
+
+ from airflow.timetables.base import DagRunInfo, DataInterval, TimeRestriction, Timetable
+
+
+class DatasetTimetable(DatasetTriggeredTimetable):
Review Comment:
I am just reading about TimeTable and have not really used it before.
Looking at the current codebase, would like to understand the below point?
I am wondering if it's wise to have called the existing class as `DatasetTriggeredTimetable` - I am thinking that since the trigger is not actually based on time, should it still be a TimeTable (meaning should it contain TimeTable in its name DatasetTriggeredTimetable)?
If we feel it wise, then I have a +1 to merging these 2 with default `time=None` and also +1 to `DatasetOrTimeTiggeredTimetable`.
Otherwise, if we consider renaming the existing class and creating another class for the OR, then can rethink of the name to provide inputs here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Introduce DatasetTimetable [airflow]
Posted by "sunank200 (via GitHub)" <gi...@apache.org>.
sunank200 commented on code in PR #36710:
URL: https://github.com/apache/airflow/pull/36710#discussion_r1469598323
##########
airflow/timetables/datasets.py:
##########
@@ -0,0 +1,90 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+import collections.abc
+import typing
+
+import attrs
+
+from airflow.datasets import Dataset
+from airflow.exceptions import AirflowTimetableInvalid
+from airflow.timetables.simple import DatasetTriggeredTimetable
+from airflow.utils.types import DagRunType
+
+if typing.TYPE_CHECKING:
+ import pendulum
+
+ from airflow.timetables.base import DagRunInfo, DataInterval, TimeRestriction, Timetable
+
+
+class DatasetTimetable(DatasetTriggeredTimetable):
+ """Combine time-based scheduling with event-based scheduling."""
+
+ def __init__(self, time: Timetable, event: collections.abc.Collection[Dataset]) -> None:
Review Comment:
Changed it here: https://github.com/apache/airflow/pull/36710/commits/9c3abb7b2e2dad2fd7122db535a364b2433087c1
##########
airflow/timetables/datasets.py:
##########
@@ -0,0 +1,90 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+import collections.abc
+import typing
+
+import attrs
+
+from airflow.datasets import Dataset
+from airflow.exceptions import AirflowTimetableInvalid
+from airflow.timetables.simple import DatasetTriggeredTimetable
+from airflow.utils.types import DagRunType
+
+if typing.TYPE_CHECKING:
+ import pendulum
+
+ from airflow.timetables.base import DagRunInfo, DataInterval, TimeRestriction, Timetable
+
+
+class DatasetTimetable(DatasetTriggeredTimetable):
Review Comment:
Changed this: https://github.com/apache/airflow/pull/36710/commits/9c3abb7b2e2dad2fd7122db535a364b2433087c1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Introduce DatasetTimetable [airflow]
Posted by "dstandish (via GitHub)" <gi...@apache.org>.
dstandish commented on code in PR #36710:
URL: https://github.com/apache/airflow/pull/36710#discussion_r1464333679
##########
airflow/timetables/datasets.py:
##########
@@ -0,0 +1,90 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+import collections.abc
+import typing
+
+import attrs
+
+from airflow.datasets import Dataset
+from airflow.exceptions import AirflowTimetableInvalid
+from airflow.timetables.simple import DatasetTriggeredTimetable
+from airflow.utils.types import DagRunType
+
+if typing.TYPE_CHECKING:
+ import pendulum
+
+ from airflow.timetables.base import DagRunInfo, DataInterval, TimeRestriction, Timetable
+
+
+class DatasetTimetable(DatasetTriggeredTimetable):
Review Comment:
We do need to come up with a better name. Maybe start a thread on slack for some brainstorming?
There should be something to indicate that it is dataset plus schedule... unless.... unless ... actually maybe we should just update DatasetTriggeredTimetable so that it works like this? @uranusjr ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Introduce DatasetTimetable [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #36710:
URL: https://github.com/apache/airflow/pull/36710#discussion_r1466592295
##########
airflow/timetables/datasets.py:
##########
@@ -0,0 +1,90 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+import collections.abc
+import typing
+
+import attrs
+
+from airflow.datasets import Dataset
+from airflow.exceptions import AirflowTimetableInvalid
+from airflow.timetables.simple import DatasetTriggeredTimetable
+from airflow.utils.types import DagRunType
+
+if typing.TYPE_CHECKING:
+ import pendulum
+
+ from airflow.timetables.base import DagRunInfo, DataInterval, TimeRestriction, Timetable
+
+
+class DatasetTimetable(DatasetTriggeredTimetable):
Review Comment:
But that would have a bit of a ripple-effect on a number of classes I am afraid.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Introduce DatasetTimetable [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #36710:
URL: https://github.com/apache/airflow/pull/36710#discussion_r1465617777
##########
airflow/timetables/datasets.py:
##########
@@ -0,0 +1,90 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+import collections.abc
+import typing
+
+import attrs
+
+from airflow.datasets import Dataset
+from airflow.exceptions import AirflowTimetableInvalid
+from airflow.timetables.simple import DatasetTriggeredTimetable
+from airflow.utils.types import DagRunType
+
+if typing.TYPE_CHECKING:
+ import pendulum
+
+ from airflow.timetables.base import DagRunInfo, DataInterval, TimeRestriction, Timetable
+
+
+class DatasetTimetable(DatasetTriggeredTimetable):
Review Comment:
Yeah. I would be also for just merging it with DatasetTriggeredTimetable (where default behaviour on dataset dependencies would be time = None. I think there is no need to muliply the classes. As the docstring on DatasetTriggeredTimetable states - it should not be used anywhere by the users in DAGs, so we should be able to change the name easily.
My proposals for that combined classs would be `DatasetOrTimeTiggeredTimetable`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org