You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Lee-W (via GitHub)" <gi...@apache.org> on 2023/07/07 09:08:53 UTC
[GitHub] [airflow] Lee-W opened a new pull request, #32422: docs(deferring): add type annotation to code examples
Lee-W opened a new pull request, #32422:
URL: https://github.com/apache/airflow/pull/32422
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] syedahsn commented on a diff in pull request #32422: docs(deferring): add type annotation to code examples
Posted by "syedahsn (via GitHub)" <gi...@apache.org>.
syedahsn commented on code in PR #32422:
URL: https://github.com/apache/airflow/pull/32422#discussion_r1256202070
##########
docs/apache-airflow/authoring-and-scheduling/deferring.rst:
##########
@@ -56,38 +56,44 @@ Writing a deferrable operator takes a bit more work. There are some main points
* You can defer multiple times, and you can defer before/after your Operator does significant work, or only defer if certain conditions are met (e.g. a system does not have an immediate answer). Deferral is entirely under your control.
* Any Operator can defer; no special marking on its class is needed, and it's not limited to Sensors.
* In order for any changes to a Trigger to be reflected, the *triggerer* needs to be restarted whenever the Trigger is modified.
-* If you want add an operator or sensor that supports both deferrable and non-deferrable modes. It's suggested to add ``deferable: bool = conf.getboolean("operators", "default_deferrable", fallback=False)`` to the ``__init__`` method of the operator and use it to decide whether to run the operator in deferrable mode. You'll be able to configure the default value of ``deferrable`` of all the operators and sensors that supports switch between deferrable and non-deferrable mode through ``default_deferrable`` in the ``operator`` section. Here's an example of a sensor that supports both modes.::
+* If you want add an operator or sensor that supports both deferrable and non-deferrable modes. It's suggested to add ``deferrable: bool = conf.getboolean("operators", "default_deferrable", fallback=False)`` to the ``__init__`` method of the operator and use it to decide whether to run the operator in deferrable mode. You'll be able to configure the default value of ``deferrable`` of all the operators and sensors that supports switch between deferrable and non-deferrable mode through ``default_deferrable`` in the ``operator`` section. Here's an example of a sensor that supports both modes.
Review Comment:
small nitpick
```suggestion
* If you want to add an operator or sensor that supports both deferrable and non-deferrable modes, it's suggested to add ``deferrable: bool = conf.getboolean("operators", "default_deferrable", fallback=False)`` to the ``__init__`` method of the operator and use it to decide whether to run the operator in deferrable mode. You'll be able to configure the default value of ``deferrable`` of all the operators and sensors that support switching between deferrable and non-deferrable mode through ``default_deferrable`` in the ``operator`` section. Here's an example of a sensor that supports both modes.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a diff in pull request #32422: docs(deferring): add type annotation to code examples
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #32422:
URL: https://github.com/apache/airflow/pull/32422#discussion_r1255545723
##########
docs/apache-airflow/authoring-and-scheduling/deferring.rst:
##########
@@ -109,16 +115,22 @@ You are free to set ``method_name`` to ``execute`` if you want your Operator to
Here's a basic example of how a sensor might trigger deferral::
Review Comment:
Let’s also use this chance to make these `.. code-block:: python` instead so these are better syntax-highlighted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] phanikumv commented on a diff in pull request #32422: docs(deferring): add type annotation to code examples
Posted by "phanikumv (via GitHub)" <gi...@apache.org>.
phanikumv commented on code in PR #32422:
URL: https://github.com/apache/airflow/pull/32422#discussion_r1255901266
##########
docs/apache-airflow/authoring-and-scheduling/deferring.rst:
##########
@@ -56,38 +56,44 @@ Writing a deferrable operator takes a bit more work. There are some main points
* You can defer multiple times, and you can defer before/after your Operator does significant work, or only defer if certain conditions are met (e.g. a system does not have an immediate answer). Deferral is entirely under your control.
* Any Operator can defer; no special marking on its class is needed, and it's not limited to Sensors.
* In order for any changes to a Trigger to be reflected, the *triggerer* needs to be restarted whenever the Trigger is modified.
-* If you want add an operator or sensor that supports both deferrable and non-deferrable modes. It's suggested to add ``deferable: bool = conf.getboolean("operators", "default_deferrable", fallback=False)`` to the ``__init__`` method of the operator and use it to decide whether to run the operator in deferrable mode. You'll be able to configure the default value of ``deferrable`` of all the operators and sensors that supports switch between deferrable and non-deferrable mode through ``default_deferrable`` in the ``operator`` section. Here's an example of a sensor that supports both modes.::
+* If you want add an operator or sensor that supports both deferrable and non-deferrable modes. It's suggested to add ``deferable: bool = conf.getboolean("operators", "default_deferrable", fallback=False)`` to the ``__init__`` method of the operator and use it to decide whether to run the operator in deferrable mode. You'll be able to configure the default value of ``deferrable`` of all the operators and sensors that supports switch between deferrable and non-deferrable mode through ``default_deferrable`` in the ``operator`` section. Here's an example of a sensor that supports both modes.
Review Comment:
```suggestion
* If you want add an operator or sensor that supports both deferrable and non-deferrable modes. It's suggested to add ``deferrable: bool = conf.getboolean("operators", "default_deferrable", fallback=False)`` to the ``__init__`` method of the operator and use it to decide whether to run the operator in deferrable mode. You'll be able to configure the default value of ``deferrable`` of all the operators and sensors that supports switch between deferrable and non-deferrable mode through ``default_deferrable`` in the ``operator`` section. Here's an example of a sensor that supports both modes.
```
##########
docs/apache-airflow/authoring-and-scheduling/deferring.rst:
##########
@@ -56,38 +56,44 @@ Writing a deferrable operator takes a bit more work. There are some main points
* You can defer multiple times, and you can defer before/after your Operator does significant work, or only defer if certain conditions are met (e.g. a system does not have an immediate answer). Deferral is entirely under your control.
* Any Operator can defer; no special marking on its class is needed, and it's not limited to Sensors.
* In order for any changes to a Trigger to be reflected, the *triggerer* needs to be restarted whenever the Trigger is modified.
-* If you want add an operator or sensor that supports both deferrable and non-deferrable modes. It's suggested to add ``deferable: bool = conf.getboolean("operators", "default_deferrable", fallback=False)`` to the ``__init__`` method of the operator and use it to decide whether to run the operator in deferrable mode. You'll be able to configure the default value of ``deferrable`` of all the operators and sensors that supports switch between deferrable and non-deferrable mode through ``default_deferrable`` in the ``operator`` section. Here's an example of a sensor that supports both modes.::
+* If you want add an operator or sensor that supports both deferrable and non-deferrable modes. It's suggested to add ``deferable: bool = conf.getboolean("operators", "default_deferrable", fallback=False)`` to the ``__init__`` method of the operator and use it to decide whether to run the operator in deferrable mode. You'll be able to configure the default value of ``deferrable`` of all the operators and sensors that supports switch between deferrable and non-deferrable mode through ``default_deferrable`` in the ``operator`` section. Here's an example of a sensor that supports both modes.
+
+.. code-block:: python
import time
from datetime import timedelta
+ from typing import Any
+ from airflow.configuration import conf
from airflow.sensors.base import BaseSensorOperator
from airflow.triggers.temporal import TimeDeltaTrigger
+ from airflow.utils.context import Context
class WaitOneHourSensor(BaseSensorOperator):
def __init__(
- self,
- deferable: bool = conf.getboolean("operators", "default_deferrable", fallback=False),
- **kwargs
- ):
+ self, deferable: bool = conf.getboolean("operators", "default_deferrable", fallback=False), **kwargs
Review Comment:
```suggestion
self, deferrable: bool = conf.getboolean("operators", "default_deferrable", fallback=False), **kwargs
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #32422: docs(deferring): add type annotation to code examples
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #32422:
URL: https://github.com/apache/airflow/pull/32422
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Lee-W commented on pull request #32422: docs(deferring): add type annotation to code examples
Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on PR #32422:
URL: https://github.com/apache/airflow/pull/32422#issuecomment-1625276854
ah yes, I've changed all the code block into that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Lee-W commented on a diff in pull request #32422: docs(deferring): add type annotation to code examples
Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on code in PR #32422:
URL: https://github.com/apache/airflow/pull/32422#discussion_r1257407918
##########
docs/apache-airflow/authoring-and-scheduling/deferring.rst:
##########
@@ -56,38 +56,44 @@ Writing a deferrable operator takes a bit more work. There are some main points
* You can defer multiple times, and you can defer before/after your Operator does significant work, or only defer if certain conditions are met (e.g. a system does not have an immediate answer). Deferral is entirely under your control.
* Any Operator can defer; no special marking on its class is needed, and it's not limited to Sensors.
* In order for any changes to a Trigger to be reflected, the *triggerer* needs to be restarted whenever the Trigger is modified.
-* If you want add an operator or sensor that supports both deferrable and non-deferrable modes. It's suggested to add ``deferable: bool = conf.getboolean("operators", "default_deferrable", fallback=False)`` to the ``__init__`` method of the operator and use it to decide whether to run the operator in deferrable mode. You'll be able to configure the default value of ``deferrable`` of all the operators and sensors that supports switch between deferrable and non-deferrable mode through ``default_deferrable`` in the ``operator`` section. Here's an example of a sensor that supports both modes.::
+* If you want add an operator or sensor that supports both deferrable and non-deferrable modes. It's suggested to add ``deferrable: bool = conf.getboolean("operators", "default_deferrable", fallback=False)`` to the ``__init__`` method of the operator and use it to decide whether to run the operator in deferrable mode. You'll be able to configure the default value of ``deferrable`` of all the operators and sensors that supports switch between deferrable and non-deferrable mode through ``default_deferrable`` in the ``operator`` section. Here's an example of a sensor that supports both modes.
Review Comment:
As the sensor is a kind of operator as well https://github.com/apache/airflow/blob/main/airflow/sensors/base.py#L82, it seems to me this should be fine. But I'm good with making this change as well. We probably should add `default_deferrable` [here](https://github.com/apache/airflow/blob/main/airflow/config_templates/config.yml#L2943). Maybe that should be another PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Lee-W commented on a diff in pull request #32422: docs(deferring): add type annotation to code examples
Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on code in PR #32422:
URL: https://github.com/apache/airflow/pull/32422#discussion_r1255621296
##########
docs/apache-airflow/authoring-and-scheduling/deferring.rst:
##########
@@ -109,16 +115,22 @@ You are free to set ``method_name`` to ``execute`` if you want your Operator to
Here's a basic example of how a sensor might trigger deferral::
Review Comment:
Sure. Just updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pankajastro commented on pull request #32422: docs(deferring): add type annotation to code examples
Posted by "pankajastro (via GitHub)" <gi...@apache.org>.
pankajastro commented on PR #32422:
URL: https://github.com/apache/airflow/pull/32422#issuecomment-1625236504
> Sorry, I kinda don't get it. Could you point out where is it?
sure, type annotation for trigger examples also code block https://github.com/apache/airflow/blob/main/docs/apache-airflow/authoring-and-scheduling/deferring.rst?plain=1#L111-L124
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Lee-W commented on pull request #32422: docs(deferring): add type annotation to code examples
Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on PR #32422:
URL: https://github.com/apache/airflow/pull/32422#issuecomment-1625212293
> we should add the same for example`DateTimeTrigger` too we have 2 serialize and run methods there wdyt?
Sorry, I kinda don't get it. Could you point out where is it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] phanikumv commented on a diff in pull request #32422: docs(deferring): add type annotation to code examples
Posted by "phanikumv (via GitHub)" <gi...@apache.org>.
phanikumv commented on code in PR #32422:
URL: https://github.com/apache/airflow/pull/32422#discussion_r1256062782
##########
docs/apache-airflow/authoring-and-scheduling/deferring.rst:
##########
@@ -56,38 +56,44 @@ Writing a deferrable operator takes a bit more work. There are some main points
* You can defer multiple times, and you can defer before/after your Operator does significant work, or only defer if certain conditions are met (e.g. a system does not have an immediate answer). Deferral is entirely under your control.
* Any Operator can defer; no special marking on its class is needed, and it's not limited to Sensors.
* In order for any changes to a Trigger to be reflected, the *triggerer* needs to be restarted whenever the Trigger is modified.
-* If you want add an operator or sensor that supports both deferrable and non-deferrable modes. It's suggested to add ``deferable: bool = conf.getboolean("operators", "default_deferrable", fallback=False)`` to the ``__init__`` method of the operator and use it to decide whether to run the operator in deferrable mode. You'll be able to configure the default value of ``deferrable`` of all the operators and sensors that supports switch between deferrable and non-deferrable mode through ``default_deferrable`` in the ``operator`` section. Here's an example of a sensor that supports both modes.::
+* If you want add an operator or sensor that supports both deferrable and non-deferrable modes. It's suggested to add ``deferrable: bool = conf.getboolean("operators", "default_deferrable", fallback=False)`` to the ``__init__`` method of the operator and use it to decide whether to run the operator in deferrable mode. You'll be able to configure the default value of ``deferrable`` of all the operators and sensors that supports switch between deferrable and non-deferrable mode through ``default_deferrable`` in the ``operator`` section. Here's an example of a sensor that supports both modes.
Review Comment:
Here we are talking about both operators and sensors, but giving an example of `deferrable: bool = conf.getboolean("operators", "default_deferrable", fallback=False)` only for operators.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org