You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "raphaelauv (via GitHub)" <gi...@apache.org> on 2024/07/22 07:11:04 UTC
[PR] fix: sensor - skip_policy [airflow]
raphaelauv opened a new pull request, #40924:
URL: https://github.com/apache/airflow/pull/40924
fix: https://github.com/apache/airflow/pull/40915
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Refactor `BaseSensorOperator` introduce `skip_policy` parameter [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #40924:
URL: https://github.com/apache/airflow/pull/40924#issuecomment-2252299336
Hey @raphaelauv -> This one broke our canary builds https://github.com/apache/airflow/actions/runs/10107479883/job/27951619130 -> can you also apply the new policy to all the operators that are raising warnings now ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Refactor `BaseSensorOperator` introduce `skip_policy` parameter [airflow]
Posted by "raphaelauv (via GitHub)" <gi...@apache.org>.
raphaelauv commented on PR #40924:
URL: https://github.com/apache/airflow/pull/40924#issuecomment-2250246677
@eladkal gentle ping ( I would like to finish this work for airflow 2.10 , so we can trust sensors again in airflow , cause right now we can't have at least once pattern with the soft_fail option )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Refactor `BaseSensorOperator` introduce `skip_policy` parameter [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #40924:
URL: https://github.com/apache/airflow/pull/40924#issuecomment-2252390891
BTW, This should be made compatible with Airflow 2.7 + (maybe we could use common.compat provider.
In tne meantime, I will revert this one and maybe it will be better if you combine the two together this time ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Refactor `BaseSensorOperator` introduce `skip_policy` parameter [airflow]
Posted by "raphaelauv (via GitHub)" <gi...@apache.org>.
raphaelauv commented on code in PR #40924:
URL: https://github.com/apache/airflow/pull/40924#discussion_r1693641088
##########
airflow/sensors/base.py:
##########
@@ -114,15 +118,33 @@ def _orig_start_date(
)
+class SkipPolicy(str, enum.Enum):
+ """Class with sensor's skip policies."""
+
+ # if poke method raise an exception, sensor will not be skipped on.
+ NONE = "none"
+
+ # If poke method raises an exception, sensor will be skipped on.
+ SKIP_ON_ANY_ERROR = "skip_on_any_error"
+
+ # If poke method raises AirflowSensorTimeout, AirflowTaskTimeout, AirflowFailException
+ # sensor will be skipped on.
+ SKIP_ON_SOFT_ERROR = "skip_on_soft_error"
Review Comment:
what about SKIP_ON_TIMEOUT and remove AirflowFailException from the list of exception catch by this option
naming would be explicit and correct
wdyt ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: sensor - skip_policy [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on code in PR #40924:
URL: https://github.com/apache/airflow/pull/40924#discussion_r1687030696
##########
airflow/sensors/base.py:
##########
@@ -186,11 +208,44 @@ def __init__(
self.mode = mode
self.exponential_backoff = exponential_backoff
self.max_wait = self._coerce_max_wait(max_wait)
- if soft_fail is True and never_fail is True:
- raise ValueError("soft_fail and never_fail are mutually exclusive, you can not provide both.")
+ if skip_policy != NOTSET:
+ if sum([soft_fail, silent_fail]) > 0:
+ raise ValueError(
+ "skip_policy and deprecated soft_fail and silent_fail parameters are mutually exclusive."
+ )
+
+ if skip_policy == SkipPolicy.SKIP_ON_SOFT_ERROR:
Review Comment:
Im curious what if user has custom skip exception?
Can the user add support for it from deployment side?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Refactor `BaseSensorOperator` introduce `skip_policy` parameter [airflow]
Posted by "raphaelauv (via GitHub)" <gi...@apache.org>.
raphaelauv commented on code in PR #40924:
URL: https://github.com/apache/airflow/pull/40924#discussion_r1687582966
##########
airflow/sensors/base.py:
##########
@@ -114,15 +118,33 @@ def _orig_start_date(
)
+class SkipPolicy(str, enum.Enum):
+ """Class with sensor's skip policies."""
+
+ # if poke method raise an exception, sensor will not be skipped on.
+ NONE = "none"
+
+ # If poke method raises an exception, sensor will be skipped on.
+ SKIP_ON_ANY_ERROR = "skip_on_any_error"
+
+ # If poke method raises AirflowSensorTimeout, AirflowTaskTimeout, AirflowFailException
+ # sensor will be skipped on.
+ SKIP_ON_SOFT_ERROR = "skip_on_soft_error"
+
+ # If poke method raises an exception different from AirflowSensorTimeout, AirflowTaskTimeout,
+ # AirflowSkipException, sensor will ignore exception and re-poke until timeout.
+ IGNORE_ERROR = "ignore_error"
Review Comment:
this is a copy past of the definition of `silent_fail` , could you please make a suggestion , thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Refactor `BaseSensorOperator` introduce `skip_policy` parameter [airflow]
Posted by "raphaelauv (via GitHub)" <gi...@apache.org>.
raphaelauv commented on code in PR #40924:
URL: https://github.com/apache/airflow/pull/40924#discussion_r1693641088
##########
airflow/sensors/base.py:
##########
@@ -114,15 +118,33 @@ def _orig_start_date(
)
+class SkipPolicy(str, enum.Enum):
+ """Class with sensor's skip policies."""
+
+ # if poke method raise an exception, sensor will not be skipped on.
+ NONE = "none"
+
+ # If poke method raises an exception, sensor will be skipped on.
+ SKIP_ON_ANY_ERROR = "skip_on_any_error"
+
+ # If poke method raises AirflowSensorTimeout, AirflowTaskTimeout, AirflowFailException
+ # sensor will be skipped on.
+ SKIP_ON_SOFT_ERROR = "skip_on_soft_error"
Review Comment:
what about SKIP_ON_TIMEOUT and remove AirflowFailException from the list of exception catch by option
naming would be explicit and correct
wdyt ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Refactor `BaseSensorOperator` introduce `skip_policy` parameter [airflow]
Posted by "raphaelauv (via GitHub)" <gi...@apache.org>.
raphaelauv commented on code in PR #40924:
URL: https://github.com/apache/airflow/pull/40924#discussion_r1687593473
##########
airflow/sensors/base.py:
##########
@@ -186,11 +208,44 @@ def __init__(
self.mode = mode
self.exponential_backoff = exponential_backoff
self.max_wait = self._coerce_max_wait(max_wait)
- if soft_fail is True and never_fail is True:
- raise ValueError("soft_fail and never_fail are mutually exclusive, you can not provide both.")
+ if skip_policy != NOTSET:
+ if sum([soft_fail, silent_fail]) > 0:
+ raise ValueError(
+ "skip_policy and deprecated soft_fail and silent_fail parameters are mutually exclusive."
+ )
+
+ if skip_policy == SkipPolicy.SKIP_ON_SOFT_ERROR:
Review Comment:
users already repack custom exceptions https://github.com/apache/airflow/blob/e9d2d5c86a714d6aea7c4dfb3b8cf72e54b9035c/airflow/providers/sftp/sensors/sftp.py#L103
```Can the user add support for it from deployment side```
It feel like this is another feature , we can create an issue to track that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Refactor `BaseSensorOperator` introduce `skip_policy` parameter [airflow]
Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on code in PR #40924:
URL: https://github.com/apache/airflow/pull/40924#discussion_r1690654838
##########
airflow/sensors/base.py:
##########
@@ -114,15 +118,33 @@ def _orig_start_date(
)
+class SkipPolicy(str, enum.Enum):
+ """Class with sensor's skip policies."""
+
+ # if poke method raise an exception, sensor will not be skipped on.
+ NONE = "none"
+
+ # If poke method raises an exception, sensor will be skipped on.
+ SKIP_ON_ANY_ERROR = "skip_on_any_error"
+
+ # If poke method raises AirflowSensorTimeout, AirflowTaskTimeout, AirflowFailException
+ # sensor will be skipped on.
+ SKIP_ON_SOFT_ERROR = "skip_on_soft_error"
+
+ # If poke method raises an exception different from AirflowSensorTimeout, AirflowTaskTimeout,
+ # AirflowSkipException, sensor will ignore exception and re-poke until timeout.
+ IGNORE_ERROR = "ignore_error"
Review Comment:
Hi, sorry for the late reply. Nothing comes to my mind as of now. As long as we document it, I think we're good. Thanks 🙂 Resolve this conversation.
##########
airflow/sensors/base.py:
##########
@@ -114,15 +118,33 @@ def _orig_start_date(
)
+class SkipPolicy(str, enum.Enum):
+ """Class with sensor's skip policies."""
+
+ # if poke method raise an exception, sensor will not be skipped on.
+ NONE = "none"
+
+ # If poke method raises an exception, sensor will be skipped on.
+ SKIP_ON_ANY_ERROR = "skip_on_any_error"
+
+ # If poke method raises AirflowSensorTimeout, AirflowTaskTimeout, AirflowFailException
+ # sensor will be skipped on.
+ SKIP_ON_SOFT_ERROR = "skip_on_soft_error"
Review Comment:
Hi, sorry for the late reply. Nothing comes to my mind as of now. As long as we document it, I think we're good. Thanks 🙂 Resolve this conversation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: sensor - skip_policy [airflow]
Posted by "raphaelauv (via GitHub)" <gi...@apache.org>.
raphaelauv commented on PR #40924:
URL: https://github.com/apache/airflow/pull/40924#issuecomment-2242861441
@Lee-W @shahar1 @eladkal wdyt ? thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Refactor `BaseSensorOperator` introduce `skip_policy` parameter [airflow]
Posted by "raphaelauv (via GitHub)" <gi...@apache.org>.
raphaelauv commented on PR #40924:
URL: https://github.com/apache/airflow/pull/40924#issuecomment-2252348744
@potiuk yes I started the replace in this PR -> https://github.com/apache/airflow/pull/40923
i'm going to finish in few hours
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Refactor `BaseSensorOperator` introduce `skip_policy` parameter [airflow]
Posted by "raphaelauv (via GitHub)" <gi...@apache.org>.
raphaelauv commented on PR #40924:
URL: https://github.com/apache/airflow/pull/40924#issuecomment-2248485115
@eladkal should we merge this part and continue to iterate with the next PR's or wait for another review of @Lee-W ? thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Refactor `BaseSensorOperator` introduce `skip_policy` parameter [airflow]
Posted by "raphaelauv (via GitHub)" <gi...@apache.org>.
raphaelauv commented on code in PR #40924:
URL: https://github.com/apache/airflow/pull/40924#discussion_r1687581187
##########
airflow/sensors/base.py:
##########
@@ -114,15 +118,33 @@ def _orig_start_date(
)
+class SkipPolicy(str, enum.Enum):
+ """Class with sensor's skip policies."""
+
+ # if poke method raise an exception, sensor will not be skipped on.
+ NONE = "none"
+
+ # If poke method raises an exception, sensor will be skipped on.
+ SKIP_ON_ANY_ERROR = "skip_on_any_error"
+
+ # If poke method raises AirflowSensorTimeout, AirflowTaskTimeout, AirflowFailException
+ # sensor will be skipped on.
+ SKIP_ON_SOFT_ERROR = "skip_on_soft_error"
Review Comment:
ok , what is your proposition ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Refactor `BaseSensorOperator` introduce `skip_policy` parameter [airflow]
Posted by "raphaelauv (via GitHub)" <gi...@apache.org>.
raphaelauv commented on code in PR #40924:
URL: https://github.com/apache/airflow/pull/40924#discussion_r1687587052
##########
airflow/sensors/base.py:
##########
@@ -114,15 +118,33 @@ def _orig_start_date(
)
+class SkipPolicy(str, enum.Enum):
Review Comment:
yes I will in another PR , when we agree on the implementation , cause I don't want to do big refacto every time we change the terminology
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: sensor - skip_policy [airflow]
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on code in PR #40924:
URL: https://github.com/apache/airflow/pull/40924#discussion_r1686848992
##########
airflow/sensors/base.py:
##########
@@ -114,15 +118,27 @@ def _orig_start_date(
)
+class SkipPolicy(str, enum.Enum):
+ """Action to TODO."""
+
+ NONE = "none"
+
+ # If poke method raises an exception, sensor will be skipped.
+ SKIP_ON_ANY_ERROR = "skip_on_any_error"
+
+ SKIP_ONLY_SOFT_ERROR = "skip_only_soft_error"
Review Comment:
Namings might need some minor improvements, for example:
```suggestion
SKIP_ON_SOFT_ERROR = "skip_on_soft_error"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Refactor `BaseSensorOperator` introduce `skip_policy` parameter [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #40924:
URL: https://github.com/apache/airflow/pull/40924#issuecomment-2252300788
Warnings detected in test case(s):
- tests/providers/apache/flink/sensors/test_flink_kubernetes.py::TestFlinkKubernetesSensor::test_cluster_error_state[True-AirflowSkipException]
- tests/providers/cncf/kubernetes/sensors/test_spark_kubernetes.py::TestSparkKubernetesSensor::test_driver_logging_failure[True-AirflowSkipException]
- tests/providers/cncf/kubernetes/sensors/test_spark_kubernetes.py::TestSparkKubernetesSensor::test_failed_application[True-AirflowSkipException]
- tests/providers/cncf/kubernetes/sensors/test_spark_kubernetes.py::TestSparkKubernetesSensor::test_unknown_application[True-AirflowSkipException]
- tests/providers/datadog/sensors/test_datadog.py::TestDatadogSensor::test_sensor_fail_with_exception[True-AirflowSkipException]
- tests/providers/http/sensors/test_http.py::TestHttpSensor::test_poke_exception_with_soft_fail
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Refactor `BaseSensorOperator` introduce `skip_policy` parameter [airflow]
Posted by "raphaelauv (via GitHub)" <gi...@apache.org>.
raphaelauv commented on code in PR #40924:
URL: https://github.com/apache/airflow/pull/40924#discussion_r1687588463
##########
airflow/sensors/base.py:
##########
@@ -186,11 +208,44 @@ def __init__(
self.mode = mode
self.exponential_backoff = exponential_backoff
self.max_wait = self._coerce_max_wait(max_wait)
- if soft_fail is True and never_fail is True:
- raise ValueError("soft_fail and never_fail are mutually exclusive, you can not provide both.")
+ if skip_policy != NOTSET:
+ if sum([soft_fail, silent_fail]) > 0:
+ raise ValueError(
+ "skip_policy and deprecated soft_fail and silent_fail parameters are mutually exclusive."
+ )
+
+ if skip_policy == SkipPolicy.SKIP_ON_SOFT_ERROR:
+ self.soft_fail = True
Review Comment:
some sensors still use self.soft_fail
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Refactor `BaseSensorOperator` introduce `skip_policy` parameter [airflow]
Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on code in PR #40924:
URL: https://github.com/apache/airflow/pull/40924#discussion_r1687305708
##########
airflow/sensors/base.py:
##########
@@ -114,15 +118,33 @@ def _orig_start_date(
)
+class SkipPolicy(str, enum.Enum):
Review Comment:
I feel we might need to update documentation for this as well
##########
airflow/sensors/base.py:
##########
@@ -186,11 +208,44 @@ def __init__(
self.mode = mode
self.exponential_backoff = exponential_backoff
self.max_wait = self._coerce_max_wait(max_wait)
- if soft_fail is True and never_fail is True:
- raise ValueError("soft_fail and never_fail are mutually exclusive, you can not provide both.")
+ if skip_policy != NOTSET:
+ if sum([soft_fail, silent_fail]) > 0:
+ raise ValueError(
+ "skip_policy and deprecated soft_fail and silent_fail parameters are mutually exclusive."
+ )
+
+ if skip_policy == SkipPolicy.SKIP_ON_SOFT_ERROR:
+ self.soft_fail = True
Review Comment:
If we're already using `skip_policy` to check. Do we still need this assignment here?
##########
airflow/sensors/base.py:
##########
@@ -114,15 +118,33 @@ def _orig_start_date(
)
+class SkipPolicy(str, enum.Enum):
+ """Class with sensor's skip policies."""
+
+ # if poke method raise an exception, sensor will not be skipped on.
+ NONE = "none"
+
+ # If poke method raises an exception, sensor will be skipped on.
+ SKIP_ON_ANY_ERROR = "skip_on_any_error"
+
+ # If poke method raises AirflowSensorTimeout, AirflowTaskTimeout, AirflowFailException
+ # sensor will be skipped on.
+ SKIP_ON_SOFT_ERROR = "skip_on_soft_error"
+
+ # If poke method raises an exception different from AirflowSensorTimeout, AirflowTaskTimeout,
+ # AirflowSkipException, sensor will ignore exception and re-poke until timeout.
+ IGNORE_ERROR = "ignore_error"
Review Comment:
I'm wondering whether we could reword this. It looks similar to `SKIP_ON_ANY_ERROR` at the first glance 🤔
##########
airflow/sensors/base.py:
##########
@@ -114,15 +118,33 @@ def _orig_start_date(
)
+class SkipPolicy(str, enum.Enum):
+ """Class with sensor's skip policies."""
+
+ # if poke method raise an exception, sensor will not be skipped on.
+ NONE = "none"
+
+ # If poke method raises an exception, sensor will be skipped on.
+ SKIP_ON_ANY_ERROR = "skip_on_any_error"
+
+ # If poke method raises AirflowSensorTimeout, AirflowTaskTimeout, AirflowFailException
+ # sensor will be skipped on.
+ SKIP_ON_SOFT_ERROR = "skip_on_soft_error"
Review Comment:
I might suggest rewording this one as well. The description here is great, but `soft`_fail sounds a bit unclear to me
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: sensor - skip_policy [airflow]
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on code in PR #40924:
URL: https://github.com/apache/airflow/pull/40924#discussion_r1686819545
##########
airflow/sensors/base.py:
##########
@@ -114,15 +118,27 @@ def _orig_start_date(
)
+class SkipPolicy(str, enum.Enum):
+ """Action to TODO."""
Review Comment:
Comment is a unclear
##########
airflow/sensors/base.py:
##########
@@ -155,8 +171,8 @@ class BaseSensorOperator(BaseOperator, SkipMixin):
and AirflowFailException, the sensor will log the error and continue
its execution. Otherwise, the sensor task fails, and it can be retried
based on the provided `retries` parameter.
- :param never_fail: If true, and poke method raises an exception, sensor will be skipped.
- Mutually exclusive with soft_fail.
+ Mutually exclusive with soft_fail.
+ :param skip_policy:
Review Comment:
Description needs to be added, probably with details about each option in the enum.
I'd include answers to the following questions:
1. What is a "soft" error?
2. What's the difference between "ignore" and "skip"? (it might not be as trivial for the user)
##########
airflow/sensors/base.py:
##########
@@ -114,15 +118,27 @@ def _orig_start_date(
)
+class SkipPolicy(str, enum.Enum):
+ """Action to TODO."""
+
+ NONE = "none"
+
+ # If poke method raises an exception, sensor will be skipped.
+ SKIP_ON_ANY_ERROR = "skip_on_any_error"
+
+ SKIP_ONLY_SOFT_ERROR = "skip_only_soft_error"
Review Comment:
Namings might need some minor improvements:
```suggestion
SKIP_ON_SOFT_ERROR = "skip_on_soft_error"
```
##########
airflow/sensors/base.py:
##########
@@ -114,15 +118,27 @@ def _orig_start_date(
)
+class SkipPolicy(str, enum.Enum):
+ """Action to TODO."""
+
+ NONE = "none"
+
+ # If poke method raises an exception, sensor will be skipped.
+ SKIP_ON_ANY_ERROR = "skip_on_any_error"
+
+ SKIP_ONLY_SOFT_ERROR = "skip_only_soft_error"
+ IGNORE_ERRORS = "ignore_error"
+
+
class BaseSensorOperator(BaseOperator, SkipMixin):
"""
Sensor operators are derived from this class and inherit these attributes.
Sensor operators keep executing at a time interval and succeed when
a criteria is met and fail if and when they time out.
- :param soft_fail: Set to true to mark the task as SKIPPED on failure.
- Mutually exclusive with never_fail.
+ :param soft_fail: Set to true to mark the task as SKIPPED on AirflowSensorTimeout,
+ AirflowTaskTimeout, AirflowFailException.
Review Comment:
1. Mutual exclusiveness with `silent_fail` should be clarified.
2. It should be marked as deprecated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: sensor - skip_policy [airflow]
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on code in PR #40924:
URL: https://github.com/apache/airflow/pull/40924#discussion_r1686822585
##########
airflow/sensors/base.py:
##########
@@ -155,8 +171,8 @@ class BaseSensorOperator(BaseOperator, SkipMixin):
and AirflowFailException, the sensor will log the error and continue
its execution. Otherwise, the sensor task fails, and it can be retried
based on the provided `retries` parameter.
- :param never_fail: If true, and poke method raises an exception, sensor will be skipped.
- Mutually exclusive with soft_fail.
+ Mutually exclusive with soft_fail.
+ :param skip_policy:
Review Comment:
Description needs to be added, probably with details about each option in the enum.
On the way, I'd include answers to the following questions:
1. What is a "soft" error?
2. What's the difference between "ignore" and "skip"? (it might not be as trivial for the user)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Refactor `BaseSensorOperator` introduce `skip_policy` parameter [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal merged PR #40924:
URL: https://github.com/apache/airflow/pull/40924
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org