You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "utkarsharma2 (via GitHub)" <gi...@apache.org> on 2023/09/27 17:35:28 UTC

[GitHub] [airflow] utkarsharma2 opened a new pull request, #34652: Respect 'soft_fail' argument when running ExternalTaskSensor

utkarsharma2 opened a new pull request, #34652:
URL: https://github.com/apache/airflow/pull/34652

   This PR intends to solve the issue that the soft_fail argument in ExternalTaskSensor is not respected.
   
   
   Also closes: https://github.com/apache/airflow/issues/34497


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] Lee-W commented on a diff in pull request #34652: Add testcases to ensure that 'soft_fail' argument is respected when running ExternalTaskSensor

Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on code in PR #34652:
URL: https://github.com/apache/airflow/pull/34652#discussion_r1340227292


##########
tests/sensors/test_external_task_sensor.py:
##########
@@ -838,6 +839,172 @@ def test_external_task_group_when_there_is_no_TIs(self):
                 ignore_ti_state=True,
             )
 
+    @pytest.mark.parametrize(
+        "soft_fail, expected_exception, kwargs, expected_message",
+        (
+            (
+                False,
+                AirflowException,
+                {
+                    "external_task_ids": [TEST_TASK_ID, TEST_TASK_ID_ALTERNATE],
+                    "failed_states": [State.FAILED],
+                },
+                f"Some of the external tasks {re.escape(str([TEST_TASK_ID, TEST_TASK_ID_ALTERNATE]))}"
+                f" in DAG {TEST_DAG_ID} failed.",
+            ),
+            (
+                False,
+                AirflowException,
+                {
+                    "external_task_group_id": [TEST_TASK_ID, TEST_TASK_ID_ALTERNATE],
+                    "failed_states": [State.FAILED],
+                },
+                f"The external task_group '{re.escape(str([TEST_TASK_ID, TEST_TASK_ID_ALTERNATE]))}'"
+                f" in DAG '{TEST_DAG_ID}' failed.",
+            ),
+            (
+                False,
+                AirflowException,
+                {"failed_states": [State.FAILED]},
+                f"The external DAG {TEST_DAG_ID} failed.",
+            ),
+            (
+                True,
+                AirflowSkipException,
+                {
+                    "external_task_ids": [TEST_TASK_ID, TEST_TASK_ID_ALTERNATE],
+                    "failed_states": [State.FAILED],
+                },
+                "Skipping due to soft_fail is set to True.",
+            ),
+            (
+                True,
+                AirflowSkipException,
+                {
+                    "external_task_group_id": [TEST_TASK_ID, TEST_TASK_ID_ALTERNATE],
+                    "failed_states": [State.FAILED],
+                },
+                "Skipping due to soft_fail is set to True.",
+            ),
+            (
+                True,
+                AirflowSkipException,
+                {"failed_states": [State.FAILED]},
+                "Skipping due to soft_fail is set to True.",
+            ),
+        ),
+    )
+    @mock.patch("airflow.sensors.external_task.ExternalTaskSensor.get_count")
+    @mock.patch("airflow.sensors.external_task.ExternalTaskSensor._get_dttm_filter")
+    def test_fail_poke(
+        self, _get_dttm_filter, get_count, soft_fail, expected_exception, kwargs, expected_message

Review Comment:
   ```suggestion
           self, _get_dttm_filter, get_count, soft_fail, expected_exception, kwargs, expected_failed_message
   ```



##########
tests/sensors/test_external_task_sensor.py:
##########
@@ -838,6 +839,172 @@ def test_external_task_group_when_there_is_no_TIs(self):
                 ignore_ti_state=True,
             )
 
+    @pytest.mark.parametrize(
+        "soft_fail, expected_exception, kwargs, expected_message",
+        (
+            (
+                False,
+                AirflowException,
+                {
+                    "external_task_ids": [TEST_TASK_ID, TEST_TASK_ID_ALTERNATE],
+                    "failed_states": [State.FAILED],
+                },
+                f"Some of the external tasks {re.escape(str([TEST_TASK_ID, TEST_TASK_ID_ALTERNATE]))}"
+                f" in DAG {TEST_DAG_ID} failed.",
+            ),
+            (
+                False,
+                AirflowException,
+                {
+                    "external_task_group_id": [TEST_TASK_ID, TEST_TASK_ID_ALTERNATE],
+                    "failed_states": [State.FAILED],
+                },
+                f"The external task_group '{re.escape(str([TEST_TASK_ID, TEST_TASK_ID_ALTERNATE]))}'"
+                f" in DAG '{TEST_DAG_ID}' failed.",
+            ),
+            (
+                False,
+                AirflowException,
+                {"failed_states": [State.FAILED]},
+                f"The external DAG {TEST_DAG_ID} failed.",
+            ),
+            (
+                True,
+                AirflowSkipException,
+                {
+                    "external_task_ids": [TEST_TASK_ID, TEST_TASK_ID_ALTERNATE],
+                    "failed_states": [State.FAILED],
+                },
+                "Skipping due to soft_fail is set to True.",
+            ),
+            (
+                True,
+                AirflowSkipException,
+                {
+                    "external_task_group_id": [TEST_TASK_ID, TEST_TASK_ID_ALTERNATE],
+                    "failed_states": [State.FAILED],
+                },
+                "Skipping due to soft_fail is set to True.",
+            ),
+            (
+                True,
+                AirflowSkipException,
+                {"failed_states": [State.FAILED]},
+                "Skipping due to soft_fail is set to True.",
+            ),
+        ),
+    )
+    @mock.patch("airflow.sensors.external_task.ExternalTaskSensor.get_count")
+    @mock.patch("airflow.sensors.external_task.ExternalTaskSensor._get_dttm_filter")
+    def test_fail_poke(
+        self, _get_dttm_filter, get_count, soft_fail, expected_exception, kwargs, expected_message
+    ):
+        _get_dttm_filter.return_value = []
+        get_count.return_value = 1
+        op = ExternalTaskSensor(
+            task_id="test_external_task_duplicate_task_ids",
+            external_dag_id=TEST_DAG_ID,
+            allowed_states=["success"],
+            dag=self.dag,
+            soft_fail=soft_fail,
+            deferrable=False,
+            **kwargs,
+        )

Review Comment:
   expec
   ```suggestion
           )
           expected_message = "Skipping due to soft_fail is set to True." if soft_fail else expected_failed_message
   ```



##########
tests/sensors/test_external_task_sensor.py:
##########
@@ -838,6 +839,172 @@ def test_external_task_group_when_there_is_no_TIs(self):
                 ignore_ti_state=True,
             )
 
+    @pytest.mark.parametrize(
+        "soft_fail, expected_exception, kwargs, expected_message",
+        (
+            (
+                False,
+                AirflowException,
+                {
+                    "external_task_ids": [TEST_TASK_ID, TEST_TASK_ID_ALTERNATE],
+                    "failed_states": [State.FAILED],
+                },
+                f"Some of the external tasks {re.escape(str([TEST_TASK_ID, TEST_TASK_ID_ALTERNATE]))}"
+                f" in DAG {TEST_DAG_ID} failed.",
+            ),
+            (
+                False,
+                AirflowException,
+                {
+                    "external_task_group_id": [TEST_TASK_ID, TEST_TASK_ID_ALTERNATE],
+                    "failed_states": [State.FAILED],
+                },
+                f"The external task_group '{re.escape(str([TEST_TASK_ID, TEST_TASK_ID_ALTERNATE]))}'"
+                f" in DAG '{TEST_DAG_ID}' failed.",
+            ),
+            (
+                False,
+                AirflowException,
+                {"failed_states": [State.FAILED]},
+                f"The external DAG {TEST_DAG_ID} failed.",
+            ),
+            (
+                True,
+                AirflowSkipException,
+                {
+                    "external_task_ids": [TEST_TASK_ID, TEST_TASK_ID_ALTERNATE],
+                    "failed_states": [State.FAILED],
+                },
+                "Skipping due to soft_fail is set to True.",
+            ),
+            (
+                True,
+                AirflowSkipException,
+                {
+                    "external_task_group_id": [TEST_TASK_ID, TEST_TASK_ID_ALTERNATE],
+                    "failed_states": [State.FAILED],
+                },
+                "Skipping due to soft_fail is set to True.",
+            ),
+            (
+                True,
+                AirflowSkipException,
+                {"failed_states": [State.FAILED]},
+                "Skipping due to soft_fail is set to True.",
+            ),
+        ),
+    )
+    @mock.patch("airflow.sensors.external_task.ExternalTaskSensor.get_count")
+    @mock.patch("airflow.sensors.external_task.ExternalTaskSensor._get_dttm_filter")
+    def test_fail_poke(
+        self, _get_dttm_filter, get_count, soft_fail, expected_exception, kwargs, expected_message
+    ):
+        _get_dttm_filter.return_value = []
+        get_count.return_value = 1
+        op = ExternalTaskSensor(
+            task_id="test_external_task_duplicate_task_ids",
+            external_dag_id=TEST_DAG_ID,
+            allowed_states=["success"],
+            dag=self.dag,
+            soft_fail=soft_fail,
+            deferrable=False,
+            **kwargs,
+        )
+        with pytest.raises(expected_exception, match=expected_message):
+            op.execute(context={})
+
+    @pytest.mark.parametrize(

Review Comment:
   I might try something like suggested above.



##########
tests/sensors/test_external_task_sensor.py:
##########
@@ -838,6 +839,172 @@ def test_external_task_group_when_there_is_no_TIs(self):
                 ignore_ti_state=True,
             )
 
+    @pytest.mark.parametrize(
+        "soft_fail, expected_exception, kwargs, expected_message",
+        (
+            (
+                False,
+                AirflowException,
+                {
+                    "external_task_ids": [TEST_TASK_ID, TEST_TASK_ID_ALTERNATE],
+                    "failed_states": [State.FAILED],
+                },
+                f"Some of the external tasks {re.escape(str([TEST_TASK_ID, TEST_TASK_ID_ALTERNATE]))}"
+                f" in DAG {TEST_DAG_ID} failed.",
+            ),
+            (
+                False,
+                AirflowException,
+                {
+                    "external_task_group_id": [TEST_TASK_ID, TEST_TASK_ID_ALTERNATE],
+                    "failed_states": [State.FAILED],
+                },
+                f"The external task_group '{re.escape(str([TEST_TASK_ID, TEST_TASK_ID_ALTERNATE]))}'"
+                f" in DAG '{TEST_DAG_ID}' failed.",
+            ),
+            (
+                False,
+                AirflowException,
+                {"failed_states": [State.FAILED]},
+                f"The external DAG {TEST_DAG_ID} failed.",
+            ),
+            (
+                True,
+                AirflowSkipException,
+                {
+                    "external_task_ids": [TEST_TASK_ID, TEST_TASK_ID_ALTERNATE],
+                    "failed_states": [State.FAILED],
+                },
+                "Skipping due to soft_fail is set to True.",
+            ),
+            (
+                True,
+                AirflowSkipException,
+                {
+                    "external_task_group_id": [TEST_TASK_ID, TEST_TASK_ID_ALTERNATE],
+                    "failed_states": [State.FAILED],
+                },
+                "Skipping due to soft_fail is set to True.",
+            ),
+            (
+                True,
+                AirflowSkipException,
+                {"failed_states": [State.FAILED]},
+                "Skipping due to soft_fail is set to True.",
+            ),
+        ),
+    )

Review Comment:
   ```suggestion
       @pytest.mark.parametrize(
           "soft_fail, expected_exception",
           (
               (False, AirflowException),
               (True, AirflowSkipException)
           )
       )
       @pytest.mark.parametrize(
           "soft_fail, expected_exception, kwargs, expected_failed_message",
           (
               (
                   {
                       "external_task_ids": [TEST_TASK_ID, TEST_TASK_ID_ALTERNATE],
                       "failed_states": [State.FAILED],
                   },
                   f"Some of the external tasks {re.escape(str([TEST_TASK_ID, TEST_TASK_ID_ALTERNATE]))}"
                   f" in DAG {TEST_DAG_ID} failed.",
               ),
               (
                   {
                       "external_task_group_id": [TEST_TASK_ID, TEST_TASK_ID_ALTERNATE],
                       "failed_states": [State.FAILED],
                   },
                   f"The external task_group '{re.escape(str([TEST_TASK_ID, TEST_TASK_ID_ALTERNATE]))}'"
                   f" in DAG '{TEST_DAG_ID}' failed.",
               ),
               (
                   {"failed_states": [State.FAILED]},
                   f"The external DAG {TEST_DAG_ID} failed.",
               ),
           ),
       )
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] utkarsharma2 commented on a diff in pull request #34652: Fix `dag.test()` and Respect 'soft_fail' argument when running ExternalTaskSensor

Posted by "utkarsharma2 (via GitHub)" <gi...@apache.org>.
utkarsharma2 commented on code in PR #34652:
URL: https://github.com/apache/airflow/pull/34652#discussion_r1339576279


##########
airflow/sensors/external_task.py:
##########
@@ -163,9 +163,11 @@ def __init__(
         total_states = set(self.allowed_states + self.skipped_states + self.failed_states)
 
         if len(total_states) != len(self.allowed_states) + len(self.skipped_states) + len(self.failed_states):
-            raise AirflowException(
-                "Duplicate values provided across allowed_states, skipped_states and failed_states."
-            )
+            # TODO: remove this if block when min_airflow_version is set to higher than 2.7.1
+            message = "Duplicate values provided across allowed_states, skipped_states and failed_states."
+            if self.soft_fail:
+                raise AirflowSkipException(message)
+            raise AirflowException(message)

Review Comment:
   @hussein-awala Right, I'll change it. Thanks for pointing it out. :)
   
   Just wanted to confirm my understanding, if there is a problem in the `__init__` it will be caught in the parsing phase and because of that, we don't need the `AirflowSkipException` exception, right?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] utkarsharma2 commented on a diff in pull request #34652: Fix `dag.test()` and Respect 'soft_fail' argument when running ExternalTaskSensor

Posted by "utkarsharma2 (via GitHub)" <gi...@apache.org>.
utkarsharma2 commented on code in PR #34652:
URL: https://github.com/apache/airflow/pull/34652#discussion_r1339594679


##########
airflow/sensors/external_task.py:
##########
@@ -163,9 +163,11 @@ def __init__(
         total_states = set(self.allowed_states + self.skipped_states + self.failed_states)
 
         if len(total_states) != len(self.allowed_states) + len(self.skipped_states) + len(self.failed_states):
-            raise AirflowException(
-                "Duplicate values provided across allowed_states, skipped_states and failed_states."
-            )
+            # TODO: remove this if block when min_airflow_version is set to higher than 2.7.1
+            message = "Duplicate values provided across allowed_states, skipped_states and failed_states."
+            if self.soft_fail:
+                raise AirflowSkipException(message)
+            raise AirflowException(message)

Review Comment:
   Thanks, @Lee-W I'll remove the ToDo.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] Lee-W commented on a diff in pull request #34652: Fix `dag.test()` and Respect 'soft_fail' argument when running ExternalTaskSensor

Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on code in PR #34652:
URL: https://github.com/apache/airflow/pull/34652#discussion_r1339583073


##########
airflow/sensors/external_task.py:
##########
@@ -163,9 +163,11 @@ def __init__(
         total_states = set(self.allowed_states + self.skipped_states + self.failed_states)
 
         if len(total_states) != len(self.allowed_states) + len(self.skipped_states) + len(self.failed_states):
-            raise AirflowException(
-                "Duplicate values provided across allowed_states, skipped_states and failed_states."
-            )
+            # TODO: remove this if block when min_airflow_version is set to higher than 2.7.1
+            message = "Duplicate values provided across allowed_states, skipped_states and failed_states."
+            if self.soft_fail:
+                raise AirflowSkipException(message)
+            raise AirflowException(message)

Review Comment:
   In addition to that, I think https://github.com/apache/airflow/pull/33424 also handles it and that's why we have the TODO in providers but not in the core.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] utkarsharma2 commented on a diff in pull request #34652: Add testcases to ensure that 'soft_fail' argument is respected when running ExternalTaskSensor

Posted by "utkarsharma2 (via GitHub)" <gi...@apache.org>.
utkarsharma2 commented on code in PR #34652:
URL: https://github.com/apache/airflow/pull/34652#discussion_r1340950032


##########
tests/sensors/test_external_task_sensor.py:
##########
@@ -838,6 +839,172 @@ def test_external_task_group_when_there_is_no_TIs(self):
                 ignore_ti_state=True,
             )
 
+    @pytest.mark.parametrize(
+        "soft_fail, expected_exception, kwargs, expected_message",
+        (
+            (
+                False,
+                AirflowException,
+                {
+                    "external_task_ids": [TEST_TASK_ID, TEST_TASK_ID_ALTERNATE],
+                    "failed_states": [State.FAILED],
+                },
+                f"Some of the external tasks {re.escape(str([TEST_TASK_ID, TEST_TASK_ID_ALTERNATE]))}"
+                f" in DAG {TEST_DAG_ID} failed.",
+            ),
+            (
+                False,
+                AirflowException,
+                {
+                    "external_task_group_id": [TEST_TASK_ID, TEST_TASK_ID_ALTERNATE],
+                    "failed_states": [State.FAILED],
+                },
+                f"The external task_group '{re.escape(str([TEST_TASK_ID, TEST_TASK_ID_ALTERNATE]))}'"
+                f" in DAG '{TEST_DAG_ID}' failed.",
+            ),
+            (
+                False,
+                AirflowException,
+                {"failed_states": [State.FAILED]},
+                f"The external DAG {TEST_DAG_ID} failed.",
+            ),
+            (
+                True,
+                AirflowSkipException,
+                {
+                    "external_task_ids": [TEST_TASK_ID, TEST_TASK_ID_ALTERNATE],
+                    "failed_states": [State.FAILED],
+                },
+                "Skipping due to soft_fail is set to True.",
+            ),
+            (
+                True,
+                AirflowSkipException,
+                {
+                    "external_task_group_id": [TEST_TASK_ID, TEST_TASK_ID_ALTERNATE],
+                    "failed_states": [State.FAILED],
+                },
+                "Skipping due to soft_fail is set to True.",
+            ),
+            (
+                True,
+                AirflowSkipException,
+                {"failed_states": [State.FAILED]},
+                "Skipping due to soft_fail is set to True.",
+            ),
+        ),
+    )
+    @mock.patch("airflow.sensors.external_task.ExternalTaskSensor.get_count")
+    @mock.patch("airflow.sensors.external_task.ExternalTaskSensor._get_dttm_filter")
+    def test_fail_poke(
+        self, _get_dttm_filter, get_count, soft_fail, expected_exception, kwargs, expected_message
+    ):
+        _get_dttm_filter.return_value = []
+        get_count.return_value = 1
+        op = ExternalTaskSensor(
+            task_id="test_external_task_duplicate_task_ids",
+            external_dag_id=TEST_DAG_ID,
+            allowed_states=["success"],
+            dag=self.dag,
+            soft_fail=soft_fail,
+            deferrable=False,
+            **kwargs,
+        )
+        with pytest.raises(expected_exception, match=expected_message):
+            op.execute(context={})
+
+    @pytest.mark.parametrize(

Review Comment:
   Oh yeah, yup, that works. Now I'm a little concerned that it might be overly complicated. Nevertheless, I have added the change to make the testcase compact and elegant. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] utkarsharma2 commented on a diff in pull request #34652: Fix `dag.test()` and Respect 'soft_fail' argument when running ExternalTaskSensor

Posted by "utkarsharma2 (via GitHub)" <gi...@apache.org>.
utkarsharma2 commented on code in PR #34652:
URL: https://github.com/apache/airflow/pull/34652#discussion_r1339576279


##########
airflow/sensors/external_task.py:
##########
@@ -163,9 +163,11 @@ def __init__(
         total_states = set(self.allowed_states + self.skipped_states + self.failed_states)
 
         if len(total_states) != len(self.allowed_states) + len(self.skipped_states) + len(self.failed_states):
-            raise AirflowException(
-                "Duplicate values provided across allowed_states, skipped_states and failed_states."
-            )
+            # TODO: remove this if block when min_airflow_version is set to higher than 2.7.1
+            message = "Duplicate values provided across allowed_states, skipped_states and failed_states."
+            if self.soft_fail:
+                raise AirflowSkipException(message)
+            raise AirflowException(message)

Review Comment:
   @hussein-awala Right, I'll change it. Thanks for pointing it out. :)
   
   Just wanted to confirm my understanding, if there is a problem in the `__init__` it will be caught in the parsing phase and because of that, we don't need to raise the `AirflowSkipException` exception in __init__, right?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] Lee-W commented on a diff in pull request #34652: Fix `dag.test()` and Respect 'soft_fail' argument when running ExternalTaskSensor

Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on code in PR #34652:
URL: https://github.com/apache/airflow/pull/34652#discussion_r1339602913


##########
airflow/sensors/external_task.py:
##########
@@ -163,9 +163,11 @@ def __init__(
         total_states = set(self.allowed_states + self.skipped_states + self.failed_states)
 
         if len(total_states) != len(self.allowed_states) + len(self.skipped_states) + len(self.failed_states):
-            raise AirflowException(
-                "Duplicate values provided across allowed_states, skipped_states and failed_states."
-            )
+            # TODO: remove this if block when min_airflow_version is set to higher than 2.7.1
+            message = "Duplicate values provided across allowed_states, skipped_states and failed_states."
+            if self.soft_fail:
+                raise AirflowSkipException(message)
+            raise AirflowException(message)

Review Comment:
   https://github.com/apache/airflow/blob/19450e03f534f63399bf5db2df7690fdd47b09c8/airflow/sensors/base.py#L242-L259
   
   it's now handled in the base sensor



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] utkarsharma2 commented on a diff in pull request #34652: Fix `dag.test()` and Respect 'soft_fail' argument when running ExternalTaskSensor

Posted by "utkarsharma2 (via GitHub)" <gi...@apache.org>.
utkarsharma2 commented on code in PR #34652:
URL: https://github.com/apache/airflow/pull/34652#discussion_r1339651243


##########
airflow/sensors/external_task.py:
##########
@@ -163,9 +163,11 @@ def __init__(
         total_states = set(self.allowed_states + self.skipped_states + self.failed_states)
 
         if len(total_states) != len(self.allowed_states) + len(self.skipped_states) + len(self.failed_states):
-            raise AirflowException(
-                "Duplicate values provided across allowed_states, skipped_states and failed_states."
-            )
+            # TODO: remove this if block when min_airflow_version is set to higher than 2.7.1
+            message = "Duplicate values provided across allowed_states, skipped_states and failed_states."
+            if self.soft_fail:
+                raise AirflowSkipException(message)
+            raise AirflowException(message)

Review Comment:
   Thanks for clarification,  I'll update the PR and will only retain the meaningful tests.



##########
airflow/sensors/external_task.py:
##########
@@ -163,9 +163,11 @@ def __init__(
         total_states = set(self.allowed_states + self.skipped_states + self.failed_states)
 
         if len(total_states) != len(self.allowed_states) + len(self.skipped_states) + len(self.failed_states):
-            raise AirflowException(
-                "Duplicate values provided across allowed_states, skipped_states and failed_states."
-            )
+            # TODO: remove this if block when min_airflow_version is set to higher than 2.7.1
+            message = "Duplicate values provided across allowed_states, skipped_states and failed_states."
+            if self.soft_fail:
+                raise AirflowSkipException(message)
+            raise AirflowException(message)

Review Comment:
   Thanks for the clarification,  I'll update the PR and will only retain the meaningful tests.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] hussein-awala merged pull request #34652: Add testcases to ensure that 'soft_fail' argument is respected when running ExternalTaskSensor

Posted by "hussein-awala (via GitHub)" <gi...@apache.org>.
hussein-awala merged PR #34652:
URL: https://github.com/apache/airflow/pull/34652


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] hussein-awala commented on a diff in pull request #34652: Fix `dag.test()` and Respect 'soft_fail' argument when running ExternalTaskSensor

Posted by "hussein-awala (via GitHub)" <gi...@apache.org>.
hussein-awala commented on code in PR #34652:
URL: https://github.com/apache/airflow/pull/34652#discussion_r1339084563


##########
airflow/sensors/external_task.py:
##########
@@ -163,9 +163,11 @@ def __init__(
         total_states = set(self.allowed_states + self.skipped_states + self.failed_states)
 
         if len(total_states) != len(self.allowed_states) + len(self.skipped_states) + len(self.failed_states):
-            raise AirflowException(
-                "Duplicate values provided across allowed_states, skipped_states and failed_states."
-            )
+            # TODO: remove this if block when min_airflow_version is set to higher than 2.7.1
+            message = "Duplicate values provided across allowed_states, skipped_states and failed_states."
+            if self.soft_fail:
+                raise AirflowSkipException(message)
+            raise AirflowException(message)

Review Comment:
   If there is a problem in the init, it will never arrive to an execution state, so don't need it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] utkarsharma2 commented on a diff in pull request #34652: Fix `dag.test()` and Respect 'soft_fail' argument when running ExternalTaskSensor

Posted by "utkarsharma2 (via GitHub)" <gi...@apache.org>.
utkarsharma2 commented on code in PR #34652:
URL: https://github.com/apache/airflow/pull/34652#discussion_r1339572230


##########
airflow/sensors/external_task.py:
##########
@@ -163,9 +163,11 @@ def __init__(
         total_states = set(self.allowed_states + self.skipped_states + self.failed_states)
 
         if len(total_states) != len(self.allowed_states) + len(self.skipped_states) + len(self.failed_states):
-            raise AirflowException(
-                "Duplicate values provided across allowed_states, skipped_states and failed_states."
-            )
+            # TODO: remove this if block when min_airflow_version is set to higher than 2.7.1
+            message = "Duplicate values provided across allowed_states, skipped_states and failed_states."
+            if self.soft_fail:
+                raise AirflowSkipException(message)
+            raise AirflowException(message)

Review Comment:
   Right, I'll change it. Thanks :)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] utkarsharma2 commented on a diff in pull request #34652: Fix `dag.test()` and Respect 'soft_fail' argument when running ExternalTaskSensor

Posted by "utkarsharma2 (via GitHub)" <gi...@apache.org>.
utkarsharma2 commented on code in PR #34652:
URL: https://github.com/apache/airflow/pull/34652#discussion_r1339572230


##########
airflow/sensors/external_task.py:
##########
@@ -163,9 +163,11 @@ def __init__(
         total_states = set(self.allowed_states + self.skipped_states + self.failed_states)
 
         if len(total_states) != len(self.allowed_states) + len(self.skipped_states) + len(self.failed_states):
-            raise AirflowException(
-                "Duplicate values provided across allowed_states, skipped_states and failed_states."
-            )
+            # TODO: remove this if block when min_airflow_version is set to higher than 2.7.1
+            message = "Duplicate values provided across allowed_states, skipped_states and failed_states."
+            if self.soft_fail:
+                raise AirflowSkipException(message)
+            raise AirflowException(message)

Review Comment:
   Right, I'll change it. Thanks :)
   
   Just to confirm if there is a problem in init it will caught in the dag parsing phase right?



##########
airflow/sensors/external_task.py:
##########
@@ -163,9 +163,11 @@ def __init__(
         total_states = set(self.allowed_states + self.skipped_states + self.failed_states)
 
         if len(total_states) != len(self.allowed_states) + len(self.skipped_states) + len(self.failed_states):
-            raise AirflowException(
-                "Duplicate values provided across allowed_states, skipped_states and failed_states."
-            )
+            # TODO: remove this if block when min_airflow_version is set to higher than 2.7.1
+            message = "Duplicate values provided across allowed_states, skipped_states and failed_states."
+            if self.soft_fail:
+                raise AirflowSkipException(message)
+            raise AirflowException(message)

Review Comment:
   Right, I'll change it. Thanks :)
   
   Just to confirm if there is a problem in init it will caught in the dag parsing phase right?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] utkarsharma2 commented on a diff in pull request #34652: Add testcases to ensure that 'soft_fail' argument is respected when running ExternalTaskSensor

Posted by "utkarsharma2 (via GitHub)" <gi...@apache.org>.
utkarsharma2 commented on code in PR #34652:
URL: https://github.com/apache/airflow/pull/34652#discussion_r1340461496


##########
tests/sensors/test_external_task_sensor.py:
##########
@@ -838,6 +839,172 @@ def test_external_task_group_when_there_is_no_TIs(self):
                 ignore_ti_state=True,
             )
 
+    @pytest.mark.parametrize(
+        "soft_fail, expected_exception, kwargs, expected_message",
+        (
+            (
+                False,
+                AirflowException,
+                {
+                    "external_task_ids": [TEST_TASK_ID, TEST_TASK_ID_ALTERNATE],
+                    "failed_states": [State.FAILED],
+                },
+                f"Some of the external tasks {re.escape(str([TEST_TASK_ID, TEST_TASK_ID_ALTERNATE]))}"
+                f" in DAG {TEST_DAG_ID} failed.",
+            ),
+            (
+                False,
+                AirflowException,
+                {
+                    "external_task_group_id": [TEST_TASK_ID, TEST_TASK_ID_ALTERNATE],
+                    "failed_states": [State.FAILED],
+                },
+                f"The external task_group '{re.escape(str([TEST_TASK_ID, TEST_TASK_ID_ALTERNATE]))}'"
+                f" in DAG '{TEST_DAG_ID}' failed.",
+            ),
+            (
+                False,
+                AirflowException,
+                {"failed_states": [State.FAILED]},
+                f"The external DAG {TEST_DAG_ID} failed.",
+            ),
+            (
+                True,
+                AirflowSkipException,
+                {
+                    "external_task_ids": [TEST_TASK_ID, TEST_TASK_ID_ALTERNATE],
+                    "failed_states": [State.FAILED],
+                },
+                "Skipping due to soft_fail is set to True.",
+            ),
+            (
+                True,
+                AirflowSkipException,
+                {
+                    "external_task_group_id": [TEST_TASK_ID, TEST_TASK_ID_ALTERNATE],
+                    "failed_states": [State.FAILED],
+                },
+                "Skipping due to soft_fail is set to True.",
+            ),
+            (
+                True,
+                AirflowSkipException,
+                {"failed_states": [State.FAILED]},
+                "Skipping due to soft_fail is set to True.",
+            ),
+        ),
+    )

Review Comment:
   @Lee-W I have added your changes in PR. Please take a look 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] utkarsharma2 commented on a diff in pull request #34652: Add testcases to ensure that 'soft_fail' argument is respected when running ExternalTaskSensor

Posted by "utkarsharma2 (via GitHub)" <gi...@apache.org>.
utkarsharma2 commented on code in PR #34652:
URL: https://github.com/apache/airflow/pull/34652#discussion_r1340950032


##########
tests/sensors/test_external_task_sensor.py:
##########
@@ -838,6 +839,172 @@ def test_external_task_group_when_there_is_no_TIs(self):
                 ignore_ti_state=True,
             )
 
+    @pytest.mark.parametrize(
+        "soft_fail, expected_exception, kwargs, expected_message",
+        (
+            (
+                False,
+                AirflowException,
+                {
+                    "external_task_ids": [TEST_TASK_ID, TEST_TASK_ID_ALTERNATE],
+                    "failed_states": [State.FAILED],
+                },
+                f"Some of the external tasks {re.escape(str([TEST_TASK_ID, TEST_TASK_ID_ALTERNATE]))}"
+                f" in DAG {TEST_DAG_ID} failed.",
+            ),
+            (
+                False,
+                AirflowException,
+                {
+                    "external_task_group_id": [TEST_TASK_ID, TEST_TASK_ID_ALTERNATE],
+                    "failed_states": [State.FAILED],
+                },
+                f"The external task_group '{re.escape(str([TEST_TASK_ID, TEST_TASK_ID_ALTERNATE]))}'"
+                f" in DAG '{TEST_DAG_ID}' failed.",
+            ),
+            (
+                False,
+                AirflowException,
+                {"failed_states": [State.FAILED]},
+                f"The external DAG {TEST_DAG_ID} failed.",
+            ),
+            (
+                True,
+                AirflowSkipException,
+                {
+                    "external_task_ids": [TEST_TASK_ID, TEST_TASK_ID_ALTERNATE],
+                    "failed_states": [State.FAILED],
+                },
+                "Skipping due to soft_fail is set to True.",
+            ),
+            (
+                True,
+                AirflowSkipException,
+                {
+                    "external_task_group_id": [TEST_TASK_ID, TEST_TASK_ID_ALTERNATE],
+                    "failed_states": [State.FAILED],
+                },
+                "Skipping due to soft_fail is set to True.",
+            ),
+            (
+                True,
+                AirflowSkipException,
+                {"failed_states": [State.FAILED]},
+                "Skipping due to soft_fail is set to True.",
+            ),
+        ),
+    )
+    @mock.patch("airflow.sensors.external_task.ExternalTaskSensor.get_count")
+    @mock.patch("airflow.sensors.external_task.ExternalTaskSensor._get_dttm_filter")
+    def test_fail_poke(
+        self, _get_dttm_filter, get_count, soft_fail, expected_exception, kwargs, expected_message
+    ):
+        _get_dttm_filter.return_value = []
+        get_count.return_value = 1
+        op = ExternalTaskSensor(
+            task_id="test_external_task_duplicate_task_ids",
+            external_dag_id=TEST_DAG_ID,
+            allowed_states=["success"],
+            dag=self.dag,
+            soft_fail=soft_fail,
+            deferrable=False,
+            **kwargs,
+        )
+        with pytest.raises(expected_exception, match=expected_message):
+            op.execute(context={})
+
+    @pytest.mark.parametrize(

Review Comment:
   Oh yeah, yup, that works, thanks. Now I'm a little concerned that it might be overly complicated. Nevertheless, I have added the change to make the testcase compact and elegant. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] Lee-W commented on a diff in pull request #34652: Fix `dag.test()` and Respect 'soft_fail' argument when running ExternalTaskSensor

Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on code in PR #34652:
URL: https://github.com/apache/airflow/pull/34652#discussion_r1339602707


##########
airflow/sensors/external_task.py:
##########
@@ -163,9 +163,11 @@ def __init__(
         total_states = set(self.allowed_states + self.skipped_states + self.failed_states)
 
         if len(total_states) != len(self.allowed_states) + len(self.skipped_states) + len(self.failed_states):
-            raise AirflowException(
-                "Duplicate values provided across allowed_states, skipped_states and failed_states."
-            )
+            # TODO: remove this if block when min_airflow_version is set to higher than 2.7.1
+            message = "Duplicate values provided across allowed_states, skipped_states and failed_states."
+            if self.soft_fail:
+                raise AirflowSkipException(message)
+            raise AirflowException(message)

Review Comment:
   Sorry, what I meant was the whole section is not needed instead of just the TODO



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] utkarsharma2 commented on a diff in pull request #34652: Add testcases to ensure that 'soft_fail' argument is respected when running ExternalTaskSensor

Posted by "utkarsharma2 (via GitHub)" <gi...@apache.org>.
utkarsharma2 commented on code in PR #34652:
URL: https://github.com/apache/airflow/pull/34652#discussion_r1340465287


##########
tests/sensors/test_external_task_sensor.py:
##########
@@ -838,6 +839,172 @@ def test_external_task_group_when_there_is_no_TIs(self):
                 ignore_ti_state=True,
             )
 
+    @pytest.mark.parametrize(
+        "soft_fail, expected_exception, kwargs, expected_message",
+        (
+            (
+                False,
+                AirflowException,
+                {
+                    "external_task_ids": [TEST_TASK_ID, TEST_TASK_ID_ALTERNATE],
+                    "failed_states": [State.FAILED],
+                },
+                f"Some of the external tasks {re.escape(str([TEST_TASK_ID, TEST_TASK_ID_ALTERNATE]))}"
+                f" in DAG {TEST_DAG_ID} failed.",
+            ),
+            (
+                False,
+                AirflowException,
+                {
+                    "external_task_group_id": [TEST_TASK_ID, TEST_TASK_ID_ALTERNATE],
+                    "failed_states": [State.FAILED],
+                },
+                f"The external task_group '{re.escape(str([TEST_TASK_ID, TEST_TASK_ID_ALTERNATE]))}'"
+                f" in DAG '{TEST_DAG_ID}' failed.",
+            ),
+            (
+                False,
+                AirflowException,
+                {"failed_states": [State.FAILED]},
+                f"The external DAG {TEST_DAG_ID} failed.",
+            ),
+            (
+                True,
+                AirflowSkipException,
+                {
+                    "external_task_ids": [TEST_TASK_ID, TEST_TASK_ID_ALTERNATE],
+                    "failed_states": [State.FAILED],
+                },
+                "Skipping due to soft_fail is set to True.",
+            ),
+            (
+                True,
+                AirflowSkipException,
+                {
+                    "external_task_group_id": [TEST_TASK_ID, TEST_TASK_ID_ALTERNATE],
+                    "failed_states": [State.FAILED],
+                },
+                "Skipping due to soft_fail is set to True.",
+            ),
+            (
+                True,
+                AirflowSkipException,
+                {"failed_states": [State.FAILED]},
+                "Skipping due to soft_fail is set to True.",
+            ),
+        ),
+    )
+    @mock.patch("airflow.sensors.external_task.ExternalTaskSensor.get_count")
+    @mock.patch("airflow.sensors.external_task.ExternalTaskSensor._get_dttm_filter")
+    def test_fail_poke(
+        self, _get_dttm_filter, get_count, soft_fail, expected_exception, kwargs, expected_message
+    ):
+        _get_dttm_filter.return_value = []
+        get_count.return_value = 1
+        op = ExternalTaskSensor(
+            task_id="test_external_task_duplicate_task_ids",
+            external_dag_id=TEST_DAG_ID,
+            allowed_states=["success"],
+            dag=self.dag,
+            soft_fail=soft_fail,
+            deferrable=False,
+            **kwargs,
+        )
+        with pytest.raises(expected_exception, match=expected_message):
+            op.execute(context={})
+
+    @pytest.mark.parametrize(

Review Comment:
   In this testcase, it might be a little difficult because the error message is coming from a location. tightly coupling `soft_fail`, `AirflowSkipException`, `kwargs` and `expected_message`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] utkarsharma2 commented on pull request #34652: Fix `dag.test()` and Respect 'soft_fail' argument when running ExternalTaskSensor

Posted by "utkarsharma2 (via GitHub)" <gi...@apache.org>.
utkarsharma2 commented on PR #34652:
URL: https://github.com/apache/airflow/pull/34652#issuecomment-1738537373

   > No need to add this patch in the core operators, we already fixed it, and for that we add the comment (TODO: remove this if block when min_airflow_version is set to higher than 2.7.1)
   > 
   > However, you can keep the tests (only the tests for the execute/patch method).
   
   @hussein-awala I'm assuming you mean the entire code snippet when you said `this patch`
   ```
   # TODO: remove this if block when min_airflow_version is set to higher than 2.7.1
   message = ""
   if self.soft_fail:
       raise AirflowSkipException(message)
   raise AirflowException(message)
   ```
   if so, `_check_for_existence(self, session)` was not respecting the `soft_fail` param and causing the `dag.test()` to halt the flow because `AirflowException` was raised. 
   
   
    


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] utkarsharma2 commented on a diff in pull request #34652: Add testcases to ensure that 'soft_fail' argument is respected when running ExternalTaskSensor

Posted by "utkarsharma2 (via GitHub)" <gi...@apache.org>.
utkarsharma2 commented on code in PR #34652:
URL: https://github.com/apache/airflow/pull/34652#discussion_r1340465287


##########
tests/sensors/test_external_task_sensor.py:
##########
@@ -838,6 +839,172 @@ def test_external_task_group_when_there_is_no_TIs(self):
                 ignore_ti_state=True,
             )
 
+    @pytest.mark.parametrize(
+        "soft_fail, expected_exception, kwargs, expected_message",
+        (
+            (
+                False,
+                AirflowException,
+                {
+                    "external_task_ids": [TEST_TASK_ID, TEST_TASK_ID_ALTERNATE],
+                    "failed_states": [State.FAILED],
+                },
+                f"Some of the external tasks {re.escape(str([TEST_TASK_ID, TEST_TASK_ID_ALTERNATE]))}"
+                f" in DAG {TEST_DAG_ID} failed.",
+            ),
+            (
+                False,
+                AirflowException,
+                {
+                    "external_task_group_id": [TEST_TASK_ID, TEST_TASK_ID_ALTERNATE],
+                    "failed_states": [State.FAILED],
+                },
+                f"The external task_group '{re.escape(str([TEST_TASK_ID, TEST_TASK_ID_ALTERNATE]))}'"
+                f" in DAG '{TEST_DAG_ID}' failed.",
+            ),
+            (
+                False,
+                AirflowException,
+                {"failed_states": [State.FAILED]},
+                f"The external DAG {TEST_DAG_ID} failed.",
+            ),
+            (
+                True,
+                AirflowSkipException,
+                {
+                    "external_task_ids": [TEST_TASK_ID, TEST_TASK_ID_ALTERNATE],
+                    "failed_states": [State.FAILED],
+                },
+                "Skipping due to soft_fail is set to True.",
+            ),
+            (
+                True,
+                AirflowSkipException,
+                {
+                    "external_task_group_id": [TEST_TASK_ID, TEST_TASK_ID_ALTERNATE],
+                    "failed_states": [State.FAILED],
+                },
+                "Skipping due to soft_fail is set to True.",
+            ),
+            (
+                True,
+                AirflowSkipException,
+                {"failed_states": [State.FAILED]},
+                "Skipping due to soft_fail is set to True.",
+            ),
+        ),
+    )
+    @mock.patch("airflow.sensors.external_task.ExternalTaskSensor.get_count")
+    @mock.patch("airflow.sensors.external_task.ExternalTaskSensor._get_dttm_filter")
+    def test_fail_poke(
+        self, _get_dttm_filter, get_count, soft_fail, expected_exception, kwargs, expected_message
+    ):
+        _get_dttm_filter.return_value = []
+        get_count.return_value = 1
+        op = ExternalTaskSensor(
+            task_id="test_external_task_duplicate_task_ids",
+            external_dag_id=TEST_DAG_ID,
+            allowed_states=["success"],
+            dag=self.dag,
+            soft_fail=soft_fail,
+            deferrable=False,
+            **kwargs,
+        )
+        with pytest.raises(expected_exception, match=expected_message):
+            op.execute(context={})
+
+    @pytest.mark.parametrize(

Review Comment:
   In this testcase, it might be a little difficult because the error message is coming from a different location. tightly coupling `soft_fail`, `AirflowSkipException`, `kwargs` and `expected_message`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] Lee-W commented on a diff in pull request #34652: Add testcases to ensure that 'soft_fail' argument is respected when running ExternalTaskSensor

Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on code in PR #34652:
URL: https://github.com/apache/airflow/pull/34652#discussion_r1340828962


##########
tests/sensors/test_external_task_sensor.py:
##########
@@ -838,6 +839,172 @@ def test_external_task_group_when_there_is_no_TIs(self):
                 ignore_ti_state=True,
             )
 
+    @pytest.mark.parametrize(
+        "soft_fail, expected_exception, kwargs, expected_message",
+        (
+            (
+                False,
+                AirflowException,
+                {
+                    "external_task_ids": [TEST_TASK_ID, TEST_TASK_ID_ALTERNATE],
+                    "failed_states": [State.FAILED],
+                },
+                f"Some of the external tasks {re.escape(str([TEST_TASK_ID, TEST_TASK_ID_ALTERNATE]))}"
+                f" in DAG {TEST_DAG_ID} failed.",
+            ),
+            (
+                False,
+                AirflowException,
+                {
+                    "external_task_group_id": [TEST_TASK_ID, TEST_TASK_ID_ALTERNATE],
+                    "failed_states": [State.FAILED],
+                },
+                f"The external task_group '{re.escape(str([TEST_TASK_ID, TEST_TASK_ID_ALTERNATE]))}'"
+                f" in DAG '{TEST_DAG_ID}' failed.",
+            ),
+            (
+                False,
+                AirflowException,
+                {"failed_states": [State.FAILED]},
+                f"The external DAG {TEST_DAG_ID} failed.",
+            ),
+            (
+                True,
+                AirflowSkipException,
+                {
+                    "external_task_ids": [TEST_TASK_ID, TEST_TASK_ID_ALTERNATE],
+                    "failed_states": [State.FAILED],
+                },
+                "Skipping due to soft_fail is set to True.",
+            ),
+            (
+                True,
+                AirflowSkipException,
+                {
+                    "external_task_group_id": [TEST_TASK_ID, TEST_TASK_ID_ALTERNATE],
+                    "failed_states": [State.FAILED],
+                },
+                "Skipping due to soft_fail is set to True.",
+            ),
+            (
+                True,
+                AirflowSkipException,
+                {"failed_states": [State.FAILED]},
+                "Skipping due to soft_fail is set to True.",
+            ),
+        ),
+    )
+    @mock.patch("airflow.sensors.external_task.ExternalTaskSensor.get_count")
+    @mock.patch("airflow.sensors.external_task.ExternalTaskSensor._get_dttm_filter")
+    def test_fail_poke(
+        self, _get_dttm_filter, get_count, soft_fail, expected_exception, kwargs, expected_message
+    ):
+        _get_dttm_filter.return_value = []
+        get_count.return_value = 1
+        op = ExternalTaskSensor(
+            task_id="test_external_task_duplicate_task_ids",
+            external_dag_id=TEST_DAG_ID,
+            allowed_states=["success"],
+            dag=self.dag,
+            soft_fail=soft_fail,
+            deferrable=False,
+            **kwargs,
+        )
+        with pytest.raises(expected_exception, match=expected_message):
+            op.execute(context={})
+
+    @pytest.mark.parametrize(

Review Comment:
   We can add a one-line logic in the test case, like https://github.com/apache/airflow/pull/34652#discussion_r1340228280
   But I'm actually ok with it. Just a minor suggestion :)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org