You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "amoghrajesh (via GitHub)" <gi...@apache.org> on 2024/12/09 11:14:25 UTC

[PR] AIP-72: Handling skipped tasks in task_sdk [airflow]

amoghrajesh opened a new pull request, #44786:
URL: https://github.com/apache/airflow/pull/44786

   <!--
    Licensed to the Apache Software Foundation (ASF) under one
    or more contributor license agreements.  See the NOTICE file
    distributed with this work for additional information
    regarding copyright ownership.  The ASF licenses this file
    to you under the Apache License, Version 2.0 (the
    "License"); you may not use this file except in compliance
    with the License.  You may obtain a copy of the License at
   
      http://www.apache.org/licenses/LICENSE-2.0
   
    Unless required by applicable law or agreed to in writing,
    software distributed under the License is distributed on an
    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    KIND, either express or implied.  See the License for the
    specific language governing permissions and limitations
    under the License.
    -->
   
   <!--
   Thank you for contributing! Please make sure that your code changes
   are covered with tests. And in case of new features or big changes
   remember to adjust the documentation.
   
   Feel free to ping committers for the review!
   
   In case of an existing issue, reference it using one of the following:
   
   closes: #ISSUE
   related: #ISSUE
   
   How to write a good git commit message:
   http://chris.beams.io/posts/git-commit/
   -->
   
   
   related: https://github.com/apache/airflow/issues/44414
   
   We already have support for handling terminal states from the task execution side as well as the task SDK client side. (almost).
   
   This PR extends the task runner's `run` function to handle when an AirflowSkipException is thrown.
   
   
   1. Extended the task runner with the right `msg` when a AirflowSkipException is thrown
   2. Added a test in the `test_handle_requests` test of supervisor to show that no `client` call is made from `handle_requests` but is instead controlled by the `wait` function.
   3. Added an example DAG that throws AirflowSkipException and the task runner test that handles that.
   
   Postman working example:
   1. Create a DAG run:
   <img width="1709" alt="image" src="https://github.com/user-attachments/assets/e23388fb-db2d-446b-bfa5-6da1a5710a82">
   
   2. Login into metadata DB and get the task_id for the task you want to skip
   ![image](https://github.com/user-attachments/assets/7166c3fc-4106-441a-b682-6450a5783e9a)
   
   
   3. Once you have the task_id, send a patch request like this:
   ```
   curl --location --request PATCH 'http://localhost:29091/execution/task-instances/0193ab1b-df6a-72bb-b888-04c541e6bf12/state' \
   --data '{
       "state": "skipped",
       "end_date": "2024-10-31T12:00:00Z"
   }'
   ```
   
   ![image](https://github.com/user-attachments/assets/8e127f65-b7e1-48f4-a909-29d34f09f11c)
   
   4. Check the Airflow UI if the task was marked as "skipped"
   <img width="1552" alt="image" src="https://github.com/user-attachments/assets/5ea09da6-4e99-4ae6-8a1f-8d4231770be0">
   
   
   
   <!-- Please keep an empty line above the dashes. -->
   ---
   **^ Add meaningful description above**
   Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information.
   In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
   
   
   
   <!-- Please keep an empty line above the dashes. -->
   ---
   **^ Add meaningful description above**
   Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information.
   In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] AIP-72: Handling skipped tasks in task_sdk [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #44786:
URL: https://github.com/apache/airflow/pull/44786#discussion_r1875918658


##########
task_sdk/tests/execution_time/test_task_runner.py:
##########
@@ -137,3 +137,27 @@ def test_run_deferred_basic(test_dags_dir: Path, time_machine):
 
         # send_request will only be called when the TaskDeferred exception is raised
         mock_supervisor_comms.send_request.assert_called_once_with(msg=expected_defer_task, log=mock.ANY)
+
+
+def test_run_basic_skipped(test_dags_dir: Path, time_machine):
+    """Test running a basic task that marks itself skipped."""
+    what = StartupDetails(
+        ti=TaskInstance(id=uuid7(), task_id="skip", dag_id="basic_skipped", run_id="c", try_number=1),
+        file=str(test_dags_dir / "basic_skipped.py"),
+        requests_fd=0,
+    )
+
+    ti = parse(what)
+
+    instant = timezone.datetime(2024, 12, 3, 10, 0)
+    time_machine.move_to(instant, tick=False)
+
+    with mock.patch(
+        "airflow.sdk.execution_time.task_runner.SUPERVISOR_COMMS", create=True
+    ) as mock_supervisor_comms:
+        mock_supervisor_comms.send_request = mock.Mock()

Review Comment:
   Handled in latest commit



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] AIP-72: Handling skipped tasks in task_sdk [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh merged PR #44786:
URL: https://github.com/apache/airflow/pull/44786


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] AIP-72: Handling skipped tasks in task_sdk [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #44786:
URL: https://github.com/apache/airflow/pull/44786#discussion_r1875815906


##########
task_sdk/tests/dags/super_basic_skipped.py:
##########


Review Comment:
   Made it basic_* instead. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] AIP-72: Handling skipped tasks in task_sdk [airflow]

Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #44786:
URL: https://github.com/apache/airflow/pull/44786#discussion_r1875807589


##########
task_sdk/tests/dags/super_basic_skipped.py:
##########


Review Comment:
   Let's not keep the "super_basic" prefix unless it does "nothing" at all advanced.
   
   
   I also wonder if it's worth building up the mock/fake capability to define the DAG inline in the test and inject it, for example maybe we could stub out the "parse" function using KGB's `spy_agency` to make it use the DAG we want (I'm just somewhat wary of an ever-growing `tests/dags/` folder, as ti makes it harder to know if those dags are still in use.)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] AIP-72: Handling skipped tasks in task_sdk [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #44786:
URL: https://github.com/apache/airflow/pull/44786#discussion_r1875839110


##########
task_sdk/tests/execution_time/test_task_runner.py:
##########
@@ -137,3 +137,27 @@ def test_run_deferred_basic(test_dags_dir: Path, time_machine):
 
         # send_request will only be called when the TaskDeferred exception is raised
         mock_supervisor_comms.send_request.assert_called_once_with(msg=expected_defer_task, log=mock.ANY)
+
+
+def test_run_basic_skipped(test_dags_dir: Path, time_machine):
+    """Test running a basic task that marks itself skipped."""
+    what = StartupDetails(
+        ti=TaskInstance(id=uuid7(), task_id="skip", dag_id="basic_skipped", run_id="c", try_number=1),
+        file=str(test_dags_dir / "basic_skipped.py"),
+        requests_fd=0,
+    )
+
+    ti = parse(what)
+
+    instant = timezone.datetime(2024, 12, 3, 10, 0)
+    time_machine.move_to(instant, tick=False)
+
+    with mock.patch(
+        "airflow.sdk.execution_time.task_runner.SUPERVISOR_COMMS", create=True
+    ) as mock_supervisor_comms:
+        mock_supervisor_comms.send_request = mock.Mock()

Review Comment:
   Yeah, it should



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] AIP-72: Handling skipped tasks in task_sdk [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #44786:
URL: https://github.com/apache/airflow/pull/44786#issuecomment-2531135686

   This one is pretty straight forward. Merging it


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] AIP-72: Handling skipped tasks in task_sdk [airflow]

Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #44786:
URL: https://github.com/apache/airflow/pull/44786#discussion_r1875821302


##########
task_sdk/tests/dags/super_basic_skipped.py:
##########


Review Comment:
   Yeah sounds good!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] AIP-72: Handling skipped tasks in task_sdk [airflow]

Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #44786:
URL: https://github.com/apache/airflow/pull/44786#discussion_r1875829389


##########
task_sdk/tests/execution_time/test_supervisor.py:
##########
@@ -883,7 +892,8 @@ def test_handle_requests(
         generator.send(msg)
 
         # Verify the correct client method was called
-        mock_client_method.assert_called_once_with(*method_arg)
+        if client_attr_path:

Review Comment:
   How does L884 work when client_attr_path is empty?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] AIP-72: Handling skipped tasks in task_sdk [airflow]

Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #44786:
URL: https://github.com/apache/airflow/pull/44786#discussion_r1875830623


##########
task_sdk/tests/execution_time/test_task_runner.py:
##########
@@ -137,3 +137,27 @@ def test_run_deferred_basic(test_dags_dir: Path, time_machine):
 
         # send_request will only be called when the TaskDeferred exception is raised
         mock_supervisor_comms.send_request.assert_called_once_with(msg=expected_defer_task, log=mock.ANY)
+
+
+def test_run_basic_skipped(test_dags_dir: Path, time_machine):
+    """Test running a basic task that marks itself skipped."""
+    what = StartupDetails(
+        ti=TaskInstance(id=uuid7(), task_id="skip", dag_id="basic_skipped", run_id="c", try_number=1),
+        file=str(test_dags_dir / "basic_skipped.py"),
+        requests_fd=0,
+    )
+
+    ti = parse(what)
+
+    instant = timezone.datetime(2024, 12, 3, 10, 0)
+    time_machine.move_to(instant, tick=False)
+
+    with mock.patch(
+        "airflow.sdk.execution_time.task_runner.SUPERVISOR_COMMS", create=True
+    ) as mock_supervisor_comms:
+        mock_supervisor_comms.send_request = mock.Mock()

Review Comment:
   Nit: I don't think this is needed -- won't mock handle this for us automatically?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] AIP-72: Handling skipped tasks in task_sdk [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #44786:
URL: https://github.com/apache/airflow/pull/44786#discussion_r1875838754


##########
task_sdk/tests/execution_time/test_supervisor.py:
##########
@@ -883,7 +892,8 @@ def test_handle_requests(
         generator.send(msg)
 
         # Verify the correct client method was called
-        mock_client_method.assert_called_once_with(*method_arg)
+        if client_attr_path:

Review Comment:
   It just goes ahead with a blank mock, I let it be that that way as it is only used to define a `mock_response` which we have defined above in parameterize.
   ![image](https://github.com/user-attachments/assets/0747f680-eeb1-4dd9-a1fe-dfd69edefa94)
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] AIP-72: Handling skipped tasks in task_sdk [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #44786:
URL: https://github.com/apache/airflow/pull/44786#discussion_r1875819757


##########
task_sdk/tests/dags/super_basic_skipped.py:
##########


Review Comment:
   @ashb are you OK with me picking the second part of stubbing the parse and related things in a follow up? It isn't directly related to this PR



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org