You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@airflow.apache.org by "mariotaddeucci (via GitHub)" <gi...@apache.org> on 2025/03/01 15:31:03 UTC

[PR] Handles flow exceptions on EcsRunTaskOperator using provided exit codes [airflow]

mariotaddeucci opened a new pull request, #47256:
URL: https://github.com/apache/airflow/pull/47256

**Title:** Handles flow exceptions on EcsRunTaskOperator using provided exit codes

**Description:**
This PR introduces the ability to raise `AirflowFailException` or `AirflowSkipException` in `EcsRunTaskOperator` based on the exit code of the ECS task.

### **Motivation**
Currently, when an ECS task fails with specific exit codes, such as `137` (Out of Memory), the task may be retried unnecessarily. This can lead to resource wastage and inefficient execution.

With this change, users can define exit codes that should trigger an `AirflowFailException`, immediately failing the task and preventing retries. Additionally, they can specify exit codes that should raise an `AirflowSkipException`, allowing the DAG to skip the task and avoid consuming unnecessary resources.

### **Changes**
- Added support for mapping exit codes to `AirflowFailException` or `AirflowSkipException` in `EcsRunTaskOperator`.
- Introduced new parameters:
- `fail_on_exit_codes`: List of exit codes that trigger an `AirflowFailException`.
- `skip_on_exit_codes`: List of exit codes that trigger an `AirflowSkipException`.
- Updated documentation and tests to cover the new functionality.

### **Example Usage**
```python
EcsRunTaskOperator(
task_id="ecs_task",
cluster="my-cluster",
task_definition="my-task-def",
overrides={},
launch_type="FARGATE",
fail_on_exit_codes=[137],
skip_on_exit_codes=[2]
)
```

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Handles flow exceptions on EcsRunTaskOperator using provided exit codes [airflow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.

github-actions[bot] commented on PR #47256:
URL: https://github.com/apache/airflow/pull/47256#issuecomment-2926087317

   This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Handles flow exceptions on EcsRunTaskOperator using provided exit codes [airflow]

Posted by "mariotaddeucci (via GitHub)" <gi...@apache.org>.

mariotaddeucci commented on PR #47256:
URL: https://github.com/apache/airflow/pull/47256#issuecomment-2714068019

   > Have you done some manual testing in real DAGs that show this is working as you expect?
   
   Yes It Works, in fact Im running this in production. It follows the same aproach of BashOperator but instead of using only one exit code, this passes a list of exist code 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Handles flow exceptions on EcsRunTaskOperator using provided exit codes [airflow]

Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.

o-nikolas commented on code in PR #47256:
URL: https://github.com/apache/airflow/pull/47256#discussion_r1978085731


##########
providers/amazon/src/airflow/providers/amazon/aws/operators/ecs.py:
##########
@@ -393,6 +393,9 @@ class EcsRunTaskOperator(EcsBaseOperator):
     :param deferrable: If True, the operator will wait asynchronously for the job to complete.
         This implies waiting for completion. This mode requires aiobotocore module to be installed.
         (default: False)
+    :param skip_on_exit_codes: List of exit codes which will make the task to be marked as skipped with AirflowSkipException.
+    :param fail_on_exit_codes: List of exit codes which will make the task to be immediately marked as failed with AirflowFailException.
+

Review Comment:
   ```suggestion
       :param skip_on_exit_codes: List of exit codes which will cause the task to be marked as skipped with AirflowSkipException.
       :param fail_on_exit_codes: List of exit codes which will cause the task to be immediately marked as failed with AirflowFailException.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Handles flow exceptions on EcsRunTaskOperator using provided exit codes [airflow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.

github-actions[bot] closed pull request #47256: Handles flow exceptions on EcsRunTaskOperator using provided exit codes
URL: https://github.com/apache/airflow/pull/47256


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Handles flow exceptions on EcsRunTaskOperator using provided exit codes [airflow]

Posted by "eladkal (via GitHub)" <gi...@apache.org>.

eladkal commented on PR #47256:
URL: https://github.com/apache/airflow/pull/47256#issuecomment-2810355701

   @mariotaddeucci can you rebase and resolve conflicts?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org