You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/08/10 10:34:41 UTC
[GitHub] [airflow] ukaaa opened a new issue, #25635: AzureBatchOperator not handling exit code correctly
ukaaa opened a new issue, #25635:
URL: https://github.com/apache/airflow/issues/25635
### Apache Airflow Provider(s)
microsoft-azure
### Versions of Apache Airflow Providers
[apache-airflow-providers-microsoft-azure 3.9.0](https://airflow.apache.org/docs/apache-airflow-providers-microsoft-azure/3.9.0/)
### Apache Airflow version
v2.3.2
### Operating System
Debian GNU/Linux 11 (bullseye)
### Deployment
Other 3rd-party Helm chart
### Deployment details
_No response_
### What happened
I have a task in my Airflow DAG that uses AzureBatchOperator.
As `batch_task_command_line` we pass something like `/bin/bash -c "some-script.sh"`.
The Azure Batch task correctly executes this command, and runs `some-script.sh`. All good.
When `some-script.sh` exits with a non-zero exit code, the Azure Batch task is correctly marked as failed (in Azure Portal), as is the job containing the task. However, in Airflow, the AzureBatchOperator task _always_ shows up as succeeded, ignoring the underlying Azure Batch job or task status.
It even shows in the Airflow DAG logs. Below are the logs of a run with the shell script returning a non-zero exit code. Airflow still considers the task to be a SUCCESS.
```sh
[2022-08-10, 10:01:27 UTC] {batch.py:362} INFO - Waiting for {hidden} to complete, currently on running state
[2022-08-10, 10:01:42 UTC] {taskinstance.py:1395} INFO - Marking task as SUCCESS. dag_id={hidden}, task_id={hidden}, execution_date=20220809T141257, start_date=20220810T100024, end_date=20220810T100142
[2022-08-10, 10:01:42 UTC] {local_task_job.py:156} INFO - Task exited with return code 0
```
The `some-script.sh` contains the following at the top, so that can't be the issue I think.
```bash
#!/bin/bash
set -euo pipefail
```
I tried passing `set -e` to the `batch_task_command_line`, so `set -e; /bin/bash -c "some-script.sh"` but that doesn't work, it gives me a `CommandProgramNotFound` exception.
### What you think should happen instead
When using AzureBatchOperator, I want this task in Airflow to fail when the command line that is passed to AzureBatchOperator exits with a non-zero exit code.
### How to reproduce
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #25635: AzureBatchOperator not handling exit code correctly
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #25635: AzureBatchOperator not handling exit code correctly
URL: https://github.com/apache/airflow/issues/25635
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #25635: AzureBatchOperator not handling exit code correctly
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #25635:
URL: https://github.com/apache/airflow/issues/25635#issuecomment-1210485909
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org