You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "mesaverde0207 (via GitHub)" <gi...@apache.org> on 2023/02/10 08:34:01 UTC
[GitHub] [airflow] mesaverde0207 opened a new issue, #29459: Unexpected error when using Cloud Run jobs command in BashOperator
mesaverde0207 opened a new issue, #29459:
URL: https://github.com/apache/airflow/issues/29459
### Apache Airflow version
2.5.1
### What happened
When I ran the Cloud Run jobs execute command in BashOperator:
```Shell
gcloud beta run jobs execute extract-dev-****** --project "products-*****" --region "us-east1" --wait
```
And the task failed, and I got an error whose message said that the task process group received SIGTERM:
- [bashop_cloud_run_jobs_error.log](https://github.com/apache/airflow/files/10705132/error.log)
But It went green when I dropped the "wait" option.
### What you think should happen instead
The error message says, "WARNING - Recorded pid 6032 does not match the current pid 6033"
and so Airflow sent Signals.SIGTERM to group 6033, whose PIDs of all processes in the group: [6036, 6033].
I think this SIGTERM signal makes the BashOperator crash.
### How to reproduce
_No response_
### Operating System
Ubuntu 22.04.1 LTS
### Versions of Apache Airflow Providers
apache-airflow-providers-common-sql==1.3.3
apache-airflow-providers-ftp==3.3.0
apache-airflow-providers-google==8.8.0
apache-airflow-providers-http==4.1.1
apache-airflow-providers-imap==3.1.1
apache-airflow-providers-postgres==5.4.0
apache-airflow-providers-sqlite==3.3.1
### Deployment
Virtualenv installation
### Deployment details
_No response_
### Anything else
There are some issues similar to this problem:
- https://github.com/apache/airflow/issues/17507
- https://github.com/apache/airflow/issues/20992
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mesaverde0207 commented on issue #29459: Unexpected error when using Cloud Run jobs command in BashOperator
Posted by "mesaverde0207 (via GitHub)" <gi...@apache.org>.
mesaverde0207 commented on issue #29459:
URL: https://github.com/apache/airflow/issues/29459#issuecomment-1431534336
Thank you for the comment, and sorry for my late response.
So if the gcloud CLI ended the original process and created a new one in this case, is this the correct behavior of BashOperator?
Should I report the crash problem to the gcloud CLI dev. group rather than Airflow's group?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #29459: Unexpected error when using Cloud Run jobs command in BashOperator
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #29459:
URL: https://github.com/apache/airflow/issues/29459#issuecomment-1425414119
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Taragolis commented on issue #29459: Unexpected error when using Cloud Run jobs command in BashOperator
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on issue #29459:
URL: https://github.com/apache/airflow/issues/29459#issuecomment-1425627663
As I understand that it only happen when you add `--wait` flag, and did failed if not.
I'm not familiar with `gcloud` but if it create new process and close previous than this behaviour of `local_task_job.py` is valid, it detects that process which initially create task run not exists anymore and kill all process group.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal closed issue #29459: Unexpected error when using Cloud Run jobs command in BashOperator
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal closed issue #29459: Unexpected error when using Cloud Run jobs command in BashOperator
URL: https://github.com/apache/airflow/issues/29459
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Taragolis commented on issue #29459: Unexpected error when using Cloud Run jobs command in BashOperator
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on issue #29459:
URL: https://github.com/apache/airflow/issues/29459#issuecomment-1432016262
It is just my assumption about `gcloud` I could be wrong.
The main question are you have any problem with other tasks?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org