You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/08/17 08:39:10 UTC
[GitHub] [airflow] varundhussa opened a new issue #10357: DataProcHook cancel method does not execute the REST call
varundhussa opened a new issue #10357:
URL: https://github.com/apache/airflow/issues/10357
**Apache Airflow version**: 1.10.x
**What happened**:
The cancel method of the DataProcHook does not cancel a running DataProc job.
**What you expected to happen**:
The job should get cancelled and return the response JSON
Documentation says: returns A Job json dictionary representing the canceled job: https://github.com/apache/airflow/blob/1.10.11/airflow/contrib/hooks/gcp_dataproc_hook.py#L325
**What do you think went wrong**:
The cancel method in the hook only creates the googleapiclient.http.HttpRequest object and returns it. It should call execute on the object and return the response.
It can be fixed by:
https://github.com/apache/airflow/blob/1.10.11/airflow/contrib/hooks/gcp_dataproc_hook.py#L336
```
return self.get_conn().projects().regions().jobs().cancel(
projectId=project_id,
region=region,
jobId=job_id
).execute(num_retries=self.num_retries)
```
**How to reproduce it**:
Submit a dataproc job and call an Airflow operator to cancel the job. It returns the request object instead of the JSON
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on issue #10357: DataProcHook cancel method does not execute the REST call
Posted by GitBox <gi...@apache.org>.
turbaszek commented on issue #10357:
URL: https://github.com/apache/airflow/issues/10357#issuecomment-674802200
The issue still persists on `v1-10-test` branch:
https://github.com/apache/airflow/blob/f487f2f56b2a4ffa474306633c0c2d808a3e2a11/airflow/contrib/hooks/gcp_dataproc_hook.py#L336-L340
That is the branch we use to add fixes to 1.10.X releases. However, I strongly suggest to use backport packages as this is what will be available in Airflow 2.0
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on issue #10357: DataProcHook cancel method does not execute the REST call
Posted by GitBox <gi...@apache.org>.
turbaszek commented on issue #10357:
URL: https://github.com/apache/airflow/issues/10357#issuecomment-674863946
Should we close this issue?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #10357: DataProcHook cancel method does not execute the REST call
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #10357:
URL: https://github.com/apache/airflow/issues/10357#issuecomment-674744863
Thanks for opening your first issue here! Be sure to follow the issue template!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #10357: DataProcHook cancel method does not execute the REST call
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #10357:
URL: https://github.com/apache/airflow/issues/10357#issuecomment-674840432
@varundhussa - we generally stopped fixing problems in 1.10 operators and we fully move our efforts to backport packages. We are going to release the second wave of backport packages soon - see #10014 and I definitely recommend to switch to those.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] varundhussa commented on issue #10357: DataProcHook cancel method does not execute the REST call
Posted by GitBox <gi...@apache.org>.
varundhussa commented on issue #10357:
URL: https://github.com/apache/airflow/issues/10357#issuecomment-674792983
Hi @turbaszek Since v1-10-stable branch has the v2.0 code now, the issue does not exist there anymore. Releases up to tag 1-10.11 have the issue since the older contrib hook was being used. I'm not sure which branch to raise a PR on. What would you recommend?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] varundhussa commented on issue #10357: DataProcHook cancel method does not execute the REST call
Posted by GitBox <gi...@apache.org>.
varundhussa commented on issue #10357:
URL: https://github.com/apache/airflow/issues/10357#issuecomment-674843488
Thanks @potiuk . I completely understand. I will be moving to the 2.0 ports.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] varundhussa closed issue #10357: DataProcHook cancel method does not execute the REST call
Posted by GitBox <gi...@apache.org>.
varundhussa closed issue #10357:
URL: https://github.com/apache/airflow/issues/10357
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on issue #10357: DataProcHook cancel method does not execute the REST call
Posted by GitBox <gi...@apache.org>.
turbaszek commented on issue #10357:
URL: https://github.com/apache/airflow/issues/10357#issuecomment-674746815
Thanks @varundhussa would you like to fix it? In the mean time I strongly suggest to take a look at
https://github.com/apache/airflow#backport-packages
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org