You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/11/05 20:35:20 UTC

[GitHub] [airflow] lidalei opened a new pull request, #27521: set project_id and location when cancelijng bigquery job

lidalei opened a new pull request, #27521:
URL: https://github.com/apache/airflow/pull/27521

   <!--
   Thank you for contributing! Please make sure that your code changes
   are covered with tests. And in case of new features or big changes
   remember to adjust the documentation.
   
   Feel free to ping committers for the review!
   
   In case of an existing issue, reference it using one of the following:
   
   closes: #ISSUE
   related: #ISSUE
   
   How to write a good git commit message:
   http://chris.beams.io/posts/git-commit/
   -->
   This PR fixes an issue when canceling BigQuery jobs. The issue arises when the service account is in a Google Cloud project A but the job runs in another project B. The default project and location won't work anymore.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] lidalei commented on a diff in pull request #27521: set project_id and location when canceling BigQuery job

Posted by GitBox <gi...@apache.org>.
lidalei commented on code in PR #27521:
URL: https://github.com/apache/airflow/pull/27521#discussion_r1015874777


##########
tests/providers/google/cloud/hooks/test_bigquery.py:
##########
@@ -203,7 +203,10 @@ def test_cancel_queries(self, mock_client, mock_poll_job_complete):
         self.hook.running_job_id = running_job_id
         self.hook.cancel_query()
 
-        calls = [mock.call(job_id=running_job_id), mock.call(running_job_id)]
+        calls = [

Review Comment:
   @uranusjr Thx for your comment. The two test cases test `cancel_query` which calls `cancel_job`. They also check the calls invoking `poll_job_complete`. So I think it is good enough. During test, I need to set an environment variable `AIRFLOW_CONN_GOOGLE_CLOUD_DEFAULT=google-cloud-platform://` because the `GoogleBaseHook` [calls](https://github.com/apache/airflow/blob/main/airflow/providers/google/common/hooks/base_google.py#L231) `get_connection` in its constructor. I think we should move that function call within [a member function](https://github.com/apache/airflow/blob/main/airflow/providers/google/common/hooks/base_google.py#L333). But it is out of the scope of this PR.
   
   One concern is that `cancel_query` is being deprecated. When we fully deprecate it, we shall convert some of the test cases to cover `cancel_job`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] uranusjr merged pull request #27521: set project_id and location when canceling BigQuery job

Posted by GitBox <gi...@apache.org>.
uranusjr merged PR #27521:
URL: https://github.com/apache/airflow/pull/27521


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] uranusjr commented on pull request #27521: set project_id and location when cancelijng bigquery job

Posted by GitBox <gi...@apache.org>.
uranusjr commented on PR #27521:
URL: https://github.com/apache/airflow/pull/27521#issuecomment-1306074533

   Should we have a test for this?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] lidalei commented on a diff in pull request #27521: set project_id and location when canceling BigQuery job

Posted by GitBox <gi...@apache.org>.
lidalei commented on code in PR #27521:
URL: https://github.com/apache/airflow/pull/27521#discussion_r1015874777


##########
tests/providers/google/cloud/hooks/test_bigquery.py:
##########
@@ -203,7 +203,10 @@ def test_cancel_queries(self, mock_client, mock_poll_job_complete):
         self.hook.running_job_id = running_job_id
         self.hook.cancel_query()
 
-        calls = [mock.call(job_id=running_job_id), mock.call(running_job_id)]
+        calls = [

Review Comment:
   @uranusjr Thx for your comment. The two test cases test `cancel_query` which calls `cancel_job`. They also check the calls invoking `poll_job_complete`. So I think it is good enough. During test, I need to set an environment variable `AIRFLOW_CONN_GOOGLE_CLOUD_DEFAULT=google-cloud-platform://` because the `GoogleBaseHook` [calls](https://github.com/apache/airflow/blob/main/airflow/providers/google/common/hooks/base_google.py#L231) `get_connection` in its constructor. I think we should move that function call within a member function. But it is out of the scope of this PR.
   
   One concern is that `cancel_query` is being deprecated. When we fully deprecate it, we shall convert some of the test cases to cover `cancel_job`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org