You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/06/30 20:09:16 UTC
[GitHub] [airflow] leahecole opened a new pull request, #24768: Add project_id as a templated variable in two BQ operators
leahecole opened a new pull request, #24768:
URL: https://github.com/apache/airflow/pull/24768
Hi all! This is related to #23129. In BigQuery, fully qualified dataset names can be made using the GCP project ID even if the project ID isn't needed for the underlying API to work. It's not unusual to make `project_id` a default arg and to have it be templated, especially if you're using many BQ operators together. I've added it to two operators where I have used it to construct a dataset name.
I'm not 100% sure how we test templated fields - I looked at the BQ operator test and didn't see any kind of validation, and the test values we're passing to the mocked calls are not templated. Any recommendations for how to handle that are appreciated!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] leahecole commented on pull request #24768: Add project_id as a templated variable in two BQ operators
Posted by GitBox <gi...@apache.org>.
leahecole commented on PR #24768:
URL: https://github.com/apache/airflow/pull/24768#issuecomment-1177731470
Yup - sorry, was OOO for a US holiday, I'll look into it in the next couple of days
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on a diff in pull request #24768: Add project_id as a templated variable in two BQ operators
Posted by GitBox <gi...@apache.org>.
eladkal commented on code in PR #24768:
URL: https://github.com/apache/airflow/pull/24768#discussion_r920456050
##########
airflow/providers/google/cloud/operators/bigquery.py:
##########
@@ -1853,6 +1853,7 @@ class BigQueryUpsertTableOperator(BaseOperator):
'dataset_id',
'table_resource',
'impersonation_chain',
+ 'project_id'
Review Comment:
```suggestion
'project_id',
```
##########
airflow/providers/google/cloud/operators/bigquery.py:
##########
@@ -2068,6 +2069,7 @@ class BigQueryInsertJobOperator(BaseOperator):
"configuration",
"job_id",
"impersonation_chain",
+ 'project_id'
Review Comment:
```suggestion
"project_id",
```
These will probably fix the static checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #24768: Add project_id as a templated variable in two BQ operators
Posted by GitBox <gi...@apache.org>.
potiuk merged PR #24768:
URL: https://github.com/apache/airflow/pull/24768
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on pull request #24768: Add project_id as a templated variable in two BQ operators
Posted by GitBox <gi...@apache.org>.
eladkal commented on PR #24768:
URL: https://github.com/apache/airflow/pull/24768#issuecomment-1172070849
@leahecole I'm find with merging this without the test (most operators don't have it)
can you please fix the static tests?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on pull request #24768: Add project_id as a templated variable in two BQ operators
Posted by GitBox <gi...@apache.org>.
eladkal commented on PR #24768:
URL: https://github.com/apache/airflow/pull/24768#issuecomment-1171655860
> I'm not 100% sure how we test templated fields - I looked at the BQ operator test and didn't see any kind of validation, and the test values we're passing to the mocked calls are not templated. Any recommendations for how to handle that are appreciated!
You can check this example:
https://github.com/apache/airflow/blob/a3c9956f79310b529a79c1e43fb881c5e173d321/tests/providers/google/cloud/operators/test_cloud_storage_transfer_service.py#L717-L734
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] leahecole commented on pull request #24768: Add project_id as a templated variable in two BQ operators
Posted by GitBox <gi...@apache.org>.
leahecole commented on PR #24768:
URL: https://github.com/apache/airflow/pull/24768#issuecomment-1184492012
ah - you beat me to it, sorry for slowness! been focused on my intern. thank you @potiuk and @eladkal
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org