You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "jroachgolf84 (via GitHub)" <gi...@apache.org> on 2025/12/12 13:33:06 UTC
[PR] `issue-59344`: Returning `destination_cloud_storage_uris` from `BigQueryToGCSOperator` [airflow]
jroachgolf84 opened a new pull request, #59367:
URL: https://github.com/apache/airflow/pull/59367
## Description
Per #59344, the `BigQueryToGCSOperator` has no return value. Ideally, this Operator would provide information about the files that are being written to GCS for other Tasks, Sensors, etc. to read.
This PR adds the return of the `destination_cloud_storage_uris` from the `execute` and `execute_complete` methods.
## Testing
Unit-tests in the `proviers/google/tests/unit/google/cloud/transfers/test_bigquery_to_gcs.py` were updated to properly reflect the new return values in the `execute` and `execute_complete` methods. All unit-tests for `BigQueryToGCSOperator` (including the newly updated/created ones) can be run using the command below:
```bash
breeze testing providers-tests providers/google/tests/unit/google/cloud/transfers/test_bigquery_to_gcs.py
```
closes: https://github.com/apache/airflow/issues/59344
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Returning `destination_cloud_storage_uris` from `BigQueryToGCSOperator` [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on code in PR #59367:
URL: https://github.com/apache/airflow/pull/59367#discussion_r2614922490
##########
providers/google/src/airflow/providers/google/cloud/transfers/bigquery_to_gcs.py:
##########
@@ -275,6 +275,9 @@ def execute(self, context: Context):
else:
job.result(timeout=self.result_timeout, retry=self.result_retry)
+ # issue-59344: Returning destination URIs to be persisted in XCom
Review Comment:
@shahar1, I've updated the existing docstring at the class-level to capture the return type.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Returning `destination_cloud_storage_uris` from `BigQueryToGCSOperator` [airflow]
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 merged PR #59367:
URL: https://github.com/apache/airflow/pull/59367
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Returning `destination_cloud_storage_uris` from `BigQueryToGCSOperator` [airflow]
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on code in PR #59367:
URL: https://github.com/apache/airflow/pull/59367#discussion_r2614738131
##########
providers/google/src/airflow/providers/google/cloud/transfers/bigquery_to_gcs.py:
##########
@@ -275,6 +275,9 @@ def execute(self, context: Context):
else:
job.result(timeout=self.result_timeout, retry=self.result_retry)
+ # issue-59344: Returning destination URIs to be persisted in XCom
Review Comment:
Could you please remove both comments (here and in `execute_complete`), and instead add a docstring in the class level?
You may take inspiration from [DbtCloudRunJobOperator](https://github.com/apache/airflow/blob/8256ee16d1de5b47ad996af82608d2bb836dfe1d/providers/dbt/cloud/src/airflow/providers/dbt/cloud/operators/dbt.py#L87)'s docstring.
I think that it would be more informative for the end-user.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Returning `destination_cloud_storage_uris` from `BigQueryToGCSOperator` [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on code in PR #59367:
URL: https://github.com/apache/airflow/pull/59367#discussion_r2614743843
##########
providers/google/src/airflow/providers/google/cloud/transfers/bigquery_to_gcs.py:
##########
@@ -275,6 +275,9 @@ def execute(self, context: Context):
else:
job.result(timeout=self.result_timeout, retry=self.result_retry)
+ # issue-59344: Returning destination URIs to be persisted in XCom
Review Comment:
You bet, will do!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org