You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "irvifa (via GitHub)" <gi...@apache.org> on 2023/06/26 10:54:16 UTC
[GitHub] [airflow] irvifa opened a new pull request, #32143: fix: Fix hardcoded BigQuery to GCS project id
irvifa opened a new pull request, #32143:
URL: https://github.com/apache/airflow/pull/32143
In the airflow configuration, we defined the self.project_id where we pass the information regarding the project_id where we run the job. There's another configuration that is defined in _prepare_configuration regarding how we define the job id given for the BQ table source.
closes: https://github.com/apache/airflow/issues/32106
related: https://github.com/apache/airflow/issues/32106
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal closed pull request #32143: fix: Fix hardcoded BigQuery to GCS project id
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal closed pull request #32143: fix: Fix hardcoded BigQuery to GCS project id
URL: https://github.com/apache/airflow/pull/32143
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on pull request #32143: fix: Fix hardcoded BigQuery to GCS project id
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on PR #32143:
URL: https://github.com/apache/airflow/pull/32143#issuecomment-1623060206
Closed in favor of https://github.com/apache/airflow/pull/32232
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] avinashpandeshwar-sharechat commented on pull request #32143: fix: Fix hardcoded BigQuery to GCS project id
Posted by "avinashpandeshwar-sharechat (via GitHub)" <gi...@apache.org>.
avinashpandeshwar-sharechat commented on PR #32143:
URL: https://github.com/apache/airflow/pull/32143#issuecomment-1611109923
I can confirm that below change to `airflow/providers/google/cloud/transfers/bigquery_to_gcs.py` also works, while keeping @Yaro1 's changes and also respecting the project_id parameter in the operator.
```
return hook.insert_job(
configuration=configuration,
project_id=self.project_id or configuration["extract"]["sourceTable"]["projectId"],
location=self.location,
job_id=job_id,
timeout=self.result_timeout,
retry=self.result_retry,
nowait=self.deferrable,
)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #32143: fix: Fix hardcoded BigQuery to GCS project id
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #32143:
URL: https://github.com/apache/airflow/pull/32143#issuecomment-1610225420
@Yaro1 - What do you think about this change ? Your change from hook.project_id from https://github.com/apache/airflow/pull/30053 seems to be now reverting somehow
@irvifa and @Yaro1 -> maybe you can discuss which is the best approach?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] avinashpandeshwar commented on pull request #32143: fix: Fix hardcoded BigQuery to GCS project id
Posted by "avinashpandeshwar (via GitHub)" <gi...@apache.org>.
avinashpandeshwar commented on PR #32143:
URL: https://github.com/apache/airflow/pull/32143#issuecomment-1611114408
I can confirm that below change to `airflow/providers/google/cloud/transfers/bigquery_to_gcs.py` also works, while keeping @Yaro1 's changes and also respecting the project_id parameter in the operator.
How this works - For job computation, use project_id parameter if provided, else use the project_id from the table.
```
return hook.insert_job(
configuration=configuration,
project_id=self.project_id or configuration["extract"]["sourceTable"]["projectId"],
location=self.location,
job_id=job_id,
timeout=self.result_timeout,
retry=self.result_retry,
nowait=self.deferrable,
)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org