You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "ying-w (via GitHub)" <gi...@apache.org> on 2023/04/14 01:00:26 UTC

[GitHub] [airflow] ying-w opened a new issue, #30635: `BigQueryGetDataOperator` does not respect project_id parameter

ying-w opened a new issue, #30635:
URL: https://github.com/apache/airflow/issues/30635

   ### Apache Airflow Provider(s)
   
   google
   
   ### Versions of Apache Airflow Providers
   
   apache-airflow-providers-google==8.11.0
   google-cloud-bigquery==2.34.4
   
   ### Apache Airflow version
   
   2.5.2+astro.2
   
   ### Operating System
   
   OSX
   
   ### Deployment
   
   Astronomer
   
   ### Deployment details
   
   _No response_
   
   ### What happened
   
   When setting a `project_id` parameter for `BigQueryGetDataOperator` the default project from env is not overwritten. Maybe something broke after it was added in? https://github.com/apache/airflow/pull/25782
   
   ### What you think should happen instead
   
   Passing in as parameter should take precedence over reading in from environment
   
   ### How to reproduce
   
   ```py
   from airflow.providers.google.cloud.operators.bigquery import BigQueryGetDataOperator
   
   bq = BigQueryGetDataOperator(
     task_id=f"my_test_query_task_id",
     gcp_conn_id="bigquery",
     table_id="mytable",
     dataset_id="mydataset",
     project="my_non_default_project",
   )
   f2 = bq.execute(None)
   ```
   
   in env i have set
   ```py
   AIRFLOW_CONN_BIGQUERY=gcpbigquery://
   GOOGLE_CLOUD_PROJECT=my_primary_project
   GOOGLE_APPLICATION_CREDENTIALS=/usr/local/airflow/gcloud/application_default_credentials.json
   ```
   
   The credentials json file doesn't have project
   
   
   ### Anything else
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] nitinpandey-154 commented on issue #30635: `BigQueryGetDataOperator` does not respect project_id parameter

Posted by "nitinpandey-154 (via GitHub)" <gi...@apache.org>.
nitinpandey-154 commented on issue #30635:
URL: https://github.com/apache/airflow/issues/30635#issuecomment-1508307470

   In Part1, Do you mean `project_id` when invocing `BigQueryGetDataOperator`?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] eladkal commented on issue #30635: `BigQueryGetDataOperator` does not respect project_id parameter

Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #30635:
URL: https://github.com/apache/airflow/issues/30635#issuecomment-1508297896

   @sudohainguyen can you take a look?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] sudohainguyen commented on issue #30635: `BigQueryGetDataOperator` does not respect project_id parameter

Posted by "sudohainguyen (via GitHub)" <gi...@apache.org>.
sudohainguyen commented on issue #30635:
URL: https://github.com/apache/airflow/issues/30635#issuecomment-1508885953

   I guess you meant the billing project for the query execution, you should use `location` keyword param. `project_id` is where your table is located at


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] ying-w commented on issue #30635: `BigQueryGetDataOperator` does not respect project_id parameter

Posted by "ying-w (via GitHub)" <gi...@apache.org>.
ying-w commented on issue #30635:
URL: https://github.com/apache/airflow/issues/30635#issuecomment-1508990796

   @sudohainguyen i think i found it, `hook.get_schema()` needs `project_id=self.project_id`
   https://github.com/apache/airflow/blob/916dd96691a6c9b0866ca63c864ebbec8872ba44/airflow/providers/google/cloud/operators/bigquery.py#L905


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] ying-w commented on issue #30635: `BigQueryGetDataOperator` does not respect project_id parameter

Posted by "ying-w (via GitHub)" <gi...@apache.org>.
ying-w commented on issue #30635:
URL: https://github.com/apache/airflow/issues/30635#issuecomment-1508929440

   I'm trying to query table in `my_non_default_project` when I have `GOOGLE_CLOUD_PROJECT=my_primary_project` set in env.
   
   With `GOOGLE_CLOUD_PROJECT=my_primary_project` set and with `location=my_non_default_project` and `project_id=my_non_default_project`: same error where the GET url is using `my_primary_project` rather than `my_non_default_project`
   
   Without `GOOGLE_CLOUD_PROJECT=my_primary_project` set and with `location=my_non_default_project` and `project_id=my_non_default_project`: same error where `project_id` is not found


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] ying-w commented on issue #30635: `BigQueryGetDataOperator` does not respect project_id parameter

Posted by "ying-w (via GitHub)" <gi...@apache.org>.
ying-w commented on issue #30635:
URL: https://github.com/apache/airflow/issues/30635#issuecomment-1508898724

   > In Part1, Do you mean `project_id` when invoking `BigQueryGetDataOperator`?
   
   yes sorry, just edited code sample. `project_id=` parameter
   
   > the billing project for the query execution, you should use location keyword param
   
   Setting `location='us'` doesn't do anything


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] sudohainguyen commented on issue #30635: `BigQueryGetDataOperator` does not respect project_id parameter

Posted by "sudohainguyen (via GitHub)" <gi...@apache.org>.
sudohainguyen commented on issue #30635:
URL: https://github.com/apache/airflow/issues/30635#issuecomment-1508906620

   Try setting location=my_primary_project 🤔


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] hussein-awala closed issue #30635: `BigQueryGetDataOperator` does not respect project_id parameter

Posted by "hussein-awala (via GitHub)" <gi...@apache.org>.
hussein-awala closed issue #30635: `BigQueryGetDataOperator` does not respect project_id parameter
URL: https://github.com/apache/airflow/issues/30635


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org