You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by "esgungor (via GitHub)" <gi...@apache.org> on 2025/05/16 14:42:55 UTC
[I] BigQuery ProjectId is missing on query [superset]
esgungor opened a new issue, #33465:
URL: https://github.com/apache/superset/issues/33465
### Bug description
**Describe the bug**
When configuring a BigQuery database connection in Superset using a service account, the generated SQL queries fail if the service account's project ID is different from the project ID where the target BigQuery dataset and table reside.
The BigQuery connector appears to generate table references in the format `` `dataset`.`table_name` ``. While this works when querying tables within the same project as the service account, cross-project queries require the format `` `projectId`.`dataset`.`table_name` ``. The missing project ID in the generated query prevents successful execution.
**To Reproduce**
Steps to reproduce the behavior:
1. Configure a new BigQuery database connection in Superset.
2. Select "Service Account" as the authentication method.
3. Modify SQLAlchemy to seperate billing_project_id and project_id. It still waiting [approve](https://github.com/googleapis/python-bigquery-sqlalchemy/pull/1180)
4. Use a service account whose project ID is `project-A`.
5. Add a dataset and table from a *different* project, say `project-B`, to Superset.
6. Attempt to run a query or create a chart using the table from `project-B`.
**Actual behavior**
The query fails with an error (likely related to permissions or table not found), because the generated SQL omits the required project ID for cross-project access, like:
```sql
SELECT `column` FROM `dataset`.`table_name`
```
**Proposed Solution**
A query that includes the project ID format works correctly:
```sql
SELECT `example` AS `example` FROM `projectId`.`dataset`.`table_name`
```
Environment:
Superset version:4.1.2
### Screenshots/recordings
_No response_
### Superset version
master / latest-dev
### Python version
3.9
### Node version
16
### Browser
Chrome
### Additional context
_No response_
### Checklist
- [ ] I have searched Superset docs and Slack and didn't find a solution to my problem.
- [ ] I have searched the GitHub issue tracker and didn't find a similar bug report.
- [ ] I have checked Superset's logs for errors and if I found a relevant Python stacktrace, I included it here as text in the "additional context" section.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] BigQuery ProjectId is missing on query [superset]
Posted by "Vitor-Avila (via GitHub)" <gi...@apache.org>.
Vitor-Avila commented on issue #33465:
URL: https://github.com/apache/superset/issues/33465#issuecomment-2894272774
I believe this is handled if you have **Allow changing catalogs** enabled and then create a physical dataset selecting the appropriate catalog?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] BigQuery ProjectId is missing on query [superset]
Posted by "esgungor (via GitHub)" <gi...@apache.org>.
esgungor commented on issue #33465:
URL: https://github.com/apache/superset/issues/33465#issuecomment-2893089197
@mistercrunch Yes, that works! However, if there are any limitations on virtual dataset, the query generation should be modified. Will it broke smth?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] BigQuery ProjectId is missing on query [superset]
Posted by "mistercrunch (via GitHub)" <gi...@apache.org>.
mistercrunch commented on issue #33465:
URL: https://github.com/apache/superset/issues/33465#issuecomment-2896026930
All features should work, but Superset will run predicates against the `SELECT * FROM` as a subquery, which can have perf implications.
```sql
SELECT dim1, count(1)
FROM (SELECT * FROM tbl)
WHERE {{ data_filters }}
GROUP BY 1
```
Some engines will generate the right plans (pushing the predicate on the inner table operator), some won't, unclear what snowflake will do.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] BigQuery ProjectId is missing on query [superset]
Posted by "betodealmeida (via GitHub)" <gi...@apache.org>.
betodealmeida closed issue #33465: BigQuery ProjectId is missing on query
URL: https://github.com/apache/superset/issues/33465
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] BigQuery ProjectId is missing on query [superset]
Posted by "dosubot[bot] (via GitHub)" <gi...@apache.org>.
dosubot[bot] commented on issue #33465:
URL: https://github.com/apache/superset/issues/33465#issuecomment-2886023958
<!-- Answer -->
To address the issue of missing project IDs in SQL queries for cross-project access in BigQuery when using a service account, you need to ensure that the SQLAlchemy URI is correctly configured to include the project ID. This can be done by setting the `sqlalchemy_uri_placeholder` to include the project ID format:
```python
sqlalchemy_uri_placeholder = "bigquery://{project_id}"
```
Additionally, when building the SQLAlchemy URI, ensure that the `build_sqlalchemy_uri` method uses the `project_id` from the service account credentials. The `encrypted_extra` should contain valid service account credentials with the `project_id` field. This setup allows Superset to construct the correct SQLAlchemy URI for cross-project access in BigQuery <sup>[[1]](https://github.com/apache/superset/blob/master/superset/db_engine_specs/bigquery.py)</sup>.
If the SQLAlchemy modification to separate `billing_project_id` and `project_id` is still pending approval, you might need to manually adjust the query format or wait for the update to be merged and released.
<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*
---
<sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/b5fa1301-fb5c-4986-bba9-e4720f26d6a6?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/b5fa1301-fb5c-4986-bba9-e4720f26d6a6?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/b5fa1301-fb5c-4986-bba9-e4720f26d6a6?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/b5fa1301-fb5c-4986-bba9-e4720f26d6a6?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/b5fa1301-fb5c-4986-bba9-e4720f26d6a6?feedback_type=hallucination) | [Bug Report](https://app.dosu.dev/response-feedback/b5fa1301-fb5c-4986-bba9-e4720f26d6a6?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/b5fa1301-fb5c-4986-bba9-e4720f26d6a6?feedback_type=other)</sup>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] BigQuery ProjectId is missing on query [superset]
Posted by "betodealmeida (via GitHub)" <gi...@apache.org>.
betodealmeida commented on issue #33465:
URL: https://github.com/apache/superset/issues/33465#issuecomment-3128758620
I have a fix for this here: https://github.com/apache/superset/pull/34360
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] BigQuery ProjectId is missing on query [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on issue #33465:
URL: https://github.com/apache/superset/issues/33465#issuecomment-2891794420
@mistercrunch @Vitor-Avila got any pro tips on this one?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [I] BigQuery ProjectId is missing on query [superset]
Posted by "mistercrunch (via GitHub)" <gi...@apache.org>.
mistercrunch commented on issue #33465:
URL: https://github.com/apache/superset/issues/33465#issuecomment-2892285700
Sounds like you should be able to create a virtual dataset that's a simple
```
SELECT `example` AS `example` FROM `projectId`.`dataset`.`table_name`
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org