You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "joyhaldar (via GitHub)" <gi...@apache.org> on 2025/10/27 07:30:31 UTC
[I] BigQueryMetastoreCatalog configuration properties: gcp_project vs gcp.bigquery.project-id, which is correct? [iceberg]
joyhaldar opened a new issue, #14422:
URL: https://github.com/apache/iceberg/issues/14422
### Query engine
Spark locally and GCP Dataproc Serverless on the cloud.
### Question
_**Description:**_
I'm confused about the correct configuration properties for **BigQueryMetastoreCatalog** and believe there may be an inconsistency or documentation gap.
_**Context:**_
When using BigQueryMetastoreCatalog on Google Dataproc, the configuration from [Google's official documentation](https://docs.cloud.google.com/biglake/docs/configure-blms) works fine:
```
spark.sql.catalog.my_catalog.gcp_project=PROJECT_ID
spark.sql.catalog.my_catalog.gcp_location=LOCATION
```
However, when I try to use the Apache Iceberg JAR directly (e.g., iceberg:1.10.0, iceberg-bigquery:1.10.0, ) with Spark running elsewhere (outside Dataproc), this configuration doesn't work.
**_Investigation:_**
Looking at the Iceberg source code (**[BigQueryMetastoreCatalog.java](https://github.com/apache/iceberg/blob/main/bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/BigQueryMetastoreCatalog.java)**), the properties are defined as:
```
public static final String PROJECT_ID = "gcp.bigquery.project-id";
public static final String GCP_LOCATION = "gcp.bigquery.location";
```
This suggests the configuration should be:
```
spark.sql.catalog.my_catalog.gcp.bigquery.project-id=PROJECT_ID
spark.sql.catalog.my_catalog.gcp.bigquery.location=LOCATION
```
**_Questions:_**
1. Why does the Google documentation approach (gcp_project, gcp_location) work on Dataproc but not with the standard Iceberg JAR?
2. Is there a configuration translation or aliasing layer in the Dataproc-provided JAR (gs://spark-lib/bigquery/iceberg-bigquery-catalog-1.6.1-1.0.1-beta.jar) that's not present in the Maven Central release?
3. What are the correct property names users should use when running Spark with Iceberg outside of Dataproc?
4. Should the Iceberg codebase support both property name formats for compatibility, or should Google's documentation be updated?
**_Expected Behavior:_**
Documentation on which property names to use, and ideally support for both formats to avoid user confusion.
**_Actual Behavior:_**
Users following Google's documentation may face issues when using Iceberg JARs from Maven Central outside of Dataproc environments.
Also, please correct me if I am wrong and way off.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] BigQueryMetastoreCatalog configuration properties: gcp_project vs gcp.bigquery.project-id, which is correct? [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #14422: BigQueryMetastoreCatalog configuration properties: gcp_project vs gcp.bigquery.project-id, which is correct?
URL: https://github.com/apache/iceberg/issues/14422
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] BigQueryMetastoreCatalog configuration properties: gcp_project vs gcp.bigquery.project-id, which is correct? [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on issue #14422:
URL: https://github.com/apache/iceberg/issues/14422#issuecomment-3450011943
/cc @talatuyarer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] BigQueryMetastoreCatalog configuration properties: gcp_project vs gcp.bigquery.project-id, which is correct? [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #14422:
URL: https://github.com/apache/iceberg/issues/14422#issuecomment-4320906524
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] BigQueryMetastoreCatalog configuration properties: gcp_project vs gcp.bigquery.project-id, which is correct? [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #14422:
URL: https://github.com/apache/iceberg/issues/14422#issuecomment-4414063790
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org