You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "ramesh-aqfer (via GitHub)" <gi...@apache.org> on 2025/09/04 01:13:00 UTC
[I] To create Iceberg tables for BigQuery consumption which catalog should we use ? [iceberg]
ramesh-aqfer opened a new issue, #13990:
URL: https://github.com/apache/iceberg/issues/13990
### Query engine
I am using our own data processing application.
### Question
Apace Iceberg library offers **RESTCatalog** through which I am able to connect with BigLake Metastore.
There is another relatively new catalog called **BigQueryMetastoreCatalog** that I have not tried yet.
Which one to pick to create/update Iceberg tables for BigQuery consumption ?
The RESTCatalog has some limitations when dealing with BigQuery:
- The tables created through RESTCatalog do not automatically appear in BigQuery. We need to manually create an external table in BigQuery using the metadata location.
- Whenever the table's snapshot is updated, the new metadata location need to be manually updated in BigQuery.
- These tables are readonly in BigQuery
The new catalog BigQueryMetastoreCatalog is supposed to address any of these limitations ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Which catalog object should we use, to create Iceberg tables for BigQuery consumption ? [iceberg]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #13990:
URL: https://github.com/apache/iceberg/issues/13990#issuecomment-3256006915
Hi @ramesh-aqfer i think this is more of a product question for Google / BigQuery.
The [`BigQueryMetastoreCatalog` catalog implementation](https://github.com/apache/iceberg/blob/173d4c32840e1be1efb559a174e8b97a0e62fb98/bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/BigQueryMetastoreCatalog.java#L57-L58) refers to the "Bigquery Metastore Catalog"
I've found these references
https://cloud.google.com/blog/products/data-analytics/introducing-bigquery-metastore-fully-managed-metadata-service
https://cloud.google.com/bigquery/docs/blms-rest-catalog
cc @talatuyarer who added the `BigQueryMetastoreCatalog` implementation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Which catalog object should we use, to create Iceberg tables for BigQuery consumption ? [iceberg]
Posted by "talatuyarer (via GitHub)" <gi...@apache.org>.
talatuyarer commented on issue #13990:
URL: https://github.com/apache/iceberg/issues/13990#issuecomment-3271262422
You are welcome @ramesh-aqfer
BigQuery Metastore will be included in the 1.10 release, which will be published in the coming days. In the interim, you may utilize the 1.10 Snapshot version: https://repository.apache.org/content/groups/snapshots/org/apache/iceberg/iceberg-bigquery/1.10.0-SNAPSHOT/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Which catalog object should we use, to create Iceberg tables for BigQuery consumption ? [iceberg]
Posted by "ramesh-aqfer (via GitHub)" <gi...@apache.org>.
ramesh-aqfer commented on issue #13990:
URL: https://github.com/apache/iceberg/issues/13990#issuecomment-3268875491
thanks @talatuyarer . I will try out the BigQuery meta store for our use case.
One last question. The `iceberg-gcp-1.9.2.jar` from the Maven central does not have `org.apache.iceberg.gcp.bigquery` package. Is there a different source for this jar file, or should we build from the source ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Which catalog object should we use, to create Iceberg tables for BigQuery consumption ? [iceberg]
Posted by "talatuyarer (via GitHub)" <gi...@apache.org>.
talatuyarer commented on issue #13990:
URL: https://github.com/apache/iceberg/issues/13990#issuecomment-3266845540
Hi @ramesh-aqfer
Our plan is BigLake Rest Catalog is best and greater for GCP users but it is not there yet.
You dont need to wait BigLake Rest Catalog, you can currently use BigQuery Meta Store for managed or external use cases.
I wrote below statement because it is obvious for Biglake Rest Catalog. It does not support BigQuery Tables. This may change in 6 months. But this is current state.
> BigQueryMetastoreCatalog : The workflow is to create and manage your Iceberg tables directly within BigQuery using standard SQL DDL. These tables are fully read/write, show up in the UI, and get all the native BigQuery features like fine-grained access control. You then use the BigQueryMetastoreCatalog from open-source engines like Spark or Flink to read these BigQuery-managed tables.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Which catalog object should we use, to create Iceberg tables for BigQuery consumption ? [iceberg]
Posted by "talatuyarer (via GitHub)" <gi...@apache.org>.
talatuyarer commented on issue #13990:
URL: https://github.com/apache/iceberg/issues/13990#issuecomment-3259469771
Hi @ramesh-aqfer
Great question! It can be a bit confusing since there are two parallel paths for this.
You're spot on about the limitations of the Big Lake Metastore RESTCatalog right now. Since it's still in Preview, tables managed through it are treated as external, read-only tables by BigQuery, and they don't show up automatically in the UI. However you can query them. You can find [more details here](https://cloud.google.com/bigquery/docs/blms-rest-catalog#query_a_table).
The BigQueryMetastoreCatalog is the GA (General Availability) solution designed to solve these exact problems. It's meant for BigQuery-managed Iceberg tables.
Here’s the breakdown:
**BigQueryMetastoreCatalog (GA)**: This is your best bet for now, if you are going to use Managed Iceberg Tables. The workflow is to create and manage your Iceberg tables directly within BigQuery using standard SQL DDL. These tables are fully read/write, show up in the UI, and get all the native BigQuery features like fine-grained access control. You then use the BigQueryMetastoreCatalog from open-source engines like Spark or Flink to read these BigQuery-managed tables.
**BigLake RESTCatalog (Preview)**: This is the newer path and aligns with the open-source Iceberg REST API standard. It's the recommended direction for the future, and it will eventually have deep integration with BigQuery. For now, it's best for use cases where you're managing tables outside BQ and just want to query them from there.
For specific roadmap questions or further technical questions, please email biglake-help@google.com
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Which catalog object should we use, to create Iceberg tables for BigQuery consumption ? [iceberg]
Posted by "ramesh-aqfer (via GitHub)" <gi...@apache.org>.
ramesh-aqfer commented on issue #13990:
URL: https://github.com/apache/iceberg/issues/13990#issuecomment-3257884637
thanks @kevinjqliu for the inputs.
@talatuyarer It will be helpful if I can get this clarity :
Apace Iceberg RESTCatalog works fine with BigLake metastore, but the table is not automatically visible in the BigQuery. Using the **BigQueryMetastoreCatalog** will address this gap ?
If I want to site a similar scenario with AWS, when I create a table in the AWS Glue using Apace Iceberg RESTCatalog, then those tables automatically appear in Athena. Looking for similar behavior for BigQuery's externally managed Iceberg tables.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Which catalog object should we use, to create Iceberg tables for BigQuery consumption ? [iceberg]
Posted by "ramesh-aqfer (via GitHub)" <gi...@apache.org>.
ramesh-aqfer commented on issue #13990:
URL: https://github.com/apache/iceberg/issues/13990#issuecomment-3261548028
thanks @talatuyarer for the details.
> BigQueryMetastoreCatalog : The workflow is to create and manage your Iceberg tables directly within BigQuery using standard SQL DDL. These tables are fully read/write, show up in the UI, and get all the native BigQuery features like fine-grained access control. You then use the BigQueryMetastoreCatalog from open-source engines like Spark or Flink to read these BigQuery-managed tables.
Our use case is opposite to this : we process the data and we want to update the final output into an Iceberg table - which should be later queried by our customers in BigQuery. For BigQuery this is an externally managed Iceberg table. Based on your clarification my understanding is that we should use BigLake RESTCatalog to register our updates in the BigLake, but we need to wait for the BigLake - BigQuery integration if our customers want to query this data in BigQuery. Am I correct ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Which catalog object should we use, to create Iceberg tables for BigQuery consumption ? [iceberg]
Posted by "ramesh-aqfer (via GitHub)" <gi...@apache.org>.
ramesh-aqfer closed issue #13990: Which catalog object should we use, to create Iceberg tables for BigQuery consumption ?
URL: https://github.com/apache/iceberg/issues/13990
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org