(iceberg) branch main updated: Kafka Connect: Add BigQuery Metastore catalog (#13041)

[ru...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

russellspitzer pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/iceberg.git


The following commit(s) were added to refs/heads/main by this push:
     new acffd4dc5d Kafka Connect: Add BigQuery Metastore catalog (#13041)
acffd4dc5d is described below

commit acffd4dc5d65303ab71d9a65903891b7b0c249b1
Author: Julien Guitton <31...@users.noreply.github.com>
AuthorDate: Fri May 16 20:49:43 2025 +0200

    Kafka Connect: Add BigQuery Metastore catalog (#13041)
---
 docs/docs/kafka-connect.md | 15 +++++++++++++--
 kafka-connect/build.gradle |  3 +++
 2 files changed, 16 insertions(+), 2 deletions(-)

diff --git a/docs/docs/kafka-connect.md b/docs/docs/kafka-connect.md
index 92bcb8c708..5815987de0 100644
--- a/docs/docs/kafka-connect.md
+++ b/docs/docs/kafka-connect.md
@@ -106,7 +106,7 @@ Messages should be converted to a struct or map using the appropriate Kafka Conn
 
 The `iceberg.catalog.*` properties are required for connecting to the Iceberg catalog. The core catalog
 types are included in the default distribution, including REST, Glue, DynamoDB, Hadoop, Nessie,
-JDBC, and Hive. JDBC drivers are not included in the default distribution, so you will need to include
+JDBC, Hive and BigQuery Metastore. JDBC drivers are not included in the default distribution, so you will need to include
 those if needed. When using a Hive catalog, you can use the distribution that includes the Hive metastore client,
 otherwise you will need to include that yourself.
 
@@ -125,7 +125,7 @@ catalog types, you need to instead set `iceberg.catalog.catalog-impl` to the nam
 #### Hive example
 
 NOTE: Use the distribution that includes the HMS client (or include the HMS client yourself). Use `S3FileIO` when
-using S3 for storage (the default is `HadoopFileIO` with `HiveCatalog`).
+using S3 for storage and `GCSFileIO` when using GCS (the default is `HadoopFileIO` with `HiveCatalog`).
 ```
 "iceberg.catalog.type": "hive",
 "iceberg.catalog.uri": "thrift://hive:9083",
@@ -154,6 +154,17 @@ using S3 for storage (the default is `HadoopFileIO` with `HiveCatalog`).
 "iceberg.catalog.io-impl": "org.apache.iceberg.aws.s3.S3FileIO",
 ```
 
+#### BigQuery Metastore example
+
+```
+"iceberg.catalog.catalog-impl": "org.apache.iceberg.gcp.bigquery.BigQueryMetastoreCatalog",
+"iceberg.catalog.gcp.bigquery.project-id": "my-project",
+"iceberg.catalog.gcp.bigquery.location": "us-east1",
+"iceberg.catalog.warehouse": "gs://bucket/warehouse",
+"iceberg.catalog.io-impl": "org.apache.iceberg.gcp.gcs.GCSFileIO",
+"iceberg.tables.auto-create-props.bq_connection": "projects/my-project/locations/us-east1/connections/my-connection",
+```
+
 #### Notes
 
 Depending on your setup, you may need to also set `iceberg.catalog.s3.endpoint`, `iceberg.catalog.s3.staging-dir`,
diff --git a/kafka-connect/build.gradle b/kafka-connect/build.gradle
index 75d8fbe9b7..06653e1d62 100644
--- a/kafka-connect/build.gradle
+++ b/kafka-connect/build.gradle
@@ -126,8 +126,11 @@ project(':iceberg-kafka-connect:iceberg-kafka-connect-runtime') {
     implementation 'software.amazon.awssdk:lakeformation'
 
     implementation project(':iceberg-gcp')
+    implementation project(':iceberg-bigquery')
     implementation platform(libs.google.libraries.bom)
     implementation 'com.google.cloud:google-cloud-storage'
+    implementation 'com.google.cloud:google-cloud-bigquery'
+    implementation 'com.google.cloud:google-cloud-core'
 
     implementation project(':iceberg-azure')
     implementation platform(libs.azuresdk.bom)