You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "rambleraptor (via GitHub)" <gi...@apache.org> on 2025/06/05 23:02:31 UTC
[PR] Added BigQuery Metastore Catalog [iceberg-python]
rambleraptor opened a new pull request, #2068:
URL: https://github.com/apache/iceberg-python/pull/2068
<!--
Thanks for opening a pull request!
-->
<!-- In the case this PR will resolve an issue, please replace ${GITHUB_ISSUE_ID} below with the actual Github issue id. -->
<!-- Closes #${GITHUB_ISSUE_ID} -->
# Rationale for this change
This PR brings BigQuery Metastore support to Python after it was merged into the [Java implementation](https://github.com/apache/iceberg/pull/12808)
# Are these changes tested?
Integration and unit tests included.
# Are there any user-facing changes?
Introduces a new Catalog type.
<!-- In the case of user-facing changes, please add the changelog label. -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Added BigQuery Metastore Catalog [iceberg-python]
Posted by "rambleraptor (via GitHub)" <gi...@apache.org>.
rambleraptor commented on code in PR #2068:
URL: https://github.com/apache/iceberg-python/pull/2068#discussion_r2252677047
##########
pyiceberg/catalog/bigquery_metastore.py:
##########
@@ -0,0 +1,419 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import json
+from typing import TYPE_CHECKING, Any, List, Optional, Set, Tuple, Union
+
+from google.api_core.exceptions import NotFound
+from google.cloud.bigquery import Client, Dataset, DatasetReference, TableReference
+from google.cloud.bigquery import Table as BQTable
+from google.cloud.bigquery.external_config import ExternalCatalogDatasetOptions, ExternalCatalogTableOptions
+from google.cloud.bigquery.schema import SerDeInfo, StorageDescriptor
+from google.cloud.exceptions import Conflict
+from google.oauth2 import service_account
+
+from pyiceberg.catalog import WAREHOUSE_LOCATION, MetastoreCatalog, PropertiesUpdateSummary
+from pyiceberg.exceptions import NamespaceAlreadyExistsError, NoSuchNamespaceError, NoSuchTableError, TableAlreadyExistsError
+from pyiceberg.io import load_file_io
+from pyiceberg.partitioning import UNPARTITIONED_PARTITION_SPEC, PartitionSpec
+from pyiceberg.schema import Schema
+from pyiceberg.serializers import FromInputFile
+from pyiceberg.table import CommitTableResponse, Table
+from pyiceberg.table.locations import load_location_provider
+from pyiceberg.table.metadata import TableMetadata, new_table_metadata
+from pyiceberg.table.snapshots import TOTAL_DATA_FILES, TOTAL_FILE_SIZE, TOTAL_RECORDS
+from pyiceberg.table.sorting import UNSORTED_SORT_ORDER, SortOrder
+from pyiceberg.table.update import TableRequirement, TableUpdate
+from pyiceberg.typedef import EMPTY_DICT, Identifier, Properties
+from pyiceberg.utils.config import Config
+
+if TYPE_CHECKING:
+ import pyarrow as pa
+
+GCP_PROJECT_ID = "gcp.project-id"
+GCP_LOCATION = "gcp.location"
+GCP_CREDENTIALS_LOCATION = "gcp.credentials-location"
+GCP_CREDENTIALS_INFO = "gcp.credentials-info"
+
+METADATA_LOCATION_PROP = "metadata_location"
+PREVIOUS_METADATA_LOCATION_PROP = "previous_metadata_location"
+TABLE_TYPE_PROP = "table_type"
+ICEBERG_TABLE_TYPE_VALUE = "ICEBERG"
+
+HIVE_SERIALIZATION_LIBRARY = "org.apache.iceberg.mr.hive.HiveIcebergSerDe"
Review Comment:
The [Java implementation](https://github.com/apache/iceberg/blob/main/bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/BigQueryMetastoreUtils.java#L69) currently adds all of these options
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Added BigQuery Metastore Catalog [iceberg-python]
Posted by "jayceslesar (via GitHub)" <gi...@apache.org>.
jayceslesar commented on code in PR #2068:
URL: https://github.com/apache/iceberg-python/pull/2068#discussion_r2173510707
##########
tests/catalog/integration_test_bigquery_metastore.py:
##########
Review Comment:
are special credentials needed to run this in CI?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Added BigQuery Metastore Catalog [iceberg-python]
Posted by "rambleraptor (via GitHub)" <gi...@apache.org>.
rambleraptor commented on PR #2068:
URL: https://github.com/apache/iceberg-python/pull/2068#issuecomment-3156329503
@kevinjqliu rebased!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Add BigQuery Metastore Catalog [iceberg-python]
Posted by "rambleraptor (via GitHub)" <gi...@apache.org>.
rambleraptor commented on PR #2068:
URL: https://github.com/apache/iceberg-python/pull/2068#issuecomment-3201924105
@Fokko @kevinjqliu yeah, that's the perfect description of the situation. Our product naming is...less than ideal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Added BigQuery Metastore Catalog [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on PR #2068:
URL: https://github.com/apache/iceberg-python/pull/2068#issuecomment-3189226983
Thanks for the pr @rambleraptor
I have some concerns around adding a vendor-specific catalog implementation to the repo. Is it possible to reuse the rest client? or the jdbc client?
I know we already have the glue and dynamodb catalog, but if its possible, id like not add another vendor catalogs. With S3 Tables, we did not end up merging the vendor specific catalog implementation (#1404) since S3 Tables provided support for the REST interface.
Are there plans to provide the REST interface in the future?
I see that the BigQuery Metastore catalog was merged on the java side (https://github.com/apache/iceberg/pull/12808).
I would like to hear other opinions from the community before proceeding.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Added BigQuery Metastore Catalog [iceberg-python]
Posted by "rambleraptor (via GitHub)" <gi...@apache.org>.
rambleraptor commented on code in PR #2068:
URL: https://github.com/apache/iceberg-python/pull/2068#discussion_r2252681655
##########
tests/catalog/integration_test_bigquery_metastore.py:
##########
Review Comment:
For the integration tests, yes. I'm unclear how credentials work on the current AWS test, so I could use some pointers on that.
Unlike AWS + mock_aws, GCP doesn't have a full mock implementation that we can use.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Add BigQuery Metastore Catalog [iceberg-python]
Posted by "talatuyarer (via GitHub)" <gi...@apache.org>.
talatuyarer commented on PR #2068:
URL: https://github.com/apache/iceberg-python/pull/2068#issuecomment-3202122422
Thank you @Fokko and @kevinjqliu
Yes There is two independed product. BQ does not support Rest catalog. BigQuery Metastore whhich we implement in this PR, is a GA product since This January https://cloud.google.com/blog/products/data-analytics/introducing-bigquery-metastore-fully-managed-metadata-service
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Added BigQuery Metastore Catalog [iceberg-python]
Posted by "djouallah (via GitHub)" <gi...@apache.org>.
djouallah commented on PR #2068:
URL: https://github.com/apache/iceberg-python/pull/2068#issuecomment-2978786103
@rambleraptor user here, I tried to use bigquery metastore recently when it was announced it has added rest API interface, I am just wondering, why you need to add support for pyiceberg if it is already using rest API ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Add BigQuery Metastore Catalog [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko merged PR #2068:
URL: https://github.com/apache/iceberg-python/pull/2068
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Add BigQuery Metastore Catalog [iceberg-python]
Posted by "rambleraptor (via GitHub)" <gi...@apache.org>.
rambleraptor commented on PR #2068:
URL: https://github.com/apache/iceberg-python/pull/2068#issuecomment-3211920336
I added a pytest.mark.skipif to ensure that integration tests are not run if credentials are not present.
I also filed #2368 to get the integration tests running + clean up the GCP integration tests in general.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Added BigQuery Metastore Catalog [iceberg-python]
Posted by "jayceslesar (via GitHub)" <gi...@apache.org>.
jayceslesar commented on PR #2068:
URL: https://github.com/apache/iceberg-python/pull/2068#issuecomment-3015520546
Might be wise to also try and include the changes here https://github.com/apache/iceberg/pull/13052 -- via https://github.com/apache/iceberg/pull/12808#discussion_r2074868327
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Added BigQuery Metastore Catalog [iceberg-python]
Posted by "talatuyarer (via GitHub)" <gi...@apache.org>.
talatuyarer commented on code in PR #2068:
URL: https://github.com/apache/iceberg-python/pull/2068#discussion_r2252419432
##########
pyiceberg/catalog/bigquery_metastore.py:
##########
@@ -0,0 +1,419 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import json
+from typing import TYPE_CHECKING, Any, List, Optional, Set, Tuple, Union
+
+from google.api_core.exceptions import NotFound
+from google.cloud.bigquery import Client, Dataset, DatasetReference, TableReference
+from google.cloud.bigquery import Table as BQTable
+from google.cloud.bigquery.external_config import ExternalCatalogDatasetOptions, ExternalCatalogTableOptions
+from google.cloud.bigquery.schema import SerDeInfo, StorageDescriptor
+from google.cloud.exceptions import Conflict
+from google.oauth2 import service_account
+
+from pyiceberg.catalog import WAREHOUSE_LOCATION, MetastoreCatalog, PropertiesUpdateSummary
+from pyiceberg.exceptions import NamespaceAlreadyExistsError, NoSuchNamespaceError, NoSuchTableError, TableAlreadyExistsError
+from pyiceberg.io import load_file_io
+from pyiceberg.partitioning import UNPARTITIONED_PARTITION_SPEC, PartitionSpec
+from pyiceberg.schema import Schema
+from pyiceberg.serializers import FromInputFile
+from pyiceberg.table import CommitTableResponse, Table
+from pyiceberg.table.locations import load_location_provider
+from pyiceberg.table.metadata import TableMetadata, new_table_metadata
+from pyiceberg.table.snapshots import TOTAL_DATA_FILES, TOTAL_FILE_SIZE, TOTAL_RECORDS
+from pyiceberg.table.sorting import UNSORTED_SORT_ORDER, SortOrder
+from pyiceberg.table.update import TableRequirement, TableUpdate
+from pyiceberg.typedef import EMPTY_DICT, Identifier, Properties
+from pyiceberg.utils.config import Config
+
+if TYPE_CHECKING:
+ import pyarrow as pa
+
+GCP_PROJECT_ID = "gcp.project-id"
+GCP_LOCATION = "gcp.location"
+GCP_CREDENTIALS_LOCATION = "gcp.credentials-location"
+GCP_CREDENTIALS_INFO = "gcp.credentials-info"
Review Comment:
@rambleraptor Please align settings with Java BigQuery Catalog
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Added BigQuery Metastore Catalog [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on PR #2068:
URL: https://github.com/apache/iceberg-python/pull/2068#issuecomment-3156068336
we merged a lot of library updates this morning. Could you rebase the PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Added BigQuery Metastore Catalog [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on PR #2068:
URL: https://github.com/apache/iceberg-python/pull/2068#issuecomment-3201909833
Since the Java side already has merged, I think it makes sense to move this forward here as well. From @talatuyarer I understand that the BigQuery Metadata store doesn't have a REST endpoint (yet), in constrast to BigLake (confusing naming :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Added BigQuery Metastore Catalog [iceberg-python]
Posted by "rambleraptor (via GitHub)" <gi...@apache.org>.
rambleraptor commented on code in PR #2068:
URL: https://github.com/apache/iceberg-python/pull/2068#discussion_r2252678876
##########
pyiceberg/catalog/bigquery_metastore.py:
##########
@@ -0,0 +1,419 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import json
+from typing import TYPE_CHECKING, Any, List, Optional, Set, Tuple, Union
+
+from google.api_core.exceptions import NotFound
+from google.cloud.bigquery import Client, Dataset, DatasetReference, TableReference
+from google.cloud.bigquery import Table as BQTable
+from google.cloud.bigquery.external_config import ExternalCatalogDatasetOptions, ExternalCatalogTableOptions
+from google.cloud.bigquery.schema import SerDeInfo, StorageDescriptor
+from google.cloud.exceptions import Conflict
+from google.oauth2 import service_account
+
+from pyiceberg.catalog import WAREHOUSE_LOCATION, MetastoreCatalog, PropertiesUpdateSummary
+from pyiceberg.exceptions import NamespaceAlreadyExistsError, NoSuchNamespaceError, NoSuchTableError, TableAlreadyExistsError
+from pyiceberg.io import load_file_io
+from pyiceberg.partitioning import UNPARTITIONED_PARTITION_SPEC, PartitionSpec
+from pyiceberg.schema import Schema
+from pyiceberg.serializers import FromInputFile
+from pyiceberg.table import CommitTableResponse, Table
+from pyiceberg.table.locations import load_location_provider
+from pyiceberg.table.metadata import TableMetadata, new_table_metadata
+from pyiceberg.table.snapshots import TOTAL_DATA_FILES, TOTAL_FILE_SIZE, TOTAL_RECORDS
+from pyiceberg.table.sorting import UNSORTED_SORT_ORDER, SortOrder
+from pyiceberg.table.update import TableRequirement, TableUpdate
+from pyiceberg.typedef import EMPTY_DICT, Identifier, Properties
+from pyiceberg.utils.config import Config
+
+if TYPE_CHECKING:
+ import pyarrow as pa
+
+GCP_PROJECT_ID = "gcp.project-id"
+GCP_LOCATION = "gcp.location"
+GCP_CREDENTIALS_LOCATION = "gcp.credentials-location"
+GCP_CREDENTIALS_INFO = "gcp.credentials-info"
+
+METADATA_LOCATION_PROP = "metadata_location"
+PREVIOUS_METADATA_LOCATION_PROP = "previous_metadata_location"
+TABLE_TYPE_PROP = "table_type"
+ICEBERG_TABLE_TYPE_VALUE = "ICEBERG"
+
+HIVE_SERIALIZATION_LIBRARY = "org.apache.iceberg.mr.hive.HiveIcebergSerDe"
+HIVE_FILE_INPUT_FORMAT = "org.apache.iceberg.mr.hive.HiveIcebergInputFormat"
+HIVE_FILE_OUTPUT_FORMAT = "org.apache.iceberg.mr.hive.HiveIcebergOutputFormat"
+
+
+class BigQueryMetastoreCatalog(MetastoreCatalog):
+ def __init__(self, name: str, **properties: str):
+ super().__init__(name, **properties)
+
+ project_id: Optional[str] = self.properties.get(GCP_PROJECT_ID)
+ location: Optional[str] = self.properties.get(GCP_LOCATION)
+ credentials_location: Optional[str] = self.properties.get(GCP_CREDENTIALS_LOCATION)
+ credentials_info_str: Optional[str] = self.properties.get(GCP_CREDENTIALS_INFO)
+
+ if not project_id:
+ raise ValueError(f"Missing property: {GCP_PROJECT_ID}")
+
+ # BigQuery requires current-snapshot-id to be present for tables to be created.
+ if not Config().get_bool("legacy-current-snapshot-id"):
+ raise ValueError("legacy-current-snapshot-id must be enabled to work with BigQuery.")
+
+ gcp_credentials = None
+ if credentials_location:
+ gcp_credentials = service_account.Credentials.from_service_account_file(credentials_location)
+ elif credentials_info_str:
+ try:
Review Comment:
Added an error message on this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Added BigQuery Metastore Catalog [iceberg-python]
Posted by "rambleraptor (via GitHub)" <gi...@apache.org>.
rambleraptor commented on code in PR #2068:
URL: https://github.com/apache/iceberg-python/pull/2068#discussion_r2252681655
##########
tests/catalog/integration_test_bigquery_metastore.py:
##########
Review Comment:
For the integration tests, yes. I'm unclear how credentials work on the current AWS test, so I could use some pointers on that. If we need real cloud resources to run these integration tests, they should be owned by Iceberg.
Unlike AWS + mock_aws, GCP doesn't have a full mock implementation that we can use.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Added BigQuery Metastore Catalog [iceberg-python]
Posted by "jayceslesar (via GitHub)" <gi...@apache.org>.
jayceslesar commented on code in PR #2068:
URL: https://github.com/apache/iceberg-python/pull/2068#discussion_r2173509573
##########
pyiceberg/catalog/bigquery_metastore.py:
##########
@@ -0,0 +1,419 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import json
+from typing import TYPE_CHECKING, Any, List, Optional, Set, Tuple, Union
+
+from google.api_core.exceptions import NotFound
+from google.cloud.bigquery import Client, Dataset, DatasetReference, TableReference
+from google.cloud.bigquery import Table as BQTable
+from google.cloud.bigquery.external_config import ExternalCatalogDatasetOptions, ExternalCatalogTableOptions
+from google.cloud.bigquery.schema import SerDeInfo, StorageDescriptor
+from google.cloud.exceptions import Conflict
+from google.oauth2 import service_account
+
+from pyiceberg.catalog import WAREHOUSE_LOCATION, MetastoreCatalog, PropertiesUpdateSummary
+from pyiceberg.exceptions import NamespaceAlreadyExistsError, NoSuchNamespaceError, NoSuchTableError, TableAlreadyExistsError
+from pyiceberg.io import load_file_io
+from pyiceberg.partitioning import UNPARTITIONED_PARTITION_SPEC, PartitionSpec
+from pyiceberg.schema import Schema
+from pyiceberg.serializers import FromInputFile
+from pyiceberg.table import CommitTableResponse, Table
+from pyiceberg.table.locations import load_location_provider
+from pyiceberg.table.metadata import TableMetadata, new_table_metadata
+from pyiceberg.table.snapshots import TOTAL_DATA_FILES, TOTAL_FILE_SIZE, TOTAL_RECORDS
+from pyiceberg.table.sorting import UNSORTED_SORT_ORDER, SortOrder
+from pyiceberg.table.update import TableRequirement, TableUpdate
+from pyiceberg.typedef import EMPTY_DICT, Identifier, Properties
+from pyiceberg.utils.config import Config
+
+if TYPE_CHECKING:
+ import pyarrow as pa
+
+GCP_PROJECT_ID = "gcp.project-id"
+GCP_LOCATION = "gcp.location"
+GCP_CREDENTIALS_LOCATION = "gcp.credentials-location"
+GCP_CREDENTIALS_INFO = "gcp.credentials-info"
Review Comment:
upstream defines these as
```java
public static final String CREDENTIAL_FILE = "gcp.bigquery.credential-file";
public static final String PROJECT_ID = "gcp.bigquery.project-id";
public static final String GCP_LOCATION = "gcp.bigquery.location";
public static final String LIST_ALL_TABLES = "gcp.bigquery.list-all-tables";
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Added BigQuery Metastore Catalog [iceberg-python]
Posted by "jayceslesar (via GitHub)" <gi...@apache.org>.
jayceslesar commented on code in PR #2068:
URL: https://github.com/apache/iceberg-python/pull/2068#discussion_r2173510100
##########
pyiceberg/catalog/bigquery_metastore.py:
##########
@@ -0,0 +1,419 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import json
+from typing import TYPE_CHECKING, Any, List, Optional, Set, Tuple, Union
+
+from google.api_core.exceptions import NotFound
+from google.cloud.bigquery import Client, Dataset, DatasetReference, TableReference
+from google.cloud.bigquery import Table as BQTable
+from google.cloud.bigquery.external_config import ExternalCatalogDatasetOptions, ExternalCatalogTableOptions
+from google.cloud.bigquery.schema import SerDeInfo, StorageDescriptor
+from google.cloud.exceptions import Conflict
+from google.oauth2 import service_account
+
+from pyiceberg.catalog import WAREHOUSE_LOCATION, MetastoreCatalog, PropertiesUpdateSummary
+from pyiceberg.exceptions import NamespaceAlreadyExistsError, NoSuchNamespaceError, NoSuchTableError, TableAlreadyExistsError
+from pyiceberg.io import load_file_io
+from pyiceberg.partitioning import UNPARTITIONED_PARTITION_SPEC, PartitionSpec
+from pyiceberg.schema import Schema
+from pyiceberg.serializers import FromInputFile
+from pyiceberg.table import CommitTableResponse, Table
+from pyiceberg.table.locations import load_location_provider
+from pyiceberg.table.metadata import TableMetadata, new_table_metadata
+from pyiceberg.table.snapshots import TOTAL_DATA_FILES, TOTAL_FILE_SIZE, TOTAL_RECORDS
+from pyiceberg.table.sorting import UNSORTED_SORT_ORDER, SortOrder
+from pyiceberg.table.update import TableRequirement, TableUpdate
+from pyiceberg.typedef import EMPTY_DICT, Identifier, Properties
+from pyiceberg.utils.config import Config
+
+if TYPE_CHECKING:
+ import pyarrow as pa
+
+GCP_PROJECT_ID = "gcp.project-id"
+GCP_LOCATION = "gcp.location"
+GCP_CREDENTIALS_LOCATION = "gcp.credentials-location"
+GCP_CREDENTIALS_INFO = "gcp.credentials-info"
+
+METADATA_LOCATION_PROP = "metadata_location"
+PREVIOUS_METADATA_LOCATION_PROP = "previous_metadata_location"
+TABLE_TYPE_PROP = "table_type"
+ICEBERG_TABLE_TYPE_VALUE = "ICEBERG"
+
+HIVE_SERIALIZATION_LIBRARY = "org.apache.iceberg.mr.hive.HiveIcebergSerDe"
+HIVE_FILE_INPUT_FORMAT = "org.apache.iceberg.mr.hive.HiveIcebergInputFormat"
+HIVE_FILE_OUTPUT_FORMAT = "org.apache.iceberg.mr.hive.HiveIcebergOutputFormat"
+
+
+class BigQueryMetastoreCatalog(MetastoreCatalog):
+ def __init__(self, name: str, **properties: str):
+ super().__init__(name, **properties)
+
+ project_id: Optional[str] = self.properties.get(GCP_PROJECT_ID)
+ location: Optional[str] = self.properties.get(GCP_LOCATION)
+ credentials_location: Optional[str] = self.properties.get(GCP_CREDENTIALS_LOCATION)
+ credentials_info_str: Optional[str] = self.properties.get(GCP_CREDENTIALS_INFO)
+
+ if not project_id:
+ raise ValueError(f"Missing property: {GCP_PROJECT_ID}")
+
+ # BigQuery requires current-snapshot-id to be present for tables to be created.
+ if not Config().get_bool("legacy-current-snapshot-id"):
+ raise ValueError("legacy-current-snapshot-id must be enabled to work with BigQuery.")
+
+ gcp_credentials = None
+ if credentials_location:
+ gcp_credentials = service_account.Credentials.from_service_account_file(credentials_location)
+ elif credentials_info_str:
+ try:
Review Comment:
`credentials_location` will take precedence if the user specifies both -- do we want to allow a user to specify both? idrk
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Add BigQuery Metastore Catalog [iceberg-python]
Posted by "rambleraptor (via GitHub)" <gi...@apache.org>.
rambleraptor commented on PR #2068:
URL: https://github.com/apache/iceberg-python/pull/2068#issuecomment-3216098565
Tests should be fixed!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Added BigQuery Metastore Catalog [iceberg-python]
Posted by "talatuyarer (via GitHub)" <gi...@apache.org>.
talatuyarer commented on code in PR #2068:
URL: https://github.com/apache/iceberg-python/pull/2068#discussion_r2252418744
##########
pyiceberg/catalog/bigquery_metastore.py:
##########
@@ -0,0 +1,419 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import json
+from typing import TYPE_CHECKING, Any, List, Optional, Set, Tuple, Union
+
+from google.api_core.exceptions import NotFound
+from google.cloud.bigquery import Client, Dataset, DatasetReference, TableReference
+from google.cloud.bigquery import Table as BQTable
+from google.cloud.bigquery.external_config import ExternalCatalogDatasetOptions, ExternalCatalogTableOptions
+from google.cloud.bigquery.schema import SerDeInfo, StorageDescriptor
+from google.cloud.exceptions import Conflict
+from google.oauth2 import service_account
+
+from pyiceberg.catalog import WAREHOUSE_LOCATION, MetastoreCatalog, PropertiesUpdateSummary
+from pyiceberg.exceptions import NamespaceAlreadyExistsError, NoSuchNamespaceError, NoSuchTableError, TableAlreadyExistsError
+from pyiceberg.io import load_file_io
+from pyiceberg.partitioning import UNPARTITIONED_PARTITION_SPEC, PartitionSpec
+from pyiceberg.schema import Schema
+from pyiceberg.serializers import FromInputFile
+from pyiceberg.table import CommitTableResponse, Table
+from pyiceberg.table.locations import load_location_provider
+from pyiceberg.table.metadata import TableMetadata, new_table_metadata
+from pyiceberg.table.snapshots import TOTAL_DATA_FILES, TOTAL_FILE_SIZE, TOTAL_RECORDS
+from pyiceberg.table.sorting import UNSORTED_SORT_ORDER, SortOrder
+from pyiceberg.table.update import TableRequirement, TableUpdate
+from pyiceberg.typedef import EMPTY_DICT, Identifier, Properties
+from pyiceberg.utils.config import Config
+
+if TYPE_CHECKING:
+ import pyarrow as pa
+
+GCP_PROJECT_ID = "gcp.project-id"
+GCP_LOCATION = "gcp.location"
+GCP_CREDENTIALS_LOCATION = "gcp.credentials-location"
+GCP_CREDENTIALS_INFO = "gcp.credentials-info"
+
+METADATA_LOCATION_PROP = "metadata_location"
+PREVIOUS_METADATA_LOCATION_PROP = "previous_metadata_location"
+TABLE_TYPE_PROP = "table_type"
+ICEBERG_TABLE_TYPE_VALUE = "ICEBERG"
+
+HIVE_SERIALIZATION_LIBRARY = "org.apache.iceberg.mr.hive.HiveIcebergSerDe"
Review Comment:
@rambleraptor Could you remove all HIVE related code. I dont think you need these.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Add BigQuery Metastore Catalog [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on PR #2068:
URL: https://github.com/apache/iceberg-python/pull/2068#issuecomment-3212149820
I looks like some tests are still running:
```
> ???
E OSError: google::cloud::Status(UNAVAILABLE: Retry policy exhausted, with a last message of Could not create a OAuth2 access token to authenticate the request. The request was not sent, as such an access token is required to complete the request successfully. Learn more about Google Cloud authentication at https://cloud.google.com/docs/authentication. The underlying error message was: PerformWork() - CURL error [6]=Couldn't resolve host name error_info={reason=, domain=, metadata={gcloud-cpp.retry.on-entry=false, gcloud-cpp.retry.function=GetObjectMetadata, gcloud-cpp.retry.reason=retry-policy-exhausted, gcloud-cpp.retry.original-message=Could not create a OAuth2 access token to authenticate the request. The request was not sent, as such an access token is required to complete the request successfully. Learn more about Google Cloud authentication at https://cloud.google.com/docs/authentication. The underlying error message was: PerformWork() - CURL error [6]=Couldn't resolve ho
st name}})
pyarrow/error.pxi:92: OSError
------------------------------ Captured log setup ------------------------------
INFO werkzeug:_internal.py:97 127.0.0.1 - - [21/Aug/2025 21:17:50] "PUT /test_bucket HTTP/1.1" 200 -
=========================== short test summary info ============================
FAILED tests/catalog/test_bigquery_metastore.py::test_create_table_with_database_location - OSError: google::cloud::Status(UNAVAILABLE: Retry policy exhausted, with a last message of Could not create a OAuth2 access token to authenticate the request. The request was not sent, as such an access token is required to complete the request successfully. Learn more about Google Cloud authentication at https://cloud.google.com/docs/authentication. The underlying error message was: PerformWork() - CURL error [6]=Couldn't resolve host name error_info={reason=, domain=, metadata={gcloud-cpp.retry.on-entry=false, gcloud-cpp.retry.function=GetObjectMetadata, gcloud-cpp.retry.reason=retry-policy-exhausted, gcloud-cpp.retry.original-message=Could not create a OAuth2 access token to authenticate the request. The request was not sent, as such an access token is required to complete the request successfully. Learn more about Google Cloud authentication at https://cloud.google.com/docs/authenticati
on. The underlying error message was: PerformWork() - CURL error [6]=Couldn't resolve host name}})
FAILED tests/catalog/test_bigquery_metastore.py::test_drop_table_with_database_location - OSError: google::cloud::Status(UNAVAILABLE: Retry policy exhausted, with a last message of Could not create a OAuth2 access token to authenticate the request. The request was not sent, as such an access token is required to complete the request successfully. Learn more about Google Cloud authentication at https://cloud.google.com/docs/authentication. The underlying error message was: PerformWork() - CURL error [6]=Couldn't resolve host name error_info={reason=, domain=, metadata={gcloud-cpp.retry.on-entry=false, gcloud-cpp.retry.function=GetObjectMetadata, gcloud-cpp.retry.reason=retry-policy-exhausted, gcloud-cpp.retry.original-message=Could not create a OAuth2 access token to authenticate the request. The request was not sent, as such an access token is required to complete the request successfully. Learn more about Google Cloud authentication at https://cloud.google.com/docs/authentication
. The underlying error message was: PerformWork() - CURL error [6]=Couldn't resolve host name}})
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org