You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "ccancellieri (via GitHub)" <gi...@apache.org> on 2025/06/19 16:26:38 UTC
[I] Google BigLake Metastore Catalog issue [iceberg-python]
ccancellieri opened a new issue, #2122:
URL: https://github.com/apache/iceberg-python/issues/2122
Dear all,
I'm working on a GCP environment and I'm configuring pyIceberg to work over the BigLake API Metastore catalog.
I'm pretty satisfied of the result (it almost works!) but I've a blocking issue that prevent me to instanciate the Catalog.
The issue is located here:
https://github.com/apache/iceberg-python/blob/f71806ee816cf0fb1e7f785aec81932741a0c6ca/pyiceberg/catalog/rest/__init__.py#L181
Pydantic in fact validates the output of the Catalog requiring a mandatory field called "defaults".
This is unfortunately NOT returned by the BigLake catalog and the result is that we're not able to correctly instantiate the catalog.
I'm now testing the catalog using the following configuration:
`
config = {
"type": "rest",
"uri": "https://biglake.googleapis.com/iceberg/v1beta/restcatalog",
"warehouse": gcs_warehouse_path,
"py-io-impl": "pyiceberg.io.pyarrow.PyArrowFileIO", # Crucial for GCS
"rest-metrics-reporting-enabled": "false", # Disable metrics reporting if not needed
"oauth2-server-uri": "https://oauth2.googleapis.com/token",
"token": access_token,
"header.x-goog-user-project": biglake_project_id,
# Optional: Set the logging level for pyiceberg if you need more debug info
"pyiceberg.logging-level": "DEBUG",
}
`
For this reason instead of forking I would like to ask to apply the followin fix if possible:
```
class ConfigResponse(IcebergBaseModel):
defaults: Optional[Properties] = Field(default={})
overrides: Properties = Field()
```
This will allow _fetch_config() to not fail while passing the response.json() to the ConfigResponse constructor [here](https://github.com/apache/iceberg-python/blob/f71806ee816cf0fb1e7f785aec81932741a0c6ca/pyiceberg/catalog/rest/__init__.py#L353):
```
def _fetch_config(self) -> None:
params = {}
if warehouse_location := self.properties.get(WAREHOUSE_LOCATION):
params[WAREHOUSE_LOCATION] = warehouse_location
with self._create_session() as session:
response = session.get(self.url(Endpoints.get_config, prefixed=False), params=params)
try:
response.raise_for_status()
except HTTPError as exc:
self._handle_non_200_response(exc, {})
config_response = ConfigResponse(**response.json())
config = config_response.defaults
config.update(self.properties)
config.update(config_response.overrides)
self.properties = config
```
Doing this I'm able to have a working BigLake catalog and all the calls are working now.
_Another issue is that list_namespaces() and list_tables() are failing in a similar way since BigLake is not returning an empty list but we could survive catching the exception and creating the first namespace and table, this works and after that all the calls are working fine._
I'm not sure about the Iceberg spec but I hope we could apply the suggested fix so we will be able to use pyIceberg with no issue also in GCP!!!
Thanks all.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Google BigLake Metastore Catalog issue [iceberg-python]
Posted by "ccancellieri (via GitHub)" <gi...@apache.org>.
ccancellieri commented on issue #2122:
URL: https://github.com/apache/iceberg-python/issues/2122#issuecomment-3004147632
I'm in contact with the BigLake Google developers and they are keen to work to make this happen, anyhow I hope we could relax a bit the API to make it start working.
The empty object are pruned by the google api for performance reasons but as said they are trying to make a step into the standard direction.
I hope this simple pull request could be accepted so we can start adopting this cool and light library instead of having to leverage on the sedona java bindings (which is the unique implementation provided by google).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Google BigLake Metastore Catalog issue [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #2122:
URL: https://github.com/apache/iceberg-python/issues/2122#issuecomment-3005241028
@ccancellieri thanks for raising this! We've seen similar issues from other Iceberg REST Catalog implementations (such as Nessie). I think its a good idea to relax the constraint when parsing the response from v1/config.
I made a few changes to #2148
> Another issue is that list_namespaces() and list_tables() are failing in a similar way since BigLake is not returning an empty list but we could survive catching the exception and creating the first namespace and table, this works and after that all the calls are working fine.
I'm hesitant to change it for `list_namespaces` and `list_tables`. BigLake's REST catalog API should be fixed to adhere to the Iceberg Rest Spec.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Google BigLake Metastore Catalog issue [iceberg-python]
Posted by "jayceslesar (via GitHub)" <gi...@apache.org>.
jayceslesar commented on issue #2122:
URL: https://github.com/apache/iceberg-python/issues/2122#issuecomment-2989059846
Not unique to BigLake but rather the rest implementation here is definitely behind the upstream definition https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml?ref=chaosgenius.io
Maybe its worth gathering all the issues into a larger suite of tasks
1. https://github.com/apache/iceberg-python/issues/2084
2. https://github.com/apache/iceberg-python/issues/2096
3. There are probably others
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Google BigLake Metastore Catalog issue [iceberg-python]
Posted by "unfrgivn (via GitHub)" <gi...@apache.org>.
unfrgivn commented on issue #2122:
URL: https://github.com/apache/iceberg-python/issues/2122#issuecomment-2989718202
I just mentioned this over here https://github.com/apache/iceberg-python/issues/1524#issuecomment-2989705907 (since that's the top hit on Google when searching this error).
Anyone looking to use the new BigLake REST catalog should provide feedback to GCP since this service is still in public preview.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org