You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "JasperHG90 (via GitHub)" <gi...@apache.org> on 2024/11/22 21:22:15 UTC
[I] `table_exists()` method does not work properly with Polaris catalog [iceberg-python]
JasperHG90 opened a new issue, #1363:
URL: https://github.com/apache/iceberg-python/issues/1363
### Apache Iceberg version
0.8.0 (latest release)
### Please describe the bug 🐞
Hi 👋
I see quite a few issues/PRs centered around the `table_exists()` method when using REST catalogs, e.g.
- https://github.com/apache/iceberg-python/pull/1096
- https://github.com/apache/iceberg-python/issues/1018
- https://github.com/apache/iceberg-python/issues/1006
- https://github.com/apache/iceberg-python/issues/735
When using Polars OSS, I also experience issues with this method. Even though a table exists the `catalog.table_exists()` method will return 'False'.
I have a devcontainer setup that you can find [here](https://github.com/JasperHG90/dagster-pyiceberg-example-polaris) for the example below if that helps.
Python code:
```python
#%%
import httpx
import polars as pl
from pyiceberg.catalog import load_catalog
# Assumes this catalog exists and credentials stored in
# .pyiceberg.yaml
# Possibly created using https://github.com/JasperHG90/dagster-pyiceberg-example-polaris/blob/main/scripts/set_up_catalog.py
catalog = load_catalog(
name="dagster_example_catalog",
)
# %% Create a namespace
catalog.create_namespace("example")
# %% Create data and add to table
df = pl.from_dict({
"a": [1, 2, 3],
"b": [4, 5, 6]
}).to_arrow()
catalog.create_table(
identifier="example.table",
schema=df.schema
)
# %% This will return 'False'
catalog.table_exists("example.table")
# %% But querying the 'table exists' Polaris endpoint returns 200
catalog_client = httpx.Client(
base_url="http://polaris:8181/api/catalog/v1",
headers={
"Authorization": "Bearer principal:root;realm:default-realm",
"Content-Type": "application/json",
}
)
# See endpoint here: https://polaris.apache.org/in-dev/unreleased/rest-catalog-open-api/#tag/Catalog-API/operation/tableExists
tables = catalog_client.get(f"dagster_example_catalog/namespaces/example/tables").json()
resp = catalog_client.head(
url="dagster_example_catalog/namespaces/example/tables/table"
)
resp.status_code # 200
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] `catalog.table_exists()` returns 'False' when table exists in Polaris catalog [iceberg-python]
Posted by "JasperHG90 (via GitHub)" <gi...@apache.org>.
JasperHG90 closed issue #1363: `catalog.table_exists()` returns 'False' when table exists in Polaris catalog
URL: https://github.com/apache/iceberg-python/issues/1363
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] `catalog.table_exists()` returns 'False' when table exists in Polaris catalog [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on issue #1363:
URL: https://github.com/apache/iceberg-python/issues/1363#issuecomment-2497462825
To make PyIceberg a bit friendlier to the catalogs, we could also allow for HTTP 200
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] `catalog.table_exists()` returns 'False' when table exists in Polaris catalog [iceberg-python]
Posted by "JasperHG90 (via GitHub)" <gi...@apache.org>.
JasperHG90 commented on issue #1363:
URL: https://github.com/apache/iceberg-python/issues/1363#issuecomment-2495318937
I've annotated this method with what I see in the debugger
```python
@retry(**_RETRY_ARGS)
def table_exists(self, identifier: Union[str, Identifier]) -> bool:
"""Check if a table exists.
Args:
identifier (str | Identifier): Table identifier.
Returns:
bool: True if the table exists, False otherwise.
"""
identifier_tuple = self._identifier_to_tuple_without_catalog(identifier) # ('example', 'test')
# Url: 'http://polaris:8181/api/catalog/v1/dagster_example_catalog/namespaces/example/tables/table'
response = self._session.head(
self.url(Endpoints.load_table, prefixed=True, **self._split_identifier_for_path(identifier_tuple))
)
# Response.status_code is 200
# This code block is skipped
if response.status_code == 404:
return False
elif response.status_code == 204:
return True
try:
response.raise_for_status()
except HTTPError as exc:
self._handle_non_200_response(exc, {})
# And 'False' is returned
return False
```
Polaris is returning a <200> status code, so that's not correct according to your spec.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] `catalog.table_exists()` returns 'False' when table exists in Polaris catalog [iceberg-python]
Posted by "JasperHG90 (via GitHub)" <gi...@apache.org>.
JasperHG90 commented on issue #1363:
URL: https://github.com/apache/iceberg-python/issues/1363#issuecomment-2504537919
Tested and works. Closing this issue. Thanks @kevinjqliu !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] `catalog.table_exists()` returns 'False' when table exists in Polaris catalog [iceberg-python]
Posted by "JasperHG90 (via GitHub)" <gi...@apache.org>.
JasperHG90 commented on issue #1363:
URL: https://github.com/apache/iceberg-python/issues/1363#issuecomment-2497134805
FYI PR that fixes status code for this and other endpoints: https://github.com/apache/polaris/pull/472
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] `catalog.table_exists()` returns 'False' when table exists in Polaris catalog [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #1363:
URL: https://github.com/apache/iceberg-python/issues/1363#issuecomment-2495556035
Thanks @JasperHG90 for the breakdown. The server spec above describes what the server should send and what the client should expect.
According to the spec, Polaris should be sending a 204 for success, not 200.
Do you mind opening an issue with Polaris?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] `catalog.table_exists()` returns 'False' when table exists in Polaris catalog [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #1363:
URL: https://github.com/apache/iceberg-python/issues/1363#issuecomment-2504515376
https://github.com/apache/polaris/pull/472 is merged @JasperHG90 do you mind testing it again with the latest code?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] `catalog.table_exists()` returns 'False' when table exists in Polaris catalog [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #1363:
URL: https://github.com/apache/iceberg-python/issues/1363#issuecomment-2504544750
cc @djouallah from https://github.com/apache/iceberg-python/issues/1018#issuecomment-2471827257
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] `catalog.table_exists()` returns 'False' when table exists in Polaris catalog [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #1363:
URL: https://github.com/apache/iceberg-python/issues/1363#issuecomment-2494980924
Thank you for bringing this up. Its been on my todo list to investigate this as part of https://github.com/apache/iceberg-python/issues/1018#issuecomment-2471827257
can you help me go through the same exercise and step through the `table_exists` call? In #1018, you can see the response code is 404, which should return `False` according to [the implementation](https://github.com/apache/iceberg-python/blob/63169004facb053d3ac3725388411ee7a05c4156/pyiceberg/catalog/rest.py#L874-L898) and [the spec](https://github.com/apache/iceberg/blob/97e034b2cec9408a6f792c410a8eb8dddb452e14/open-api/rest-catalog-open-api.yaml#L833-L864)
I want to see if this is a polaris issue or pyiceberg issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org