You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "kevinjqliu (via GitHub)" <gi...@apache.org> on 2024/06/12 00:06:04 UTC
[I] Consolidate catalog behavior [iceberg-python]
kevinjqliu opened a new issue, #813:
URL: https://github.com/apache/iceberg-python/issues/813
### Feature Request / Improvement
We should ensure consistency in catalog behaviors for all implementations.
Currently, we have a few different catalog implementations (dynamodb, glue, hive, rest, and sql).
Each catalog implementation supports a set of functions described in the Catalog ABC class (https://github.com/apache/iceberg-python/blob/main/pyiceberg/catalog/__init__.py#L276)
The behavior for each catalog implementation should be standardized. And we should test that a set of behaviors is consistent across all catalog implementations.
Here are a few related issues
* Rest Catalog: catalog.name should not be part of namespace #742
* SqlCatalog and other catalog works different on load_namespace_properties #769
* Add table_exists method to the Catalog #507
Add SqlCatalog to test console #488 is an example PR that utilizes `tests/cli/test_console.py` to ensure the same behaviors for different catalogs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Consolidate catalog behavior [iceberg-python]
Posted by "jayceslesar (via GitHub)" <gi...@apache.org>.
jayceslesar commented on issue #813:
URL: https://github.com/apache/iceberg-python/issues/813#issuecomment-2960856449
We can delete mountains of code if this is done correctly....
What about a strategy of each test_catalog providing a suite of basic fixtures that can be passed into a parameterized suite of functions?
Something like...
```py
@mock_aws
@pytest.fixture
def catalog_list_tables_fixture(...)
...
```
Like the glue test, and dynamo test (ect) could all provide a fixture like that? Unsure the best architecture but I think doing it correctly can vastly reduce the amount of code in the catalog test suite
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Consolidate catalog behavior [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #813:
URL: https://github.com/apache/iceberg-python/issues/813#issuecomment-3094736425
Got a good framework started in #2090 thanks to @.jayceslesar
Now we can add more tests, more catalog types and maybe even remove some redundant tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Consolidate catalog behavior [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #813:
URL: https://github.com/apache/iceberg-python/issues/813#issuecomment-3762319155
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Consolidate catalog behavior [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #813:
URL: https://github.com/apache/iceberg-python/issues/813#issuecomment-3826704880
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Consolidate catalog behavior [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #813:
URL: https://github.com/apache/iceberg-python/issues/813#issuecomment-2652346865
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Consolidate catalog behavior [iceberg-python]
Posted by "jayceslesar (via GitHub)" <gi...@apache.org>.
jayceslesar commented on issue #813:
URL: https://github.com/apache/iceberg-python/issues/813#issuecomment-2972869412
@kevinjqliu what do you think about something like this...
```py
from unittest.mock import MagicMock
import pytest
from pyiceberg.catalog import Catalog
from tests.conftest import BUCKET_NAME
HIVE_CATALOG_NAME = "hive"
HIVE_METASTORE_FAKE_URL = "thrift://unknown:9083"
@pytest.fixture
def glue(_bucket_initialize: None, moto_endpoint_url: str) -> Catalog:
from pyiceberg.catalog.glue import GlueCatalog
catalog = GlueCatalog(
name="glue",
bucket_name=BUCKET_NAME,
endpoint_url=moto_endpoint_url,
)
catalog.create_namespace("foobar")
return catalog
@pytest.fixture
def hive() -> Catalog:
from pyiceberg.catalog.hive import HiveCatalog
catalog = HiveCatalog(
name="hive",
uri=HIVE_METASTORE_FAKE_URL,
)
catalog._client = MagicMock()
catalog._client.__enter__().get_all_databases.return_value = ["foobar"]
return catalog
@pytest.mark.parametrize("catalog", [
pytest.lazy_fixture("glue"),
pytest.lazy_fixture("hive")
])
def test_list_namespaces_common(catalog: Catalog) -> None:
namespaces = catalog.list_namespaces()
assert namespaces == [("foobar",)]
```
I think it becomes more of a configuration workflow than a "catalog" workflow if that makes sense?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Consolidate catalog behavior [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #813:
URL: https://github.com/apache/iceberg-python/issues/813#issuecomment-2218117964
@rohithkanchukatla sure thing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Consolidate catalog behavior [iceberg-python]
Posted by "jayceslesar (via GitHub)" <gi...@apache.org>.
jayceslesar commented on issue #813:
URL: https://github.com/apache/iceberg-python/issues/813#issuecomment-2972852553
I think using the CLI is pretty wise as well @kevinjqliu
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Consolidate catalog behavior [iceberg-python]
Posted by "rohithkanchukatla (via GitHub)" <gi...@apache.org>.
rohithkanchukatla commented on issue #813:
URL: https://github.com/apache/iceberg-python/issues/813#issuecomment-2217933748
Hii Kevin, Can I take up this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Consolidate catalog behavior [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #813: Consolidate catalog behavior
URL: https://github.com/apache/iceberg-python/issues/813
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Consolidate catalog behavior [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #813:
URL: https://github.com/apache/iceberg-python/issues/813#issuecomment-2290141810
> The behavior for each catalog implementation should be standardized. And we should test that a set of behaviors is consistent across all catalog implementations.
@soumya-ghosh I want to keep this issue open until we have a solution to address future updates. Ideally, there's a set of tests that will check the behavior of all the catalog implementations.
> Add SqlCatalog to test console https://github.com/apache/iceberg-python/pull/488 is an example PR that utilizes tests/cli/test_console.py to ensure the same behaviors for different catalogs.
A possible solution is to test the input and expected output via the CLI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Consolidate catalog behavior [iceberg-python]
Posted by "soumya-ghosh (via GitHub)" <gi...@apache.org>.
soumya-ghosh commented on issue #813:
URL: https://github.com/apache/iceberg-python/issues/813#issuecomment-2288583740
@kevinjqliu this can be closed as well as all tasks in this issue are completed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org