You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "Fokko (via GitHub)" <gi...@apache.org> on 2024/08/28 21:37:12 UTC
[PR] Make `commit_table` public [iceberg-python]
Fokko opened a new pull request, #1112:
URL: https://github.com/apache/iceberg-python/pull/1112
This was brought up earlier, and I think this is a good idea. I changed the signature to also pass in the table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Make `commit_table` public [iceberg-python]
Posted by "sungwy (via GitHub)" <gi...@apache.org>.
sungwy commented on PR #1112:
URL: https://github.com/apache/iceberg-python/pull/1112#issuecomment-2326240075
Hi @Fokko
Just a heads up - sorry for adding in a merge conflict here. I merged in https://github.com/apache/iceberg-python/pull/820 first because it had been open for quite some time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Make `commit_table` public [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #1112:
URL: https://github.com/apache/iceberg-python/pull/1112#discussion_r1737130122
##########
pyiceberg/catalog/glue.py:
##########
@@ -462,10 +468,8 @@ def _commit_table(self, table_request: CommitTableRequest) -> CommitTableRespons
NoSuchTableError: If a table with the given identifier does not exist.
CommitFailedException: Requirement not met, or a conflict with a concurrent commit.
"""
- identifier_tuple = self._identifier_to_tuple_without_catalog(
- tuple(table_request.identifier.namespace.root + [table_request.identifier.name])
- )
- database_name, table_name = self.identifier_to_database_and_table(identifier_tuple)
+ table_identifier = self.identifier_to_tuple_without_catalog(table.identifier)
Review Comment:
Good catch, I've switched to the private method instead 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Make `commit_table` public [iceberg-python]
Posted by "sungwy (via GitHub)" <gi...@apache.org>.
sungwy merged PR #1112:
URL: https://github.com/apache/iceberg-python/pull/1112
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Make `commit_table` public [iceberg-python]
Posted by "sungwy (via GitHub)" <gi...@apache.org>.
sungwy commented on code in PR #1112:
URL: https://github.com/apache/iceberg-python/pull/1112#discussion_r1736166915
##########
pyiceberg/catalog/hive.py:
##########
@@ -445,7 +449,9 @@ def _commit_table(self, table_request: CommitTableRequest) -> CommitTableRespons
if lock.state == LockState.WAITING:
self._wait_for_lock(database_name, table_name, lock.lockid, open_client)
else:
- raise CommitFailedException(f"Failed to acquire lock for {table_request.identifier}, state: {lock.state}")
+ raise CommitFailedException(
+ f"Failed to acquire lock for {table_identifier.identifier}, state: {lock.state}"
Review Comment:
```suggestion
f"Failed to acquire lock for {table_identifier}, state: {lock.state}"
```
##########
pyiceberg/catalog/glue.py:
##########
@@ -462,10 +468,8 @@ def _commit_table(self, table_request: CommitTableRequest) -> CommitTableRespons
NoSuchTableError: If a table with the given identifier does not exist.
CommitFailedException: Requirement not met, or a conflict with a concurrent commit.
"""
- identifier_tuple = self._identifier_to_tuple_without_catalog(
- tuple(table_request.identifier.namespace.root + [table_request.identifier.name])
- )
- database_name, table_name = self.identifier_to_database_and_table(identifier_tuple)
+ table_identifier = self.identifier_to_tuple_without_catalog(table.identifier)
Review Comment:
How do you feel about using the private equivalent method `self._identifier_to_tuple_without_catalog` instead?
The public method will result in commit_table always throwing a deprecation warning, whereas the private method will throw a warning only if the catalog name is included in table.identifier.
https://github.com/apache/iceberg-python/blob/ba8e9a368a8c1a2798b14c3ebdca653a1c2d52b5/pyiceberg/catalog/__init__.py#L635-L652
My apologies in advance that having two similarly named methods is confusing... But given that `identifier_to_tuple_without_catalog` was already a public method, we didn't have much choice but to have these two versions of the method until 0.9.0 so that each `_commit_table` didn't raise a deprecation warning :)
##########
pyiceberg/catalog/hive.py:
##########
@@ -486,7 +492,7 @@ def _commit_table(self, table_request: CommitTableRequest) -> CommitTableRespons
)
self._create_hive_table(open_client, hive_table)
except WaitingForLockException as e:
- raise CommitFailedException(f"Failed to acquire lock for {table_request.identifier}, state: {lock.state}") from e
+ raise CommitFailedException(f"Failed to acquire lock for {table.identifier}, state: {lock.state}") from e
Review Comment:
```suggestion
raise CommitFailedException(f"Failed to acquire lock for {table_identifier}, state: {lock.state}") from e
```
##########
pyiceberg/catalog/rest.py:
##########
@@ -734,9 +739,11 @@ def _commit_table(self, table_request: CommitTableRequest) -> CommitTableRespons
CommitFailedException: Requirement not met, or a conflict with a concurrent commit.
CommitStateUnknownException: Failed due to an internal exception on the side of the catalog.
"""
+ identifier = TableIdentifier(namespace=table.identifier[1:-1], name=table.identifier[-1])
Review Comment:
Would using `self._identifier_to_tuple_without_catalog` until the use of catalog name is deprecated in 0.9.0 be safer than removing the first identifier in all cases?
##########
pyiceberg/catalog/rest.py:
##########
@@ -719,12 +721,15 @@ def _remove_catalog_name_from_table_request_identifier(self, table_request: Comm
)
return table_request
- @retry(**_RETRY_ARGS)
Review Comment:
Did we remove the retry decorator intentionally?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Make `commit_table` public [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #1112:
URL: https://github.com/apache/iceberg-python/pull/1112#discussion_r1739973689
##########
pyiceberg/table/__init__.py:
##########
@@ -1673,12 +1673,8 @@ def refs(self) -> Dict[str, SnapshotRef]:
return self.metadata.refs
def _do_commit(self, updates: Tuple[TableUpdate, ...], requirements: Tuple[TableRequirement, ...]) -> None:
- response = self.catalog._commit_table( # pylint: disable=W0212
- CommitTableRequest(
- identifier=TableIdentifier(namespace=self._identifier[:-1], name=self._identifier[-1]),
- updates=updates,
- requirements=requirements,
- )
+ response = self.catalog.commit_table( # pylint: disable=W0212
Review Comment:
Good one, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Make `commit_table` public [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on PR #1112:
URL: https://github.com/apache/iceberg-python/pull/1112#issuecomment-2331938676
@sungwy No problem at all, I've pulled in latest master 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Make `commit_table` public [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on code in PR #1112:
URL: https://github.com/apache/iceberg-python/pull/1112#discussion_r1739511962
##########
pyiceberg/table/__init__.py:
##########
@@ -1673,12 +1673,8 @@ def refs(self) -> Dict[str, SnapshotRef]:
return self.metadata.refs
def _do_commit(self, updates: Tuple[TableUpdate, ...], requirements: Tuple[TableRequirement, ...]) -> None:
- response = self.catalog._commit_table( # pylint: disable=W0212
- CommitTableRequest(
- identifier=TableIdentifier(namespace=self._identifier[:-1], name=self._identifier[-1]),
- updates=updates,
- requirements=requirements,
- )
+ response = self.catalog.commit_table( # pylint: disable=W0212
+ self, requirements, updates
) # pylint: disable=W0212
Review Comment:
```suggestion
)
```
here as well
##########
pyiceberg/table/__init__.py:
##########
@@ -1673,12 +1673,8 @@ def refs(self) -> Dict[str, SnapshotRef]:
return self.metadata.refs
def _do_commit(self, updates: Tuple[TableUpdate, ...], requirements: Tuple[TableRequirement, ...]) -> None:
- response = self.catalog._commit_table( # pylint: disable=W0212
- CommitTableRequest(
- identifier=TableIdentifier(namespace=self._identifier[:-1], name=self._identifier[-1]),
- updates=updates,
- requirements=requirements,
- )
+ response = self.catalog.commit_table( # pylint: disable=W0212
Review Comment:
```suggestion
response = self.catalog.commit_table(
```
i think we can get rid of disable pylint here since its no longer a private method
https://pylint.readthedocs.io/en/latest/user_guide/messages/warning/protected-access.html
##########
pyiceberg/catalog/hive.py:
##########
@@ -434,10 +440,8 @@ def _commit_table(self, table_request: CommitTableRequest) -> CommitTableRespons
NoSuchTableError: If a table with the given identifier does not exist.
CommitFailedException: Requirement not met, or a conflict with a concurrent commit.
"""
- identifier_tuple = self._identifier_to_tuple_without_catalog(
- tuple(table_request.identifier.namespace.root + [table_request.identifier.name])
- )
- database_name, table_name = self.identifier_to_database_and_table(identifier_tuple, NoSuchTableError)
+ table_identifier = self._identifier_to_tuple_without_catalog(table.identifier)
+ database_name, table_name = table_identifier
Review Comment:
also maybe use `identifier_to_database_and_table`
##########
pyiceberg/catalog/glue.py:
##########
@@ -462,10 +468,8 @@ def _commit_table(self, table_request: CommitTableRequest) -> CommitTableRespons
NoSuchTableError: If a table with the given identifier does not exist.
CommitFailedException: Requirement not met, or a conflict with a concurrent commit.
"""
- identifier_tuple = self._identifier_to_tuple_without_catalog(
- tuple(table_request.identifier.namespace.root + [table_request.identifier.name])
- )
- database_name, table_name = self.identifier_to_database_and_table(identifier_tuple)
+ table_identifier = self._identifier_to_tuple_without_catalog(table.identifier)
+ database_name, table_name = table_identifier
Review Comment:
maybe use `identifier_to_database_and_table` here?
https://github.com/apache/iceberg-python/blob/c647256b1d9a278a88f5e757bde4db76017d47fd/pyiceberg/catalog/glue.py#L549
##########
pyiceberg/catalog/rest.py:
##########
@@ -755,9 +750,12 @@ def _commit_table(self, table_request: CommitTableRequest) -> CommitTableRespons
CommitFailedException: Requirement not met, or a conflict with a concurrent commit.
CommitStateUnknownException: Failed due to an internal exception on the side of the catalog.
"""
+ identifier = self._identifier_to_tuple_without_catalog(table.identifier)
+ table_identifier = TableIdentifier(namespace=identifier[0:-1], name=identifier[-1])
Review Comment:
```suggestion
table_identifier = TableIdentifier(namespace=identifier[:-1], name=identifier[-1])
```
nit: more pythonic and also mirrors
https://github.com/apache/iceberg-python/blob/e4c1748fee220076f04e35ab2f182dd51ca20705/pyiceberg/table/__init__.py#L1678
##########
pyiceberg/catalog/rest.py:
##########
@@ -773,6 +771,17 @@ def _commit_table(self, table_request: CommitTableRequest) -> CommitTableRespons
)
return CommitTableResponse(**response.json())
+ @retry(**_RETRY_ARGS)
+ def list_views(self, namespace: Union[str, Identifier]) -> List[Identifier]:
Review Comment:
nit: move `list_views` above `commit_table` for better github diff
##########
tests/catalog/test_base.py:
##########
@@ -670,14 +670,13 @@ def test_commit_table(catalog: InMemoryCatalog) -> None:
)
# When
- response = given_table.catalog._commit_table( # pylint: disable=W0212
- CommitTableRequest(
- identifier=TableIdentifier(namespace=Namespace(given_table._identifier[:-1]), name=given_table._identifier[-1]),
- updates=[
- AddSchemaUpdate(schema=new_schema, last_column_id=new_schema.highest_field_id),
- SetCurrentSchemaUpdate(schema_id=-1),
- ],
- )
+ response = given_table.catalog.commit_table( # pylint: disable=W0212
Review Comment:
```suggestion
response = given_table.catalog.commit_table(
```
can remove disable pylint
##########
pyiceberg/catalog/sql.py:
##########
@@ -407,20 +412,18 @@ def _commit_table(self, table_request: CommitTableRequest) -> CommitTableRespons
NoSuchTableError: If a table with the given identifier does not exist.
CommitFailedException: Requirement not met, or a conflict with a concurrent commit.
"""
- identifier_tuple = self._identifier_to_tuple_without_catalog(
- tuple(table_request.identifier.namespace.root + [table_request.identifier.name])
- )
- namespace_tuple = Catalog.namespace_from(identifier_tuple)
+ table_identifier = self._identifier_to_tuple_without_catalog(table.identifier)
+ namespace_tuple = Catalog.namespace_from(table_identifier)
namespace = Catalog.namespace_to_string(namespace_tuple)
- table_name = Catalog.table_name_from(identifier_tuple)
+ table_name = Catalog.table_name_from(table_identifier)
current_table: Optional[Table]
try:
- current_table = self.load_table(identifier_tuple)
+ current_table = self.load_table(table_identifier)
except NoSuchTableError:
current_table = None
- updated_staged_table = self._update_and_stage_table(current_table, table_request)
+ updated_staged_table = self._update_and_stage_table(current_table, table.identifier[1:], requirements, updates)
Review Comment:
```suggestion
updated_staged_table = self._update_and_stage_table(current_table, table.identifier, requirements, updates)
```
can just use `table_identifier` here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Make `commit_table` public [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #1112:
URL: https://github.com/apache/iceberg-python/pull/1112#discussion_r1739973551
##########
pyiceberg/catalog/rest.py:
##########
@@ -773,6 +771,17 @@ def _commit_table(self, table_request: CommitTableRequest) -> CommitTableRespons
)
return CommitTableResponse(**response.json())
+ @retry(**_RETRY_ARGS)
+ def list_views(self, namespace: Union[str, Identifier]) -> List[Identifier]:
Review Comment:
Thanks, and done! This one caused a merge conflicts when pulling in main while working on this PR 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Make `commit_table` public [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #1112:
URL: https://github.com/apache/iceberg-python/pull/1112#discussion_r1737129443
##########
pyiceberg/catalog/rest.py:
##########
@@ -719,12 +721,15 @@ def _remove_catalog_name_from_table_request_identifier(self, table_request: Comm
)
return table_request
- @retry(**_RETRY_ARGS)
Review Comment:
No, I did this earlier when I was trying another route. Thanks for catching this :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Make `commit_table` public [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #1112:
URL: https://github.com/apache/iceberg-python/pull/1112#discussion_r1737194124
##########
pyiceberg/catalog/rest.py:
##########
@@ -734,9 +739,11 @@ def _commit_table(self, table_request: CommitTableRequest) -> CommitTableRespons
CommitFailedException: Requirement not met, or a conflict with a concurrent commit.
CommitStateUnknownException: Failed due to an internal exception on the side of the catalog.
"""
+ identifier = TableIdentifier(namespace=table.identifier[1:-1], name=table.identifier[-1])
Review Comment:
Thanks, yes, this was still work in progress 🙈
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Make `commit_table` public [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #1112:
URL: https://github.com/apache/iceberg-python/pull/1112#discussion_r1739974152
##########
pyiceberg/catalog/sql.py:
##########
@@ -407,20 +412,18 @@ def _commit_table(self, table_request: CommitTableRequest) -> CommitTableRespons
NoSuchTableError: If a table with the given identifier does not exist.
CommitFailedException: Requirement not met, or a conflict with a concurrent commit.
"""
- identifier_tuple = self._identifier_to_tuple_without_catalog(
- tuple(table_request.identifier.namespace.root + [table_request.identifier.name])
- )
- namespace_tuple = Catalog.namespace_from(identifier_tuple)
+ table_identifier = self._identifier_to_tuple_without_catalog(table.identifier)
+ namespace_tuple = Catalog.namespace_from(table_identifier)
namespace = Catalog.namespace_to_string(namespace_tuple)
- table_name = Catalog.table_name_from(identifier_tuple)
+ table_name = Catalog.table_name_from(table_identifier)
current_table: Optional[Table]
try:
- current_table = self.load_table(identifier_tuple)
+ current_table = self.load_table(table_identifier)
except NoSuchTableError:
current_table = None
- updated_staged_table = self._update_and_stage_table(current_table, table_request)
+ updated_staged_table = self._update_and_stage_table(current_table, table.identifier[1:], requirements, updates)
Review Comment:
I like that very much, thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org