You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "Fokko (via GitHub)" <gi...@apache.org> on 2024/08/28 21:37:12 UTC

[PR] Make `commit_table` public [iceberg-python]

Fokko opened a new pull request, #1112:
URL: https://github.com/apache/iceberg-python/pull/1112

   This was brought up earlier, and I think this is a good idea. I changed the signature to also pass in the table.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Make `commit_table` public [iceberg-python]

Posted by "sungwy (via GitHub)" <gi...@apache.org>.
sungwy commented on PR #1112:
URL: https://github.com/apache/iceberg-python/pull/1112#issuecomment-2326240075

   Hi @Fokko 
   
   Just a heads up - sorry for adding in a merge conflict here. I merged in https://github.com/apache/iceberg-python/pull/820 first because it had been open for quite some time.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Make `commit_table` public [iceberg-python]

Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #1112:
URL: https://github.com/apache/iceberg-python/pull/1112#discussion_r1737130122


##########
pyiceberg/catalog/glue.py:
##########
@@ -462,10 +468,8 @@ def _commit_table(self, table_request: CommitTableRequest) -> CommitTableRespons
             NoSuchTableError: If a table with the given identifier does not exist.
             CommitFailedException: Requirement not met, or a conflict with a concurrent commit.
         """
-        identifier_tuple = self._identifier_to_tuple_without_catalog(
-            tuple(table_request.identifier.namespace.root + [table_request.identifier.name])
-        )
-        database_name, table_name = self.identifier_to_database_and_table(identifier_tuple)
+        table_identifier = self.identifier_to_tuple_without_catalog(table.identifier)

Review Comment:
   Good catch, I've switched to the private method instead 👍 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Make `commit_table` public [iceberg-python]

Posted by "sungwy (via GitHub)" <gi...@apache.org>.
sungwy merged PR #1112:
URL: https://github.com/apache/iceberg-python/pull/1112


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Make `commit_table` public [iceberg-python]

Posted by "sungwy (via GitHub)" <gi...@apache.org>.
sungwy commented on code in PR #1112:
URL: https://github.com/apache/iceberg-python/pull/1112#discussion_r1736166915


##########
pyiceberg/catalog/hive.py:
##########
@@ -445,7 +449,9 @@ def _commit_table(self, table_request: CommitTableRequest) -> CommitTableRespons
                     if lock.state == LockState.WAITING:
                         self._wait_for_lock(database_name, table_name, lock.lockid, open_client)
                     else:
-                        raise CommitFailedException(f"Failed to acquire lock for {table_request.identifier}, state: {lock.state}")
+                        raise CommitFailedException(
+                            f"Failed to acquire lock for {table_identifier.identifier}, state: {lock.state}"

Review Comment:
   ```suggestion
                               f"Failed to acquire lock for {table_identifier}, state: {lock.state}"
   ```



##########
pyiceberg/catalog/glue.py:
##########
@@ -462,10 +468,8 @@ def _commit_table(self, table_request: CommitTableRequest) -> CommitTableRespons
             NoSuchTableError: If a table with the given identifier does not exist.
             CommitFailedException: Requirement not met, or a conflict with a concurrent commit.
         """
-        identifier_tuple = self._identifier_to_tuple_without_catalog(
-            tuple(table_request.identifier.namespace.root + [table_request.identifier.name])
-        )
-        database_name, table_name = self.identifier_to_database_and_table(identifier_tuple)
+        table_identifier = self.identifier_to_tuple_without_catalog(table.identifier)

Review Comment:
   How do you feel about using the private equivalent method `self._identifier_to_tuple_without_catalog` instead?
   
   The public method will result in commit_table always throwing a deprecation warning, whereas the private method will throw a warning only if the catalog name is included in table.identifier.
   
   https://github.com/apache/iceberg-python/blob/ba8e9a368a8c1a2798b14c3ebdca653a1c2d52b5/pyiceberg/catalog/__init__.py#L635-L652
   
   My apologies in advance that having two similarly named methods is confusing... But given that `identifier_to_tuple_without_catalog` was already a public method, we didn't have much choice but to have these two versions of the method until 0.9.0 so that each `_commit_table` didn't raise a deprecation warning :)



##########
pyiceberg/catalog/hive.py:
##########
@@ -486,7 +492,7 @@ def _commit_table(self, table_request: CommitTableRequest) -> CommitTableRespons
                     )
                     self._create_hive_table(open_client, hive_table)
             except WaitingForLockException as e:
-                raise CommitFailedException(f"Failed to acquire lock for {table_request.identifier}, state: {lock.state}") from e
+                raise CommitFailedException(f"Failed to acquire lock for {table.identifier}, state: {lock.state}") from e

Review Comment:
   ```suggestion
                   raise CommitFailedException(f"Failed to acquire lock for {table_identifier}, state: {lock.state}") from e
   ```



##########
pyiceberg/catalog/rest.py:
##########
@@ -734,9 +739,11 @@ def _commit_table(self, table_request: CommitTableRequest) -> CommitTableRespons
             CommitFailedException: Requirement not met, or a conflict with a concurrent commit.
             CommitStateUnknownException: Failed due to an internal exception on the side of the catalog.
         """
+        identifier = TableIdentifier(namespace=table.identifier[1:-1], name=table.identifier[-1])

Review Comment:
   Would using `self._identifier_to_tuple_without_catalog` until the use of catalog name is deprecated in 0.9.0 be safer than removing the first identifier in all cases?



##########
pyiceberg/catalog/rest.py:
##########
@@ -719,12 +721,15 @@ def _remove_catalog_name_from_table_request_identifier(self, table_request: Comm
             )
         return table_request
 
-    @retry(**_RETRY_ARGS)

Review Comment:
   Did we remove the retry decorator intentionally?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Make `commit_table` public [iceberg-python]

Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #1112:
URL: https://github.com/apache/iceberg-python/pull/1112#discussion_r1739973689


##########
pyiceberg/table/__init__.py:
##########
@@ -1673,12 +1673,8 @@ def refs(self) -> Dict[str, SnapshotRef]:
         return self.metadata.refs
 
     def _do_commit(self, updates: Tuple[TableUpdate, ...], requirements: Tuple[TableRequirement, ...]) -> None:
-        response = self.catalog._commit_table(  # pylint: disable=W0212
-            CommitTableRequest(
-                identifier=TableIdentifier(namespace=self._identifier[:-1], name=self._identifier[-1]),
-                updates=updates,
-                requirements=requirements,
-            )
+        response = self.catalog.commit_table(  # pylint: disable=W0212

Review Comment:
   Good one, thanks!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Make `commit_table` public [iceberg-python]

Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on PR #1112:
URL: https://github.com/apache/iceberg-python/pull/1112#issuecomment-2331938676

   @sungwy No problem at all, I've pulled in latest master 👍 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Make `commit_table` public [iceberg-python]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on code in PR #1112:
URL: https://github.com/apache/iceberg-python/pull/1112#discussion_r1739511962


##########
pyiceberg/table/__init__.py:
##########
@@ -1673,12 +1673,8 @@ def refs(self) -> Dict[str, SnapshotRef]:
         return self.metadata.refs
 
     def _do_commit(self, updates: Tuple[TableUpdate, ...], requirements: Tuple[TableRequirement, ...]) -> None:
-        response = self.catalog._commit_table(  # pylint: disable=W0212
-            CommitTableRequest(
-                identifier=TableIdentifier(namespace=self._identifier[:-1], name=self._identifier[-1]),
-                updates=updates,
-                requirements=requirements,
-            )
+        response = self.catalog.commit_table(  # pylint: disable=W0212
+            self, requirements, updates
         )  # pylint: disable=W0212

Review Comment:
   ```suggestion
           )
   ```
   here as well



##########
pyiceberg/table/__init__.py:
##########
@@ -1673,12 +1673,8 @@ def refs(self) -> Dict[str, SnapshotRef]:
         return self.metadata.refs
 
     def _do_commit(self, updates: Tuple[TableUpdate, ...], requirements: Tuple[TableRequirement, ...]) -> None:
-        response = self.catalog._commit_table(  # pylint: disable=W0212
-            CommitTableRequest(
-                identifier=TableIdentifier(namespace=self._identifier[:-1], name=self._identifier[-1]),
-                updates=updates,
-                requirements=requirements,
-            )
+        response = self.catalog.commit_table(  # pylint: disable=W0212

Review Comment:
   ```suggestion
           response = self.catalog.commit_table(
   ```
   
   i think we can get rid of disable pylint here since its no longer a private method
   https://pylint.readthedocs.io/en/latest/user_guide/messages/warning/protected-access.html



##########
pyiceberg/catalog/hive.py:
##########
@@ -434,10 +440,8 @@ def _commit_table(self, table_request: CommitTableRequest) -> CommitTableRespons
             NoSuchTableError: If a table with the given identifier does not exist.
             CommitFailedException: Requirement not met, or a conflict with a concurrent commit.
         """
-        identifier_tuple = self._identifier_to_tuple_without_catalog(
-            tuple(table_request.identifier.namespace.root + [table_request.identifier.name])
-        )
-        database_name, table_name = self.identifier_to_database_and_table(identifier_tuple, NoSuchTableError)
+        table_identifier = self._identifier_to_tuple_without_catalog(table.identifier)
+        database_name, table_name = table_identifier

Review Comment:
   also maybe use `identifier_to_database_and_table` 



##########
pyiceberg/catalog/glue.py:
##########
@@ -462,10 +468,8 @@ def _commit_table(self, table_request: CommitTableRequest) -> CommitTableRespons
             NoSuchTableError: If a table with the given identifier does not exist.
             CommitFailedException: Requirement not met, or a conflict with a concurrent commit.
         """
-        identifier_tuple = self._identifier_to_tuple_without_catalog(
-            tuple(table_request.identifier.namespace.root + [table_request.identifier.name])
-        )
-        database_name, table_name = self.identifier_to_database_and_table(identifier_tuple)
+        table_identifier = self._identifier_to_tuple_without_catalog(table.identifier)
+        database_name, table_name = table_identifier

Review Comment:
   maybe use `identifier_to_database_and_table` here? 
   
   https://github.com/apache/iceberg-python/blob/c647256b1d9a278a88f5e757bde4db76017d47fd/pyiceberg/catalog/glue.py#L549



##########
pyiceberg/catalog/rest.py:
##########
@@ -755,9 +750,12 @@ def _commit_table(self, table_request: CommitTableRequest) -> CommitTableRespons
             CommitFailedException: Requirement not met, or a conflict with a concurrent commit.
             CommitStateUnknownException: Failed due to an internal exception on the side of the catalog.
         """
+        identifier = self._identifier_to_tuple_without_catalog(table.identifier)
+        table_identifier = TableIdentifier(namespace=identifier[0:-1], name=identifier[-1])

Review Comment:
   ```suggestion
           table_identifier = TableIdentifier(namespace=identifier[:-1], name=identifier[-1])
   ```
   nit: more pythonic and also mirrors
   
   https://github.com/apache/iceberg-python/blob/e4c1748fee220076f04e35ab2f182dd51ca20705/pyiceberg/table/__init__.py#L1678



##########
pyiceberg/catalog/rest.py:
##########
@@ -773,6 +771,17 @@ def _commit_table(self, table_request: CommitTableRequest) -> CommitTableRespons
             )
         return CommitTableResponse(**response.json())
 
+    @retry(**_RETRY_ARGS)
+    def list_views(self, namespace: Union[str, Identifier]) -> List[Identifier]:

Review Comment:
   nit: move `list_views` above `commit_table` for better github diff



##########
tests/catalog/test_base.py:
##########
@@ -670,14 +670,13 @@ def test_commit_table(catalog: InMemoryCatalog) -> None:
     )
 
     # When
-    response = given_table.catalog._commit_table(  # pylint: disable=W0212
-        CommitTableRequest(
-            identifier=TableIdentifier(namespace=Namespace(given_table._identifier[:-1]), name=given_table._identifier[-1]),
-            updates=[
-                AddSchemaUpdate(schema=new_schema, last_column_id=new_schema.highest_field_id),
-                SetCurrentSchemaUpdate(schema_id=-1),
-            ],
-        )
+    response = given_table.catalog.commit_table(  # pylint: disable=W0212

Review Comment:
   ```suggestion
       response = given_table.catalog.commit_table(
   ```
   can remove disable pylint



##########
pyiceberg/catalog/sql.py:
##########
@@ -407,20 +412,18 @@ def _commit_table(self, table_request: CommitTableRequest) -> CommitTableRespons
             NoSuchTableError: If a table with the given identifier does not exist.
             CommitFailedException: Requirement not met, or a conflict with a concurrent commit.
         """
-        identifier_tuple = self._identifier_to_tuple_without_catalog(
-            tuple(table_request.identifier.namespace.root + [table_request.identifier.name])
-        )
-        namespace_tuple = Catalog.namespace_from(identifier_tuple)
+        table_identifier = self._identifier_to_tuple_without_catalog(table.identifier)
+        namespace_tuple = Catalog.namespace_from(table_identifier)
         namespace = Catalog.namespace_to_string(namespace_tuple)
-        table_name = Catalog.table_name_from(identifier_tuple)
+        table_name = Catalog.table_name_from(table_identifier)
 
         current_table: Optional[Table]
         try:
-            current_table = self.load_table(identifier_tuple)
+            current_table = self.load_table(table_identifier)
         except NoSuchTableError:
             current_table = None
 
-        updated_staged_table = self._update_and_stage_table(current_table, table_request)
+        updated_staged_table = self._update_and_stage_table(current_table, table.identifier[1:], requirements, updates)

Review Comment:
   ```suggestion
           updated_staged_table = self._update_and_stage_table(current_table, table.identifier, requirements, updates)
   ```
   can just use `table_identifier` here



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Make `commit_table` public [iceberg-python]

Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #1112:
URL: https://github.com/apache/iceberg-python/pull/1112#discussion_r1739973551


##########
pyiceberg/catalog/rest.py:
##########
@@ -773,6 +771,17 @@ def _commit_table(self, table_request: CommitTableRequest) -> CommitTableRespons
             )
         return CommitTableResponse(**response.json())
 
+    @retry(**_RETRY_ARGS)
+    def list_views(self, namespace: Union[str, Identifier]) -> List[Identifier]:

Review Comment:
   Thanks, and done! This one caused a merge conflicts when pulling in main while working on this PR 👍 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Make `commit_table` public [iceberg-python]

Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #1112:
URL: https://github.com/apache/iceberg-python/pull/1112#discussion_r1737129443


##########
pyiceberg/catalog/rest.py:
##########
@@ -719,12 +721,15 @@ def _remove_catalog_name_from_table_request_identifier(self, table_request: Comm
             )
         return table_request
 
-    @retry(**_RETRY_ARGS)

Review Comment:
   No, I did this earlier when I was trying another route. Thanks for catching this :)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Make `commit_table` public [iceberg-python]

Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #1112:
URL: https://github.com/apache/iceberg-python/pull/1112#discussion_r1737194124


##########
pyiceberg/catalog/rest.py:
##########
@@ -734,9 +739,11 @@ def _commit_table(self, table_request: CommitTableRequest) -> CommitTableRespons
             CommitFailedException: Requirement not met, or a conflict with a concurrent commit.
             CommitStateUnknownException: Failed due to an internal exception on the side of the catalog.
         """
+        identifier = TableIdentifier(namespace=table.identifier[1:-1], name=table.identifier[-1])

Review Comment:
   Thanks, yes, this was still work in progress 🙈 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Make `commit_table` public [iceberg-python]

Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #1112:
URL: https://github.com/apache/iceberg-python/pull/1112#discussion_r1739974152


##########
pyiceberg/catalog/sql.py:
##########
@@ -407,20 +412,18 @@ def _commit_table(self, table_request: CommitTableRequest) -> CommitTableRespons
             NoSuchTableError: If a table with the given identifier does not exist.
             CommitFailedException: Requirement not met, or a conflict with a concurrent commit.
         """
-        identifier_tuple = self._identifier_to_tuple_without_catalog(
-            tuple(table_request.identifier.namespace.root + [table_request.identifier.name])
-        )
-        namespace_tuple = Catalog.namespace_from(identifier_tuple)
+        table_identifier = self._identifier_to_tuple_without_catalog(table.identifier)
+        namespace_tuple = Catalog.namespace_from(table_identifier)
         namespace = Catalog.namespace_to_string(namespace_tuple)
-        table_name = Catalog.table_name_from(identifier_tuple)
+        table_name = Catalog.table_name_from(table_identifier)
 
         current_table: Optional[Table]
         try:
-            current_table = self.load_table(identifier_tuple)
+            current_table = self.load_table(table_identifier)
         except NoSuchTableError:
             current_table = None
 
-        updated_staged_table = self._update_and_stage_table(current_table, table_request)
+        updated_staged_table = self._update_and_stage_table(current_table, table.identifier[1:], requirements, updates)

Review Comment:
   I like that very much, thank you!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org