You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "jiakai-li (via GitHub)" <gi...@apache.org> on 2025/01/10 17:38:00 UTC

[I] `table.update_schema()` commits when context manager exist with an error [iceberg-python]

jiakai-li opened a new issue, #1505:
URL: https://github.com/apache/iceberg-python/issues/1505

   ### Apache Iceberg version
   
   0.8.1 (latest release)
   
   ### Please describe the bug 🐞
   
   While working on the issue of #1493 I noticed urrently when running below code, the field `field_should_not_exist` will be added to the table.
   
   ```python
   from pyiceberg.catalog.sql import SqlCatalog
   from pyiceberg.schema import Schema
   from pyiceberg.types import IntegerType, NestedField, StringType
   
   WAREHOUSE_PATH = "/workspaces/iceberg-python/warehouse"
   catalog = SqlCatalog(
       "default",
       uri=f"sqlite:///{WAREHOUSE_PATH}/pyiceberg_catalog.db", warehouse=f"file://{WAREHOUSE_PATH}",
   )
   catalog.create_namespace_if_not_exists("default")
   
   try:
       catalog.drop_table("default.test")
   except:
       pass
   
   schema = Schema(
       NestedField(1, "field1", StringType(), required=False)
   )
   
   table = catalog.create_table("default.test", schema)
   
   with table.update_schema() as update:
       update.add_column("field_should_not_exist", IntegerType())
       raise Exception("Error!")
   ```
   
   The behaviour is a bit confuse to me. Due to an error is raised within the context manager, I would expect the transaction to be aborted. But seems commit is currently the only option to exit the context manager:
   
   ```python
   class UpdateTableMetadata(ABC, Generic[U]):
       _transaction: Transaction
   
       def __init__(self, transaction: Transaction) -> None:
           self._transaction = transaction
   
       @abstractmethod
       def _commit(self) -> UpdatesAndRequirements: ...
   
       def commit(self) -> None:
           self._transaction._apply(*self._commit())
   
       def __exit__(self, _: Any, value: Any, traceback: Any) -> None:
           """Close and commit the change."""
           self.commit()  # <----- we currently ignore the exc_type
   
       def __enter__(self) -> U:
           """Update the table."""
           return self  # type: ignore
   ```
   
   ### Willingness to contribute
   
   - [X] I can contribute a fix for this bug independently
   - [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
   - [ ] I cannot contribute a fix for this bug at this time


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] `table.update_schema()` continues to commit when exist with an exception [iceberg-python]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #1505:
URL: https://github.com/apache/iceberg-python/issues/1505#issuecomment-2583565807

   Hey @jiakai-li thanks for raising this issue! 
   The root cause is the same as the one i raised in #1253 and #1497
   
   Specifically, when using UpdateSchema as a "transaction" and a context manager, I expect the transaction to be atomic
   ```
   with table.update_schema() as update:
       update.add_column("field_should_not_exist", IntegerType())
       raise Exception("Error!")
   ```
   This example shows that the first `add_column` is processed, even though the transaction errored and should not apply any updates. 
   
   The core issue is from setting Transaction's `autocommit=True` in `UpdateSchema` ([source](https://grep.app/search?q=autocommit%3DTrue&filter[repo][0]=apache/iceberg-python))
   This will cause every update to be "applied"/"flushed" as soon as possible ([source](https://github.com/apache/iceberg-python/blob/a95f9ee6e231104319c01493cb3ada59d9e782d0/pyiceberg/table/__init__.py#L249-L270))
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] `table.update_schema()` continues to commit when exist with an exception [iceberg-python]

Posted by "jiakai-li (via GitHub)" <gi...@apache.org>.
jiakai-li closed issue #1505: `table.update_schema()` continues to commit when exist with an exception
URL: https://github.com/apache/iceberg-python/issues/1505


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] `table.update_schema()` continues to commit when exist with an exception [iceberg-python]

Posted by "jiakai-li (via GitHub)" <gi...@apache.org>.
jiakai-li commented on issue #1505:
URL: https://github.com/apache/iceberg-python/issues/1505#issuecomment-2583805429

   Thank you @kevinjqliu , I'm closing this issue to avoid duplicates then. Let's keep the discussion at #1253 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] `table.update_schema()` continues to commit when exist with an exception [iceberg-python]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #1505:
URL: https://github.com/apache/iceberg-python/issues/1505#issuecomment-2583824957

   I like the example you have above! Do you mind transferring it to #1253? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org