You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "jiakai-li (via GitHub)" <gi...@apache.org> on 2025/01/10 17:38:00 UTC
[I] `table.update_schema()` commits when context manager exist with an error [iceberg-python]
jiakai-li opened a new issue, #1505:
URL: https://github.com/apache/iceberg-python/issues/1505
### Apache Iceberg version
0.8.1 (latest release)
### Please describe the bug 🐞
While working on the issue of #1493 I noticed urrently when running below code, the field `field_should_not_exist` will be added to the table.
```python
from pyiceberg.catalog.sql import SqlCatalog
from pyiceberg.schema import Schema
from pyiceberg.types import IntegerType, NestedField, StringType
WAREHOUSE_PATH = "/workspaces/iceberg-python/warehouse"
catalog = SqlCatalog(
"default",
uri=f"sqlite:///{WAREHOUSE_PATH}/pyiceberg_catalog.db", warehouse=f"file://{WAREHOUSE_PATH}",
)
catalog.create_namespace_if_not_exists("default")
try:
catalog.drop_table("default.test")
except:
pass
schema = Schema(
NestedField(1, "field1", StringType(), required=False)
)
table = catalog.create_table("default.test", schema)
with table.update_schema() as update:
update.add_column("field_should_not_exist", IntegerType())
raise Exception("Error!")
```
The behaviour is a bit confuse to me. Due to an error is raised within the context manager, I would expect the transaction to be aborted. But seems commit is currently the only option to exit the context manager:
```python
class UpdateTableMetadata(ABC, Generic[U]):
_transaction: Transaction
def __init__(self, transaction: Transaction) -> None:
self._transaction = transaction
@abstractmethod
def _commit(self) -> UpdatesAndRequirements: ...
def commit(self) -> None:
self._transaction._apply(*self._commit())
def __exit__(self, _: Any, value: Any, traceback: Any) -> None:
"""Close and commit the change."""
self.commit() # <----- we currently ignore the exc_type
def __enter__(self) -> U:
"""Update the table."""
return self # type: ignore
```
### Willingness to contribute
- [X] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] `table.update_schema()` continues to commit when exist with an exception [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #1505:
URL: https://github.com/apache/iceberg-python/issues/1505#issuecomment-2583565807
Hey @jiakai-li thanks for raising this issue!
The root cause is the same as the one i raised in #1253 and #1497
Specifically, when using UpdateSchema as a "transaction" and a context manager, I expect the transaction to be atomic
```
with table.update_schema() as update:
update.add_column("field_should_not_exist", IntegerType())
raise Exception("Error!")
```
This example shows that the first `add_column` is processed, even though the transaction errored and should not apply any updates.
The core issue is from setting Transaction's `autocommit=True` in `UpdateSchema` ([source](https://grep.app/search?q=autocommit%3DTrue&filter[repo][0]=apache/iceberg-python))
This will cause every update to be "applied"/"flushed" as soon as possible ([source](https://github.com/apache/iceberg-python/blob/a95f9ee6e231104319c01493cb3ada59d9e782d0/pyiceberg/table/__init__.py#L249-L270))
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] `table.update_schema()` continues to commit when exist with an exception [iceberg-python]
Posted by "jiakai-li (via GitHub)" <gi...@apache.org>.
jiakai-li closed issue #1505: `table.update_schema()` continues to commit when exist with an exception
URL: https://github.com/apache/iceberg-python/issues/1505
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] `table.update_schema()` continues to commit when exist with an exception [iceberg-python]
Posted by "jiakai-li (via GitHub)" <gi...@apache.org>.
jiakai-li commented on issue #1505:
URL: https://github.com/apache/iceberg-python/issues/1505#issuecomment-2583805429
Thank you @kevinjqliu , I'm closing this issue to avoid duplicates then. Let's keep the discussion at #1253
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] `table.update_schema()` continues to commit when exist with an exception [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #1505:
URL: https://github.com/apache/iceberg-python/issues/1505#issuecomment-2583824957
I like the example you have above! Do you mind transferring it to #1253?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org