You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "kevinjqliu (via GitHub)" <gi...@apache.org> on 2024/10/27 21:38:51 UTC

[I] [discuss] `Transaction` API's `autocommit` [iceberg-python]

kevinjqliu opened a new issue, #1253:
URL: https://github.com/apache/iceberg-python/issues/1253

   ### Feature Request / Improvement
   
   Porting over from #1246
   
   Can be a potential footgun (https://github.com/apache/iceberg-python/pull/1246#issuecomment-2439024843)
   Autocommit usage (https://github.com/apache/iceberg-python/pull/1246#issuecomment-2439780764)
   
   > However, there may still be some concerns around this since Transaction is a public class. If this is the case, I think we can start from making the parameter "private" (autocommit -> _autocommit) and/or adding some doc to explain the usage.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] [discuss] `Transaction` API's `autocommit` [iceberg-python]

Posted by "jiakai-li (via GitHub)" <gi...@apache.org>.
jiakai-li commented on issue #1253:
URL: https://github.com/apache/iceberg-python/issues/1253#issuecomment-2583844373

   Just raise another possibility for discussion. We could also modify the `_transaction._autocommit` when calling `UpdateSchema.__enter__` method, which indicates multiple updates is coming and we would like to avoid the autocommit. Something like:
   ```python
   class UpdateSchema(UpdateTableMetadata["UpdateSchema"]):
       ...
       def __enter__(self):
           self._transaction._autocommit = False
           return self
       ...
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] [discuss] `Transaction` API's `autocommit` [iceberg-python]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #1253:
URL: https://github.com/apache/iceberg-python/issues/1253#issuecomment-2440186317

   > The idea is to make the code simpler if we only want to evolve schema/spec/...
   i.e.
   ```
   with table.update_schema() as update:
       update.add_column("some_field", IntegerType(), "doc")
   ```
   instead of another with..transaction wrapper
   ```
   with table.transaction() as transaction:
       with transaction.update_schema() as update_schema:
           update.add_column("some_other_field", IntegerType(), "doc")
   ```
   ....
   
   
   What about moving the `autocommit` logic out of `Transaction` and into the class that uses it instead? 
   
   For example, `UpdateSchema` can implement `__enter__` and `__exit__` to commit the transaction automatically?
   
   Currently, these classes use `autocommit=True`:
   - ManageSnapshots
   - UpdateSchema
   - UpdateSpec


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] [discuss] `Transaction` API's `autocommit` [iceberg-python]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #1253:
URL: https://github.com/apache/iceberg-python/issues/1253#issuecomment-3059680622

   This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] [discuss] `Transaction` API's `autocommit` [iceberg-python]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #1253:
URL: https://github.com/apache/iceberg-python/issues/1253#issuecomment-3741156383

   This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] [discuss] `Transaction` API's `autocommit` [iceberg-python]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #1253:
URL: https://github.com/apache/iceberg-python/issues/1253#issuecomment-3808250338

   This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] [discuss] `Transaction` API's `autocommit` [iceberg-python]

Posted by "jiakai-li (via GitHub)" <gi...@apache.org>.
jiakai-li commented on issue #1253:
URL: https://github.com/apache/iceberg-python/issues/1253#issuecomment-2583876079

   Adding an example for the issue:
   ```python
   from pyiceberg.catalog.sql import SqlCatalog
   from pyiceberg.schema import Schema
   from pyiceberg.types import IntegerType, NestedField, StringType
   
   WAREHOUSE_PATH = "/workspaces/iceberg-python/warehouse"
   catalog = SqlCatalog(
       "default",
       uri=f"sqlite:///{WAREHOUSE_PATH}/pyiceberg_catalog.db", warehouse=f"file://{WAREHOUSE_PATH}",
   )
   catalog.create_namespace_if_not_exists("default")
   
   try:
       catalog.drop_table("default.test")
   except:
       pass
   
   schema = Schema(
       NestedField(1, "field1", StringType(), required=False)
   )
   
   table = catalog.create_table("default.test", schema)
   
   with table.update_schema() as update:
       update.add_column("field_should_not_exist", IntegerType())
       raise Exception("Error!")
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] [discuss] `Transaction` API's `autocommit` [iceberg-python]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #1253:
URL: https://github.com/apache/iceberg-python/issues/1253#issuecomment-2584974311

   > The reason behind my head is, to my understanding, autocommit is still somewhat a related concept with transaction.
   
   I think its fair to ask if we consider `UpdateSchema` to be a transaction. If so, we cannot allow `autocommit` since that'll break the atomicity of the transaction. 
   
   > We could also modify the _transaction._autocommit when calling UpdateSchema.__enter__ method, which indicates multiple updates is coming and we would like to avoid the autocommit. Something like:
   
   I feel like in this case, we can just get rid of `autocommit` entirely
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] [discuss] `Transaction` API's `autocommit` [iceberg-python]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #1253: [discuss] `Transaction` API's `autocommit` 
URL: https://github.com/apache/iceberg-python/issues/1253


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org