You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "ZENOTME (via GitHub)" <gi...@apache.org> on 2024/09/01 15:18:52 UTC
[I] Add apply interface in transaction [iceberg-rust]
ZENOTME opened a new issue, #596:
URL: https://github.com/apache/iceberg-rust/issues/596
This will work for now but might get problematic later on. Just a heads up.
An important concept for Iceberg is to stack snapshots in a single commit. For example, now with append being added in this PR, we can easily add support for truncate. This would be a delete operation where all the data is being dropped, and then just an append.
_Originally posted by @Fokko in https://github.com/apache/iceberg-rust/pull/349#discussion_r1579449633_
Transaction should be able reflect the update in time. According to pyiceberg, we can provide a apply interface to update the table metedata.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Add apply interface in transaction [iceberg-rust]
Posted by "ZENOTME (via GitHub)" <gi...@apache.org>.
ZENOTME commented on issue #596:
URL: https://github.com/apache/iceberg-rust/issues/596#issuecomment-2328870882
> Hi, @ZENOTME Could you elaborate on this? I'm kind of confusing about the proposal.
For now, transaction can't reflect the update in time so we can stack them together.
e.g.
```
// table is a v1 table
let tx = Transaction(table);
// This will end up sending two UpgradeFormatVersion into catalog
tx.upgrade_table_version().unwrap().
.upgrade_table_version().unwrap().commit()
```
But In pyiceberg, above behaviour will only send one UpgradeFormatVersion and the second one will see that the metadata of table has been updated. The update will be [apply](https://github.com/apache/iceberg-python/blob/9857107561d2267813b7ce150b01b4e6ac4b3e34/pyiceberg/table/__init__.py#L309) into local medata and reflect the change first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Add apply interface in transaction [iceberg-rust]
Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on issue #596:
URL: https://github.com/apache/iceberg-rust/issues/596#issuecomment-2419030826
Thanks @ZENOTME 's explaination. I think I've got your point, we need sth like `commit` in transaction action so that later transaction action could take into account previously happened changes. The requires to build stage only snapshots.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Add apply interface in transaction [iceberg-rust]
Posted by "ZENOTME (via GitHub)" <gi...@apache.org>.
ZENOTME commented on issue #596:
URL: https://github.com/apache/iceberg-rust/issues/596#issuecomment-2323395450
> Exactly. Internally you want to stack the changes together. For example, within a single transaction, you add a new field and then write the data, then the latest schema should be taken into account. We do this in PyIceberg here: https://github.com/apache/iceberg-python/blob/03a0d65ac05d556d0815e61a016effc2b8993702/pyiceberg/table/__init__.py#L715
We can implement the [update_table_metadata](https://github.com/apache/iceberg-python/blob/main/pyiceberg/table/__init__.py) based on https://github.com/apache/iceberg-rust/pull/587. cc @liurenjie1024 @Xuanwo @Fokko @c-thiel
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Add apply interface in transaction [iceberg-rust]
Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 closed issue #596: Add apply interface in transaction
URL: https://github.com/apache/iceberg-rust/issues/596
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Add apply interface in transaction [iceberg-rust]
Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on issue #596:
URL: https://github.com/apache/iceberg-rust/issues/596#issuecomment-2370693889
> Why for rest catalog, it should be sent to rest catalog server.🤔
> According to API from pyiceberg, it seems possible to [create a transaction without auto commit ](https://github.com/apache/iceberg-python/blob/e891bcddb1584c6b7a35b61537ab5802b514ec6d/pyiceberg/table/__init__.py#L1418), which means that we also can apply transaction actions in local for rest catalog(do I miss something here)
Sorry, I don't quite get the point, if updates are sent to rest catalog server, why we need to update it in local first?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Add apply interface in transaction [iceberg-rust]
Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on issue #596:
URL: https://github.com/apache/iceberg-rust/issues/596#issuecomment-2328031227
Hi, @ZENOTME Could you elaborate on this? I'm kind of confusing about the proposal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Add apply interface in transaction [iceberg-rust]
Posted by "ZENOTME (via GitHub)" <gi...@apache.org>.
ZENOTME commented on issue #596:
URL: https://github.com/apache/iceberg-rust/issues/596#issuecomment-2336554052
> We have [check](https://github.com/apache/iceberg-rust/blob/a1ec0fa98113fc02b2479d81759fccd9dab10378/crates/iceberg/src/transaction.rs#L50) to avoid such duplicated case. For metastore tables, it's supposed to apply transaction actions in local, and update metastore pointer. For rest catalog, it should be sent to rest catalog server.
Why for rest catalog, it should be sent to rest catalog server.🤔
According to API from pyiceberg, it seems possible to [create a transaction without auto commit ](https://github.com/apache/iceberg-python/blob/e891bcddb1584c6b7a35b61537ab5802b514ec6d/pyiceberg/table/__init__.py#L1418), which means that we also can apply transaction actions in local for rest catalog(do I miss something here)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Add apply interface in transaction [iceberg-rust]
Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 closed issue #596: Add apply interface in transaction
URL: https://github.com/apache/iceberg-rust/issues/596
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Add apply interface in transaction [iceberg-rust]
Posted by "ZENOTME (via GitHub)" <gi...@apache.org>.
ZENOTME commented on issue #596:
URL: https://github.com/apache/iceberg-rust/issues/596#issuecomment-2371283315
> Sorry, I don't quite get the point, if updates are sent to rest catalog server, why we need to update it in local first?
E.g. the user wants to batch multiple updates and commit them once. For the following, after `tx.action2().apply()`, the update will not be sent to the catalog, but it should apply this action to table metadata locally so that action3 can realize that.
```
let tx = table.transaction()
tx.action1().apply() // Update table versi
tx.action2().apply() // Append new data file
tx.action3().apply()
tx.commit()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Add apply interface in transaction [iceberg-rust]
Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on issue #596:
URL: https://github.com/apache/iceberg-rust/issues/596#issuecomment-2336534759
We have [check](https://github.com/apache/iceberg-rust/blob/a1ec0fa98113fc02b2479d81759fccd9dab10378/crates/iceberg/src/transaction.rs#L50) to avoid such duplicated case. For metastore tables, it's supposed to apply transaction actions in local, and update metastore pointer. For rest catalog, it should be sent to rest catalog server.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org