You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2020/06/21 15:08:04 UTC
[GitHub] [iceberg] GrigorievNick opened a new issue #1131: Question: Does it possible `Concurrent write operations` over s3?
GrigorievNick opened a new issue #1131:
URL: https://github.com/apache/iceberg/issues/1131
Hi ICBQm community please help me to understand how icbq grant commits consistency on s3 for concurrent writes.
From [icbq documentation ](https://iceberg.apache.org/reliability/#concurrent-write-operations)
```
Concurrent write operations
Iceberg supports multiple concurrent writes using optimistic concurrency.
Each writer assumes that no other writers are operating and writes out new table metadata for an operation. Then, the writer attempts to commit by atomically swapping the new table metadata file for the existing metadata file.
If the atomic swap fails because another writer has committed, the failed writer retries by writing a new metadata tree based on the new current table state.
```
But Lest image situation:
User one writes data to the table and then commits a new snapshot. But commit where? To **file**(Valid snapshots in an Iceberg table are stored the table metadata file) **on s3.**
Because of eventually consistency of s3, the Second User writes new data and going to commit.
Before commit, the user checks iceberg table metadata file for snapshots to do an optimistic lock, but S3 is eventually consistent. So the user doesn't see a new Iceberg table metadata file created by the user one and just overwrite this file on s3.
This situation leads to the loss of all changes done by the User one. Am I right? or I miss something?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] GrigorievNick commented on issue #1131: Question: Does it possible `Concurrent write operations` over s3?
Posted by GitBox <gi...@apache.org>.
GrigorievNick commented on issue #1131:
URL: https://github.com/apache/iceberg/issues/1131#issuecomment-647141663
Look like I find an answer. But in very [another place]( https://iceberg.apache.org/api-quickstart/),
```
Warning
Hadoop tables shouldn’t be used with file systems that do not support atomic rename. Iceberg relies on rename to synchronize concurrent commits for directory tables.
```
So`Concurrent writes operations` only supported with Hive metadata catalog.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on issue #1131: Question: Does it possible `Concurrent write operations` over s3?
Posted by GitBox <gi...@apache.org>.
rdblue commented on issue #1131:
URL: https://github.com/apache/iceberg/issues/1131#issuecomment-650466406
That's right, concurrent writes are supported through the Hive catalog implementation. You can customize this as @edgarRd pointed out.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] edgarRd commented on issue #1131: Question: Does it possible `Concurrent write operations` over s3?
Posted by GitBox <gi...@apache.org>.
edgarRd commented on issue #1131:
URL: https://github.com/apache/iceberg/issues/1131#issuecomment-647182476
Since Hive Catalog requires Locking support from the Hive Metastore, you could implement your own Catalog based on https://iceberg.apache.org/custom-catalog/ and implement your own distributed locking mechanism.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] GrigorievNick closed issue #1131: Question: Does it possible `Concurrent write operations` over s3?
Posted by GitBox <gi...@apache.org>.
GrigorievNick closed issue #1131:
URL: https://github.com/apache/iceberg/issues/1131
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org