You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2020/10/25 09:51:02 UTC
[GitHub] [iceberg] gallushi opened a new issue #1655: Extend HadoopTableOperations to work with other FS guarantees
gallushi opened a new issue #1655:
URL: https://github.com/apache/iceberg/issues/1655
Currently, `HadoopTableOperations` requires atomic rename from the underlying FS.
however, other guarantees such as atomic write can also suffice. this will allow other storage systems (for example, [IBM Cloud Object Storage](https://www.ibm.com/cloud/object-storage), which supports atomic write) to be used.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on issue #1655: Extend HadoopTableOperations to also work with other FS guarantees
Posted by GitBox <gi...@apache.org>.
rdblue commented on issue #1655:
URL: https://github.com/apache/iceberg/issues/1655#issuecomment-718265210
@gallushi, can you describe what you're suggesting in a bit more detail? How would you use atomic write instead? And how would you detect whether to use atomic write or atomic rename?
In general, I would not recommend using a file system for this guarantee. It's better to use a database transaction for the atomic update operation. That's why we want to have support for a variety of catalog plugins in addition to Hive, like JDBC, Nessie, and Glue.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] gallushi edited a comment on issue #1655: Extend HadoopTableOperations to also work with other FS guarantees
Posted by GitBox <gi...@apache.org>.
gallushi edited a comment on issue #1655:
URL: https://github.com/apache/iceberg/issues/1655#issuecomment-732855802
Hi @rdblue
> can you describe what you're suggesting in a bit more detail? How would you use atomic write instead?
Currently, when using hadoop tables, `HadoopTableOperations` assumes atomic rename guarantees, and (when committing) proceeds to rename the temp snapshot object from its temp name to the snapshot object name.
However - atomic write can also be used - instead of using the temp object, we can directly (and atomically) write the snapshot object. the write will succeed iff the object did not exist.
> And how would you detect whether to use atomic write or atomic rename?
I think using a config on a scheme level makes sense (this way one can control on which FS to use atomic write and on which ones to stay with atomic rename) since atomic write/ atomic rename are FS level guarantees.
> In general, I would not recommend using a file system for this guarantee. It's better to use a database transaction for the atomic update operation. That's why we want to have support for a variety of catalog plugins in addition to Hive, like JDBC, Nessie, and Glue.
Yes; however (also keeping in mind that this is all in the context of hadoop tables, which in any case rely on file system guarantees) - adding support for atomic write is a relatively low-hanging fruit, and storage systems such as IBM Cloud Object Storage can then be used even without an external catalog.
while we discuss this, i'll open a PR with the changes i have in mind.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] gallushi commented on issue #1655: Extend HadoopTableOperations to also work with other FS guarantees
Posted by GitBox <gi...@apache.org>.
gallushi commented on issue #1655:
URL: https://github.com/apache/iceberg/issues/1655#issuecomment-732855802
Hi @rdblue
> can you describe what you're suggesting in a bit more detail? How would you use atomic write instead?
Currently, when using hadoop tables, `HadoopTableOperations` assumes atomic rename guarantees, and (when committing) proceeds to rename the temp snapshot object from its temp name to the snapshot object name.
However - atomic write can also be used - instead of using the temp object, we can directly (and atomically) write the snapshot object. the write will succeed iff the object did not exist.
> And how would you detect whether to use atomic write or atomic rename?
I think using a config on a scheme level makes sense (this way one can control on which FS to use atomic write and on which ones to stay with atomic rename) since atomic write/ atomic rename are FS level guarantees.
> In general, I would not recommend using a file system for this guarantee. It's better to use a database transaction for the atomic update operation. That's why we want to have support for a variety of catalog plugins in addition to Hive, like JDBC, Nessie, and Glue.
Yes; however - adding support for atomic write is a relatively low-hanging fruit, and storage systems such as IBM Cloud Object Storage can then be used even without an external catalog.
while we discuss this, i'll open a PR with the changes i have in mind.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on issue #1655: Extend HadoopTableOperations to also work with other FS guarantees
Posted by GitBox <gi...@apache.org>.
rdblue commented on issue #1655:
URL: https://github.com/apache/iceberg/issues/1655#issuecomment-733143605
@gallushi, that sounds good to me. As long as the file commit is atomic, I think we can work with it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org