You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2020/02/05 17:14:20 UTC
[GitHub] [incubator-iceberg] alexey-fin opened a new pull request #779:
Support read-after-write eventual consistency problems on object stores
alexey-fin opened a new pull request #779: Support read-after-write eventual consistency problems on object stores
URL: https://github.com/apache/incubator-iceberg/pull/779
On s3 read-after-write consistency is eventual in case of GET-PUT-GET scenario
which causes failures like Caused by: org.apache.iceberg.exceptions.NotFoundException: Failed to open input stream for file: s3a://feature-env-bucket-fe-cdp-sf-stress/datobase/dynamic-schema-storage/brand=9524/dynamic-schema-id=49975/metadata/snap-8360570545819086573-1-f18fb942-6dec-4e51-89be-1408a953352d.avro even though the file was written and updated in hive, as described in
https://codeburst.io/quick-explanation-of-the-s3-consistency-model-6c9f325e3f82
We suggest a retry on commits to be optionally extended to FileNotFound exceptions
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] alexey-fin commented on issue #779: Support
read-after-write eventual consistency problems on object stores
Posted by GitBox <gi...@apache.org>.
alexey-fin commented on issue #779: Support read-after-write eventual consistency problems on object stores
URL: https://github.com/apache/incubator-iceberg/pull/779#issuecomment-582985723
@aokolnychyi yeah definitely, look at the s3afilesystem from hadoop-aws : try {
// get the status or throw an FNFE
status = getFileStatus(path);
// if the thread reaches here, there is something at the path
if (status.isDirectory()) {
// path references a directory: automatic error
throw new FileAlreadyExistsException(path + " is a directory");
}
It is s3
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] aokolnychyi commented on issue #779: Support
read-after-write eventual consistency problems on object stores
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on issue #779: Support read-after-write eventual consistency problems on object stores
URL: https://github.com/apache/incubator-iceberg/pull/779#issuecomment-582990214
I agree with Ryan that building another `FileIO` implementation to address this would be a better option. @alexey-fin, would you be interested to contribute this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] alexey-fin closed pull request #779: Support
read-after-write eventual consistency problems on object stores
Posted by GitBox <gi...@apache.org>.
alexey-fin closed pull request #779: Support read-after-write eventual consistency problems on object stores
URL: https://github.com/apache/incubator-iceberg/pull/779
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] rdblue commented on issue #779: Support
read-after-write eventual consistency problems on object stores
Posted by GitBox <gi...@apache.org>.
rdblue commented on issue #779: Support read-after-write eventual consistency problems on object stores
URL: https://github.com/apache/incubator-iceberg/pull/779#issuecomment-582552571
Metadata files should be created using `overwrite` instead of `create`. In most s3 file systems, that will avoid the HEAD request to see if the file exists that s3 will cache. That fixes these in most cases.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] alexey-fin commented on issue #779: Support
read-after-write eventual consistency problems on object stores
Posted by GitBox <gi...@apache.org>.
alexey-fin commented on issue #779: Support read-after-write eventual consistency problems on object stores
URL: https://github.com/apache/incubator-iceberg/pull/779#issuecomment-583015791
Actually what Ryan says there are apparently several existing implementations, one example is PrestoS3FileSystem, we might just wanna use it
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] alexey-fin commented on issue #779: Support
read-after-write eventual consistency problems on object stores
Posted by GitBox <gi...@apache.org>.
alexey-fin commented on issue #779: Support read-after-write eventual consistency problems on object stores
URL: https://github.com/apache/incubator-iceberg/pull/779#issuecomment-582554306
It not only checks if the file exists, it also checks if there is a dir
with this name, we are running into this problem
On Wed, Feb 5, 2020, 20:43 Ryan Blue <no...@github.com> wrote:
> Metadata files should be created using overwrite instead of create. In
> most s3 file systems, that will avoid the HEAD request to see if the file
> exists that s3 will cache. That fixes these in most cases.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/incubator-iceberg/pull/779?email_source=notifications&email_token=AK5UVLURZ2AEPGRYI4UMVJTRBMCEJA5CNFSM4KQO37CKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEK4QX6Y#issuecomment-582552571>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AK5UVLQDPDWCEP7UL7LF2VTRBMCEJANCNFSM4KQO37CA>
> .
>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] aokolnychyi commented on issue #779: Support
read-after-write eventual consistency problems on object stores
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on issue #779: Support read-after-write eventual consistency problems on object stores
URL: https://github.com/apache/incubator-iceberg/pull/779#issuecomment-582974652
@alexey-fin, is this S3 or S3-compatible file system?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] rdblue commented on issue #779: Support
read-after-write eventual consistency problems on object stores
Posted by GitBox <gi...@apache.org>.
rdblue commented on issue #779: Support read-after-write eventual consistency problems on object stores
URL: https://github.com/apache/incubator-iceberg/pull/779#issuecomment-582556513
In that case, I'd recommend plugging in a different FileIO for S3 file systems that will retry reads for some period of time. I don't think it makes sense to change the commit behavior if the problem is read consistency.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] aokolnychyi commented on issue #779: Support
read-after-write eventual consistency problems on object stores
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on issue #779: Support read-after-write eventual consistency problems on object stores
URL: https://github.com/apache/incubator-iceberg/pull/779#issuecomment-582973644
Quick question. The file name in the description indicates that we failed to read the manifest list. Every time we attempt to commit, we have a new manifest list location. `HadoopFileIO` will delegate the creation of the file to `HadoopOutputFile`. As Ryan pointed out, we always set the overwrite flag. Are we saying `fs.create(path, true /* createOrOverwrite */)` sends a GET request that triggers eventual consistency?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org