You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "cshuo (via GitHub)" <gi...@apache.org> on 2025/03/05 04:22:48 UTC
Re: [PR] [HUDI-9058] Use InProcessLockProvider for flink single writer ingesti⦠[hudi]
cshuo commented on PR #12857:
URL: https://github.com/apache/hudi/pull/12857#issuecomment-2699773384
I've made some research on s3 and filesystem based lock proivder. I think currently it's not feasible to implement fs lock provider on S3 based on the following infos:
* the basic mechanism of fs lock provider: file existence checking and mutually exclusive file creating.
* S3 does not support object locking for concurrent writers. If two PUT requests are simultaneously made to the same key, the request with the latest timestamp wins, i.e., last-writer-wins semantic.
* S3A fs doesn't support atomic create-no-overwrite operation, which means two processes can send `fs#create(path, overwrite=false)` requests concurrently without any exception.
reference:
* https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md#how-s3a-writes-data-to-s3
* https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html#ConsistencyModel
cc @danny0405 @CTTY, please correct me if I missed something.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org