You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Thomas Marqardt (Jira)" <ji...@apache.org> on 2020/06/24 13:04:00 UTC
[jira] [Reopened] (HADOOP-17015) ABFS: Make PUT and POST operations
idempotent
[ https://issues.apache.org/jira/browse/HADOOP-17015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Marqardt reopened HADOOP-17015:
--------------------------------------
We should revisit PR 2021 and try to find a better solution for rename. Users expect Rename to be atomic. The service implementation is atomic, but we have this client-side idempotency issue. This fix relies on time and assumes that if the destination was recently updated while we are executing a retry policy, that we succeeded. This may not be the case. For example, users may rely on rename (with overwrite = false) of a file to synchronize or act like a distributed lock, so who ever renames successfully acquires the lock. With the fix in PR 2021, more than one caller could acquire this lock at the same time.
Instead, I think we could allow the client to provide a UUID for the rename operation and persist this UUID in the metadata of the destination blob upon successful completion of a rename, then if we get into this idempotency issue and the client gets a 404 source does not exist, we can check the destination blob's metadata to see if the UUID is a match.
> ABFS: Make PUT and POST operations idempotent
> ---------------------------------------------
>
> Key: HADOOP-17015
> URL: https://issues.apache.org/jira/browse/HADOOP-17015
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
> Affects Versions: 3.2.1
> Reporter: Sneha Vijayarajan
> Assignee: Sneha Vijayarajan
> Priority: Major
> Fix For: 3.4.0
>
>
> Currently when a PUT or POST operation timeouts and the server has already successfully executed the operation, there is no check in driver to see if the operation did succeed or not and just retries the same operation again. This can cause driver to through invalid user errors.
>
> Sample scenario:
> # Rename request times out. Though server has successfully executed the operation.
> # Driver retries rename and get source not found error.
> In the scenario, driver needs to check if rename is being retried and success if source if not found, but destination is present.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org