[jira] [Commented] (HUDI-2173) Implement a DynamoDB based LockProvider

["Dave Hagman (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HUDI-2173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17382961#comment-17382961 ] 

Dave Hagman commented on HUDI-2173:
-----------------------------------

I have started to look into the various options we have when locking against Dynamo. DynamoDB does support [optimistic locking|https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DynamoDBMapper.OptimisticLocking.html]. There is also an AWS labs library, [DynamoDB Lock Client|https://github.com/awslabs/amazon-dynamodb-lock-client]. 

The lock client has a lot of prebuilt functionality built-in and according to the README it is used extensively within AWS. What I don't like is that it appears to be quite stale. The README claims there is a version *1.2* in Maven but they never actually published it. There are some recently merged PRs but this doesn't seem to be actively managed. For this reason I don't think we should consider using it at this time.

The optimistic locking functionality in the native Java SDK looks promising. It works by using a versioned attribute field. When a client wants to try to obtain a lock the client must submit their view of the most recent row version with the acquire request. If the row version submitted by the client is less than what's in Dynamo it will fail to update (which means we could not lock the row/partition/etc). The client would then check for the release of the lock (up until some maximum wait time). When the lock is released the client would again try to update the row given the client's last known latest version. At that point the row update (to set the lock) can again succeed or fail depending on if another process already acquired the lock. 

> Implement a DynamoDB based LockProvider
> ---------------------------------------
>
>                 Key: HUDI-2173
>                 URL: https://issues.apache.org/jira/browse/HUDI-2173
>             Project: Apache Hudi
>          Issue Type: Sub-task
>          Components: Writer Core
>            Reporter: Vinoth Chandar
>            Assignee: Dave Hagman
>            Priority: Major
>             Fix For: 0.10.0
>
>
> Currently, we have ZK and HMS based Lock providers, which can be limited to co-ordinating across a single EMR or Hadoop cluster. 
> For aws users, DynamoDB is a readily available , fully managed , geo replicated datastore, that can actually be used to hold locks, that can now span across EMR/hadoop clusters. 
> This effort involves supporting a new `DynamoDB` lock provider that implements org.apache.hudi.common.lock.LockProvider. We can place the implementation itself in hudi-client-common, so it can be used across Spark, Flink, Deltastreamer etc. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)