You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Lokesh Jain (Jira)" <ji...@apache.org> on 2025/04/03 12:01:00 UTC

[jira] [Updated] (HUDI-9254) Ensure lock providers are thread safe

     [ https://issues.apache.org/jira/browse/HUDI-9254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Lokesh Jain updated HUDI-9254:
------------------------------
    Description: 
Only InProcessLockProvider is thread safe as of now. The other lock providers are not thread safe. The Jira aims to ensure all the lock providers which support thread safety are made thread safe.
 - InProcessLockProvider uses ReentrantReadWriteLock which inherently is thread safe.
 - DynamoDBBasedLockProvider uses AmazonDynamoDBLockClient which is thread safe but it stores the lock item as an instance variable which can cause issues with multiple threads.
 - Similarly ZookeeperBasedLockProvider uses CuratorFramework which is thread safe but it stores InterProcessMutex as instance variable. The InterProcessMutex is then released during unlock.
 - HiveMetastoreBasedLockProvider uses IMetaStoreClient which inherently is not thread safe.
 - FileSystemBasedLockProvider is also not thread safe since it stores LockInfo and currentOwnerLockInfo as instance variables.

 

Currently lock provider is closed during unlock and if we are sharing the lock provider amongst multiple threads, we would need to fix the close semantics for lock provider as well. 

  was:
Only InProcessLockProvider is thread safe as of now. The other lock providers are not thread safe. The Jira aims to ensure all the lock providers which support thread safety are made thread safe.

- InProcessLockProvider uses ReentrantReadWriteLock which inherently is thread safe.
- DynamoDBBasedLockProvider uses AmazonDynamoDBLockClient which is thread safe but it stores the lock item as an instance variable which can cause issues with multiple threads.
- Similarly ZookeeperBasedLockProvider uses CuratorFramework which is thread safe but it stores InterProcessMutex as instance variable. The InterProcessMutex is then released during unlock.
- HiveMetastoreBasedLockProvider uses IMetaStoreClient which inherently is not thread safe.
- FileSystemBasedLockProvider is also not thread safe since it stores LockInfo and currentOwnerLockInfo as instance variables.


> Ensure lock providers are thread safe
> -------------------------------------
>
>                 Key: HUDI-9254
>                 URL: https://issues.apache.org/jira/browse/HUDI-9254
>             Project: Apache Hudi
>          Issue Type: Bug
>            Reporter: Lokesh Jain
>            Priority: Critical
>             Fix For: 1.1.0
>
>
> Only InProcessLockProvider is thread safe as of now. The other lock providers are not thread safe. The Jira aims to ensure all the lock providers which support thread safety are made thread safe.
>  - InProcessLockProvider uses ReentrantReadWriteLock which inherently is thread safe.
>  - DynamoDBBasedLockProvider uses AmazonDynamoDBLockClient which is thread safe but it stores the lock item as an instance variable which can cause issues with multiple threads.
>  - Similarly ZookeeperBasedLockProvider uses CuratorFramework which is thread safe but it stores InterProcessMutex as instance variable. The InterProcessMutex is then released during unlock.
>  - HiveMetastoreBasedLockProvider uses IMetaStoreClient which inherently is not thread safe.
>  - FileSystemBasedLockProvider is also not thread safe since it stores LockInfo and currentOwnerLockInfo as instance variables.
>  
> Currently lock provider is closed during unlock and if we are sharing the lock provider amongst multiple threads, we would need to fix the close semantics for lock provider as well. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)