[GitHub] [iceberg] pvary commented on issue #2540: Hive: Lock Issues with multithreaded commits

[GitBox <gi...@apache.org>]

pvary commented on issue #2540:
URL: https://github.com/apache/iceberg/issues/2540#issuecomment-828761375


   HMS locks are fair. If we have multiple HS2 instances doing the commits then we lose this fairness. We can use fair locks inside a JVM, but cross JVM fairness would be lost. We either share the HMS lock between threads then we risk starving the threads on another JVM, or we release the HMS lock, in this case the commits will happen in a Round Robin fashion between the JVMs.
   
   Also for MR, or TezAM side commits are running in separate JVMs, so JVM level locks does not help there (and does not hurt that much) 
   
   @RussellSpitzer: If you can patch your HMS, you might want to try out this two patches:
   - https://issues.apache.org/jira/browse/HIVE-22906
   - https://issues.apache.org/jira/browse/HIVE-22888


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org