You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2021/03/06 11:00:34 UTC

[GitHub] [iceberg] pPanda-beta opened a new issue #2301: Lock remains forever if HiveTableOperations gets killed (direct process shutdown - no signals) after lock is acquired

pPanda-beta opened a new issue #2301:
URL: https://github.com/apache/iceberg/issues/2301

Although unlock is kept inside the finally block,

https://github.com/apache/iceberg/blob/a7901992c252bb28d10686ef24c1de788c90d663/hive-metastore/src/main/java/org/apache/iceberg/hive/HiveTableOperations.java#L220

### YET INFINITE LOCK MAY HAPPEN

Steps to reproduce :

1. Start a new commit
2. Let icberg acquire the lock
3. Kill the job (signal 9 / 19 ) or disconnect network. (See below to know more about actual scenarios that happened)
4. Restart the job, a fresh new commit
5. Thats it, it will never be able to acquire a new lock on same resource again

### Lets make it more spicy

Consider it as the table create operation 🥰 , i.e. the actual "table" does not exist yet and before creation it will be killed.

Steps:

1. Start creating the table
2. Let it acquire the lock on the non-existent table resource (HMS allows that)
3. Kill the job before the table is even created
4. Go cry yourself, cause you can not remove this lock manually using beeline / hive-cli since table doesn’t exist.

### What is this "Killed" thingy?

Well here in rapido we use GCP preemptible VMs for low cost. Now on such kind of infra, VM may be preempted at any point of time with very short notice period (15 sec).
**Consider this as a network cable unplug**. We will never get a chance for InterruptedException and do clean up operations.

### Why hive never faced this issue?
Well hive has expiry of its locks which works as a TTL. So system recovers eventually.

### Disaster Recoveries that we tried so far:
1. Ensure no jobs are running for that table (which may not have been created yet)
2. Use java api to connect to the hive metastore
3. Delete all locks for that table

## Suggestions

1. Before acquiring locks, delete all locks that have not received any heartbeats since last 'x' minutes. (configurable)
2. After acquiring lock keep sending heart beats to HMS from a different daemon thread. This will ensure concurrent writes are protected
3. After operation finishes (success or fail), unlock and cancel the heartbeat thread.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

[GitHub] [iceberg] marton-bod commented on issue #2301: Lock remains forever if HiveTableOperations gets killed (direct process shutdown - no signals) after lock is acquired

Posted by GitBox <gi...@apache.org>.

marton-bod commented on issue #2301:
URL: https://github.com/apache/iceberg/issues/2301#issuecomment-792700103


   Hi @pPanda-beta 
   Transaction-less locks (where transaction id is 0) should be cleaned up the same way, once the timeout period expires. 
   They are cleaned at this point: https://github.com/apache/hive/blob/8190d2be7b7165effa62bd21b7d60ef81fb0e4af/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java#L4400
   even one step before the timed-out transactions are cleaned up.
   See: https://github.com/apache/hive/blob/8190d2be7b7165effa62bd21b7d60ef81fb0e4af/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java#L4316


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

[GitHub] [iceberg] zhangminglei commented on issue #2301: Lock remains in HMS if HiveTableOperations gets killed (direct process shutdown - no signals) after lock is acquired

Posted by GitBox <gi...@apache.org>.

zhangminglei commented on issue #2301:
URL: https://github.com/apache/iceberg/issues/2301#issuecomment-939268606


   @pvary @lirui-apache In extreme scenarios, Assuming that where the commit execution takes 2 minutes to complete and the house keeper service releases the lock at a default time of 1 minute and 50 seconds (assuming), the heartbeat may be necessary.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

[GitHub] [iceberg] marton-bod commented on issue #2301: Lock remains forever if HiveTableOperations gets killed (direct process shutdown - no signals) after lock is acquired

Posted by GitBox <gi...@apache.org>.

marton-bod commented on issue #2301:
URL: https://github.com/apache/iceberg/issues/2301#issuecomment-792676177


   What's the value of `metastore.txn.timeout` on your HMS instance? As @pvary suggested, the `AcidHouseKeeperService` should clean up any locks without that haven't heartbeated within the timeout period (which is 300 seconds by default), preventing the forever-blockage. The housekeeper service itself will run every 60 seconds by default to purge these timed-out locks (configured by `metastore.acid.housekeeper.interval`). Maybe it's worth checking the HMS logs also to double check if the house keeper service is running regularly


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

[GitHub] [iceberg] marton-bod edited a comment on issue #2301: Lock remains forever if HiveTableOperations gets killed (direct process shutdown - no signals) after lock is acquired

Posted by GitBox <gi...@apache.org>.

marton-bod edited a comment on issue #2301:
URL: https://github.com/apache/iceberg/issues/2301#issuecomment-792676177


   What's the value of `metastore.txn.timeout` on your HMS instance? As @pvary suggested, the `AcidHouseKeeperService` should clean up any locks that haven't heartbeated within the timeout period (which is 300 seconds by default), preventing the forever-blockage. The housekeeper service itself will run every 60 seconds by default to purge these timed-out locks (configured by `metastore.acid.housekeeper.interval`). Maybe it's worth checking the HMS logs also to double check if the house keeper service is running regularly


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

[GitHub] [iceberg] pvary commented on issue #2301: Lock remains forever if HiveTableOperations gets killed (direct process shutdown - no signals) after lock is acquired

Posted by GitBox <gi...@apache.org>.

pvary commented on issue #2301:
URL: https://github.com/apache/iceberg/issues/2301#issuecomment-792596347


   Quick question: Hive removes the locks by the `AcidHouseKeeperService` which should run on the HMS side. Shouldn't that remove the "forever" from the problem statement?
   
   The commits should be fairly quick so having a small timeout could be acceptable if HMS is only used as a HiveCatalog for Iceberg.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

[GitHub] [iceberg] lirui-apache commented on issue #2301: Lock remains in HMS if HiveTableOperations gets killed (direct process shutdown - no signals) after lock is acquired

Posted by GitBox <gi...@apache.org>.

lirui-apache commented on issue #2301:
URL: https://github.com/apache/iceberg/issues/2301#issuecomment-939258835


   We also encountered this issue with iceberg 0.11.1. Intuitively I wanted to enable the house keeper service in HMS. But then I noted iceberg doesn't send heartbeat for acquired locks, which seems to imply it relies on HMS not running the house keeper services. Wondering whether that's by design.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

[GitHub] [iceberg] pPanda-beta commented on issue #2301: Lock remains forever if HiveTableOperations gets killed (direct process shutdown - no signals) after lock is acquired

Posted by GitBox <gi...@apache.org>.

pPanda-beta commented on issue #2301:
URL: https://github.com/apache/iceberg/issues/2301#issuecomment-792714146


   @marton-bod Thanks for pointing it out.
   Theoretically this should work.
   
   I've the value of `metastore.task.threads.remote` as `org.apache.hadoop.hive.metastore.txn.AcidHouseKeeperService,org.apache.hadoop.hive.metastore.txn.AcidOpenTxnsCounterService,org.apache.hadoop.hive.metastore.txn.AcidCompactionHistoryService,org.apache.hadoop.hive.metastore.txn.AcidWriteSetService,org.apache.hadoop.hive.metastore.MaterializationsRebuildLockCleanerTask`
   
   And I dont see the logs "Failed to purge timedout locks due to".
   Definitely something is broken on my HMS. 
   
   Let me investigate that. Meanwhile we can remove the "forever" part.
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

[GitHub] [iceberg] marton-bod commented on issue #2301: Lock remains in HMS if HiveTableOperations gets killed (direct process shutdown - no signals) after lock is acquired

Posted by GitBox <gi...@apache.org>.

marton-bod commented on issue #2301:
URL: https://github.com/apache/iceberg/issues/2301#issuecomment-792829532


   You're correct, it needs to be turned on for the house keeper to be launched.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

[GitHub] [iceberg] szehon-ho commented on issue #2301: Lock remains in HMS if HiveTableOperations gets killed (direct process shutdown - no signals) after lock is acquired

Posted by GitBox <gi...@apache.org>.

szehon-ho commented on issue #2301:
URL: https://github.com/apache/iceberg/issues/2301#issuecomment-926820596


   I found it useful to have a small script to unlock manually as well (in case the HMS thread is not configured).  In scala for instance assuming you get a hmsclient:
   
   ```
    def unlock(db: String, table: String): Unit = {
       val lockIds = getLockIds(db, table)
       lockIds.foreach{
         hmsClient.unlock(_)
       }
     }
   
     private def getLockIds(db: String, table: String): Set[Long] = {
       val showLockRequest = new ShowLocksRequest()
       showLockRequest.setDbname(db)
       showLockRequest.setTablename(table)
       showLockRequest.setIsExtended(true)
       val response = hmsClient.showLocks(showLockRequest)
       val locks = response.getLocks.asScala
       locks.map(_.getLockid).toSet
     }
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

[GitHub] [iceberg] pPanda-beta edited a comment on issue #2301: Lock remains forever if HiveTableOperations gets killed (direct process shutdown - no signals) after lock is acquired

Posted by GitBox <gi...@apache.org>.

pPanda-beta edited a comment on issue #2301:
URL: https://github.com/apache/iceberg/issues/2301#issuecomment-792695001

@pvary @marton-bod
Thanks for the suggestion.
I've gone through their implementation.

This does cleanup via:
1. https://github.com/apache/hive/blob/8190d2be7b7165effa62bd21b7d60ef81fb0e4af/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/txn/AcidHouseKeeperService.java#L60

2. https://github.com/apache/hive/blob/8190d2be7b7165effa62bd21b7d60ef81fb0e4af/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java#L4403

and

3. https://github.com/apache/hive/blob/8190d2be7b7165effa62bd21b7d60ef81fb0e4af/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java#L3633-L3636

But I believe ALL OF THESE WONT WORK:
**cause iceberg does not create transactions before creating the locks**.
https://github.com/apache/iceberg/blob/a7901992c252bb28d10686ef24c1de788c90d663/hive-metastore/src/main/java/org/apache/iceberg/hive/HiveTableOperations.java#L322-L329

This is more like a transaction less lock 😭 . So the lifecycle needs to be maintained via iceberg only.

Also I would insist approaches which are heavily involving hive metastore features only, acid has mostly came from a hive feature. In modern days no one even installs hive. People mostly use spark/presto on top of HMS.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

[GitHub] [iceberg] pvary commented on issue #2301: Lock remains in HMS if HiveTableOperations gets killed (direct process shutdown - no signals) after lock is acquired

Posted by GitBox <gi...@apache.org>.

pvary commented on issue #2301:
URL: https://github.com/apache/iceberg/issues/2301#issuecomment-939903825


   @zhangminglei:
   The code path protected by the lock does this:
   ```
   acquireLock()
     loadHmsTable()
     setHmsTableParameters(newMetadataLocation, tbl, metadata.properties(), removedProps, hiveEngineEnabled, summary);
     persistTable(tbl, updateHiveTable)
   unlock(lockId)
   ```
   
   This is 2 HMS calls and some object data update. So this should not take very long. Also the TXN_TIMEOUT is configured to 300s by default.
   
   All that said, you are right, that in extreme scenarios this could be a problem.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

[GitHub] [iceberg] pvary commented on issue #2301: Lock remains in HMS if HiveTableOperations gets killed (direct process shutdown - no signals) after lock is acquired

Posted by GitBox <gi...@apache.org>.

pvary commented on issue #2301:
URL: https://github.com/apache/iceberg/issues/2301#issuecomment-939261058


   Heartbeat is usually not needed for Iceberg table locks. The lock is only kept for the atomicity of the snapshot swap. This should be almost immediate. Heathbeat interval is 5 min, which means no heartbeat is sent with the default conf anyway. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

[GitHub] [iceberg] pPanda-beta commented on issue #2301: Lock remains forever if HiveTableOperations gets killed (direct process shutdown - no signals) after lock is acquired

Posted by GitBox <gi...@apache.org>.

pPanda-beta commented on issue #2301:
URL: https://github.com/apache/iceberg/issues/2301#issuecomment-792695001

@pvary @marton-bod
Thanks for the suggestion.
I've gone through their implementation.

2. https://github.com/apache/hive/blob/8190d2be7b7165effa62bd21b7d60ef81fb0e4af/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java#L4403

and

3. https://github.com/apache/hive/blob/8190d2be7b7165effa62bd21b7d60ef81fb0e4af/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java#L3633-L3636

But I believe ALL OF THIS WONT WORK:
**cause iceberg does not create transactions before creating the locks**.
https://github.com/apache/iceberg/blob/a7901992c252bb28d10686ef24c1de788c90d663/hive-metastore/src/main/java/org/apache/iceberg/hive/HiveTableOperations.java#L322-L329

This is more like a transaction less lock 😭 . So the lifecycle needs to be maintained via iceberg only.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

[GitHub] [iceberg] pPanda-beta commented on issue #2301: Lock remains in HMS if HiveTableOperations gets killed (direct process shutdown - no signals) after lock is acquired

Posted by GitBox <gi...@apache.org>.

pPanda-beta commented on issue #2301:
URL: https://github.com/apache/iceberg/issues/2301#issuecomment-792803062


   @marton-bod Found it in an integration test powered by embed hive metastore. 
   We need
   ```metastore.compactor.initiator.on=true```
   At least on embedded HMS, this is `false` by default, which was not running those threads.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

[GitHub] [iceberg] marton-bod edited a comment on issue #2301: Lock remains forever if HiveTableOperations gets killed (direct process shutdown - no signals) after lock is acquired

Posted by GitBox <gi...@apache.org>.

marton-bod edited a comment on issue #2301:
URL: https://github.com/apache/iceberg/issues/2301#issuecomment-792676177


   What's the value of `metastore.txn.timeout` on your HMS instance? As @pvary suggested, the `AcidHouseKeeperService` should clean up any locks that haven't heartbeated within the timeout period (which is 300 seconds by default), preventing the forever-blockage. The housekeeper service itself will run every 180 seconds by default to purge these timed-out locks (configured by `metastore.timedout.txn.reaper.interval` on Hive 3.1). Maybe it's worth checking the HMS logs also to double check if the house keeper service is running regularly


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

[GitHub] [iceberg] lirui-apache commented on issue #2301: Lock remains in HMS if HiveTableOperations gets killed (direct process shutdown - no signals) after lock is acquired

Posted by GitBox <gi...@apache.org>.

lirui-apache commented on issue #2301:
URL: https://github.com/apache/iceberg/issues/2301#issuecomment-939267787


   Thanks @pvary for the clarifications. Then I'll go with the house keeper service.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

[GitHub] [iceberg] markope commented on issue #2301: Lock remains in HMS if HiveTableOperations gets killed (direct process shutdown - no signals) after lock is acquired

Posted by GitBox <gi...@apache.org>.

markope commented on issue #2301:
URL: https://github.com/apache/iceberg/issues/2301#issuecomment-926572473


   Is there any workaround for this? How could I manually fix my HMS?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

[GitHub] [iceberg] pPanda-beta commented on issue #2301: Lock remains forever if HiveTableOperations gets killed (direct process shutdown - no signals) after lock is acquired

Posted by GitBox <gi...@apache.org>.

pPanda-beta commented on issue #2301:
URL: https://github.com/apache/iceberg/issues/2301#issuecomment-792695777


   While dealing with independent locks we have to implement with the following columns only,
   ```
   | lockid | database | table | partition | lock_state | blocked_by | lock_type | transaction_id | last_heartbeat | acquired_at | user | hostname | agent_info
   ```


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org