[GitHub] [iceberg] flyrain opened a new issue #2387: Commit failed due to Internal error processing lock in metastore DB

[GitBox <gi...@apache.org>]

flyrain opened a new issue #2387:
URL: https://github.com/apache/iceberg/issues/2387


   I hit this issue, and got it resolved. It should only happen when you connect a postgres Hive metastore DB. Open this issue in case somebody else hit it. It is similar to issue #798, but the root case should be a bit different. 
   Here is the client side stack.
   ```
   scala> spark.sql("CREATE TABLE testdb.student (id INT, name STRING, age INT) USING ADT PARTITIONED by(age) LOCATION 'file:/tmp/testdb/student'").show
   java.lang.RuntimeException: Metastore operation failed for testdb.student
     at org.apache.iceberg.hive.HiveTableOperations.doCommit(HiveTableOperations.java:204)
     at org.apache.iceberg.BaseMetastoreTableOperations.commit(BaseMetastoreTableOperations.java:107)
   ...
   Caused by: org.apache.thrift.TApplicationException: Internal error processing lock
     at org.apache.thrift.TApplicationException.read(TApplicationException.java:111)
     at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
     at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_lock(ThriftHiveMetastore.java:3910)
     at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.lock(ThriftHiveMetastore.java:3897)
     at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.lock(HiveMetaStoreClient.java:1955)
     at org.apache.iceberg.hive.HiveTableOperations.lambda$acquireLock$8(HiveTableOperations.java:277)
     at org.apache.iceberg.hive.ClientPool.run(ClientPool.java:54)
     at org.apache.iceberg.hive.HiveTableOperations.acquireLock(HiveTableOperations.java:277)
     at org.apache.iceberg.hive.HiveTableOperations.doCommit(HiveTableOperations.java:145)
     ... 72 more
   ```
   Server side stack
   ```
   2021-03-26 18:47:03,240 ERROR thrift.ProcessFunction (ProcessFunction.java:process(41)) - Internal error processing lock
   MetaException(message:Unable to update transaction database org.postgresql.util.PSQLException: ERROR: index "hl_txnid_index" contains unexpected zero page at block 0
     Hint: Please REINDEX it.
           at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2455)
           at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2155)
           at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:288)
           at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:430)
           at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:356)
           at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:303)
           at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:289)
           at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:266)
           at org.postgresql.jdbc.PgStatement.executeUpdate(PgStatement.java:246)
           at com.jolbox.bonecp.StatementHandle.executeUpdate(StatementHandle.java:497)
           at org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:1428)
           at org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:422)
           at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:5527)
   ```
   To fix it, we need to reindex hl_txnid_index. Here is an example command.
   ```
   psql -U hive -d metastore --no-password -h localhost --command "REINDEX INDEX hl_txnid_index"
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org