You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Quanlong Huang (Jira)" <ji...@apache.org> on 2023/05/15 05:22:00 UTC
[jira] [Updated] (IMPALA-12141) IllegalMonitorStateException when trying to release the table lock
[ https://issues.apache.org/jira/browse/IMPALA-12141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Quanlong Huang updated IMPALA-12141:
------------------------------------
Summary: IllegalMonitorStateException when trying to release the table lock (was: IllegalMonitorStateException when releasing the table lock)
> IllegalMonitorStateException when trying to release the table lock
> ------------------------------------------------------------------
>
> Key: IMPALA-12141
> URL: https://issues.apache.org/jira/browse/IMPALA-12141
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Reporter: Quanlong Huang
> Priority: Critical
>
> We saw event-processor went into the ERROR state due to an IllegalMonitorStateException:
> {noformat}
> I0504 12:28:45.272922 189771 MetastoreEvents.java:401] EventId: 56369449 EventType: INSERT Incremented events skipped counter to 283902
> I0504 12:28:45.272941 189771 MetastoreEvents.java:401] EventId: 56369449 EventType: INSERT Not processing the event as it is a self-event
> I0504 14:28:45.283041 189771 MetastoreEvents.java:412] EventId: 56369450 EventType: INSERT Received exception Error during self-event evaluation for table xxx.xxxx due to lock contention. Ignoring self-event evaluation
> E0504 16:28:45.286149 189771 MetastoreEventsProcessor.java:684] Unexpected exception received while processing event
> Java exception follows:
> java.lang.IllegalMonitorStateException
> at java.base/java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryRelease(ReentrantReadWriteLock.java:372)
> at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1302)
> at java.base/java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.unlock(ReentrantReadWriteLock.java:1147)
> at org.apache.impala.catalog.Table.releaseWriteLock(Table.java:262)
> at org.apache.impala.service.CatalogOpExecutor.reloadPartitionIfExists(CatalogOpExecutor.java:3788)
> at org.apache.impala.catalog.events.MetastoreEvents$MetastoreTableEvent.reloadPartition(MetastoreEvents.java:633)
> at org.apache.impala.catalog.events.MetastoreEvents$InsertEvent.processPartitionInserts(MetastoreEvents.java:851)
> at org.apache.impala.catalog.events.MetastoreEvents$InsertEvent.process(MetastoreEvents.java:835)
> at org.apache.impala.catalog.events.MetastoreEvents$MetastoreEvent.processIfEnabled(MetastoreEvents.java:346)
> at org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:772)
> at org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:670)
> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> E0504 16:28:45.286345 189771 MetastoreEventsProcessor.java:795] Notification event is null
> {noformat}
> It's due to the following try-clause:
> {code:java}
> try {
> tryWriteLock(table, reason); // throws InternalException if timeout (2h) to get write lock
> ...
> return numOfPartsReloaded;
> } catch (TableLoadingException e) {
> ...
> } catch (InternalException e) {
> throw new CatalogException(
> "Could not acquire lock on the table " + table.getFullName(), e);
> } finally {
> UnlockWriteLockIfErronouslyLocked();
> table.releaseWriteLock();
> }
> {code}
> https://github.com/apache/impala/blob/3608ab25f13708b1ba73b0f81abe37c1cda4e342/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java#L4604-L4641
> tryWriteLock() will wait until timeout(2h) to get the table write lock. If fails, it throws an InternalException. The finally-clause of releaseWriteLock() is always invoked so it fails by lock not held by current thread.
> {code:java}
> private void tryWriteLock(Table tbl, String operation) throws InternalException {
> String type = tbl instanceof View ? "view" : "table";
> if (!catalog_.tryWriteLock(tbl)) {
> throw new InternalException(String.format("Error %s (for) %s %s due to " +
> "lock contention.", operation, type, tbl.getFullName()));
> }
> }{code}
> https://github.com/apache/impala/blob/3608ab25f13708b1ba73b0f81abe37c1cda4e342/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java#L7309-L7323
> We should check if the lock is held by the current thread before calling table.releaseWriteLock(), or move tryWriteLock() outside the try-clause.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org