You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Michael Smith (Jira)" <ji...@apache.org> on 2025/03/07 18:44:00 UTC

[jira] [Commented] (IMPALA-13842) Test against sys.impala_query_log should wait until coordinator quiesce.

    [ https://issues.apache.org/jira/browse/IMPALA-13842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17933405#comment-17933405 ] 

Michael Smith commented on IMPALA-13842:
----------------------------------------

Tests that enable workload management are generally required to use graceful shutdown. Maybe that's not happening somewhere.

> Test against sys.impala_query_log should wait until coordinator quiesce.
> ------------------------------------------------------------------------
>
>                 Key: IMPALA-13842
>                 URL: https://issues.apache.org/jira/browse/IMPALA-13842
>             Project: IMPALA
>          Issue Type: Bug
>          Components: Test
>            Reporter: Riza Suminto
>            Assignee: Jason Fehr
>            Priority: Major
>              Labels: broken-build
>
> We are seeing evidence of HMS deadlock for table sys.impala_query_log in downstream build.
> {noformat}
> I0307 02:24:06.992715 18405 SnapshotProducer.java:422] 5f4ecf63310cb7f5:e2db615800000000] Committed snapshot 4325155866837015464 (MergeAppend)
> I0307 02:24:07.002012 18405 LoggingMetricsReporter.java:38] 5f4ecf63310cb7f5:e2db615800000000] Received metrics report: CommitReport{tableName=ImpalaHiveCatalog.sys.impala_query_log, snapshotId=4325155866837015464, sequenceNumber=9, operation=append, commitMetrics=CommitMetricsResult{totalDuration=TimerResult{timeUnit=NANOSECONDS, totalDuration=PT0.241321564S, count=1}, attempts=CounterResult{unit=COUNT, value=1}, addedDataFiles=CounterResult{unit=COUNT, value=1}, removedDataFiles=null, totalDataFiles=CounterResult{unit=COUNT, value=9}, addedDeleteFiles=null, addedEqualityDeleteFiles=null, addedPositionalDeleteFiles=null, removedDeleteFiles=null, removedEqualityDeleteFiles=null, removedPositionalDeleteFiles=null, totalDeleteFiles=CounterResult{unit=COUNT, value=0}, addedRecords=CounterResult{unit=COUNT, value=1}, removedRecords=null, totalRecords=CounterResult{unit=COUNT, value=41}, addedFilesSizeInBytes=CounterResult{unit=BYTES, value=26244}, removedFilesSizeInBytes=null, totalFilesSizeInBytes=CounterResult{unit=BYTES, value=273450}, addedPositionalDeletes=null, removedPositionalDeletes=null, totalPositionalDeletes=CounterResult{unit=COUNT, value=0}, addedEqualityDeletes=null, removedEqualityDeletes=null, totalEqualityDeletes=CounterResult{unit=COUNT, value=0}}, metadata={iceberg-version=Apache Iceberg 1.5.2.2025.0.20.0-45 (commit f2db4a27a43aaec790af8b9e2ac51f4a50b26260)}}
> I0307 02:24:07.002182 18405 CatalogServiceCatalog.java:1327] 5f4ecf63310cb7f5:e2db615800000000] Added catalog version 2126 in table's sys.impala_query_log in-flight events
> W0307 02:24:07.053566 18405 Tasks.java:456] 5f4ecf63310cb7f5:e2db615800000000] Retrying task after failure: Waiting for lock on table sys.impala_query_log
> Java exception follows:
> org.apache.iceberg.hive.MetastoreLock$WaitingForLockException: Waiting for lock on table sys.impala_query_log
>         at org.apache.iceberg.hive.MetastoreLock.lambda$acquireLock$1(MetastoreLock.java:217)
>         at org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:413)
>         at org.apache.iceberg.util.Tasks$Builder.runSingleThreaded(Tasks.java:219)
>         at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:203)
>         at org.apache.iceberg.hive.MetastoreLock.acquireLock(MetastoreLock.java:209)
>         at org.apache.iceberg.hive.MetastoreLock.lock(MetastoreLock.java:146)
>         at org.apache.iceberg.hive.HiveTableOperations.doCommit(HiveTableOperations.java:182)
>         at org.apache.iceberg.BaseMetastoreTableOperations.commit(BaseMetastoreTableOperations.java:135)
>         at org.apache.iceberg.BaseTransaction.lambda$commitSimpleTransaction$3(BaseTransaction.java:427)
>         at org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:413)
>         at org.apache.iceberg.util.Tasks$Builder.runSingleThreaded(Tasks.java:219)
>         at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:203)
>         at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:196)
>         at org.apache.iceberg.BaseTransaction.commitSimpleTransaction(BaseTransaction.java:423)
>         at org.apache.iceberg.BaseTransaction.commitTransaction(BaseTransaction.java:318)
>         at org.apache.impala.service.CatalogOpExecutor.insertIntoIcebergTable(CatalogOpExecutor.java:7593)
>         at org.apache.impala.service.CatalogOpExecutor.updateCatalogImpl(CatalogOpExecutor.java:7386)
>         at org.apache.impala.service.CatalogOpExecutor.updateCatalog(CatalogOpExecutor.java:7227)
>         at org.apache.impala.service.JniCatalog.lambda$updateCatalog$15(JniCatalog.java:510)
>         at org.apache.impala.service.JniCatalogOp.lambda$execAndSerialize$1(JniCatalogOp.java:90)
>         at org.apache.impala.service.JniCatalogOp.execOp(JniCatalogOp.java:58)
>         at org.apache.impala.service.JniCatalogOp.execAndSerialize(JniCatalogOp.java:89)
>         at org.apache.impala.service.JniCatalogOp.execAndSerialize(JniCatalogOp.java:100)
>         at org.apache.impala.service.JniCatalog.execAndSerialize(JniCatalog.java:251)
>         at org.apache.impala.service.JniCatalog.execAndSerialize(JniCatalog.java:265)
>         at org.apache.impala.service.JniCatalog.updateCatalog(JniCatalog.java:509)
> ...
> W0307 02:25:49.672634 18405 Tasks.java:456] 5f4ecf63310cb7f5:e2db615800000000] Retrying task after failure: Waiting for lock on table sys.impala_query_log{noformat}
> This might be related to IMPALA-13772 that disable FETCH_ROWS_TIMEOUT_MS option.
> During custom cluster test run, impala minicluster is restart between tests. If test does not wait until coordinator quiesce (all pending INSERT INTO sys.impala_query_log is done), it might be the case that HMS lock is being hold when CatalogD terminates.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org