You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Quanlong Huang (Jira)" <ji...@apache.org> on 2025/04/23 10:17:00 UTC
[jira] [Comment Edited] (IMPALA-9135) DDLs with sync_ddl may fail with concurrent INVALIDATE METADATA
[ https://issues.apache.org/jira/browse/IMPALA-9135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17946681#comment-17946681 ]
Quanlong Huang edited comment on IMPALA-9135 at 4/23/25 10:16 AM:
------------------------------------------------------------------
If we want to see the error message of "Couldn't retrieve the catalog topic update for the SYNC_DDL operation", we can start catalogd with non-zero max_wait_time_for_sync_ddl_s:
{noformat}
bin/start-impala-cluster.py --catalogd_args=--max_wait_time_for_sync_ddl_s=10 {noformat}
Then redo the above test and run DESCRIBE on any other tables to trigger a new catalog topic update. The INSERT will fail:
{code:java}
SYNC_DDL set to 1
Query: insert into tbl values (0)
Query submitted at: 2025-04-23 18:12:28 (Coordinator: http://quanlong-Precision-3680:25000)
Query state can be monitored at: http://quanlong-Precision-3680:25000/query_plan?query_id=1f4da2f1236dc711:eeb9b4c500000000
2025-04-23 18:13:31 [Exception] ERROR: Query 1f4da2f1236dc711:eeb9b4c500000000 failed:
CatalogException: Couldn't retrieve the catalog topic update for the SYNC_DDL operation after 2 attempts and elapsed time of 52127 msec. The operation has been successfully executed but its effects may have not been broadcast to all the coordinators. {code}
was (Author: stiga-huang):
If we want to see the error message of, we can start catalogd with non-zero max_wait_time_for_sync_ddl_s:
{noformat}
bin/start-impala-cluster.py --catalogd_args=--max_wait_time_for_sync_ddl_s=10 {noformat}
Then redo the above test and run DESCRIBE on any other tables to trigger a new catalog topic update. The INSERT will fail:
{code}
SYNC_DDL set to 1
Query: insert into tbl values (0)
Query submitted at: 2025-04-23 18:12:28 (Coordinator: http://quanlong-Precision-3680:25000)
Query state can be monitored at: http://quanlong-Precision-3680:25000/query_plan?query_id=1f4da2f1236dc711:eeb9b4c500000000
2025-04-23 18:13:31 [Exception] ERROR: Query 1f4da2f1236dc711:eeb9b4c500000000 failed:
CatalogException: Couldn't retrieve the catalog topic update for the SYNC_DDL operation after 2 attempts and elapsed time of 52127 msec. The operation has been successfully executed but its effects may have not been broadcast to all the coordinators. {code}
> DDLs with sync_ddl may fail with concurrent INVALIDATE METADATA
> ---------------------------------------------------------------
>
> Key: IMPALA-9135
> URL: https://issues.apache.org/jira/browse/IMPALA-9135
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Reporter: Quanlong Huang
> Priority: Major
> Labels: concurrency
>
> This can be revealed by tests/custom_cluster/test_concurrent_ddls.py added in [https://gerrit.cloudera.org/c/14307]
> If running with INVALIDATE METADATA concurrently, the DDLs may run out of attemps in CatalogServiceCatalog.waitForSyncDdlVersion() to wait for the target update being sent, no matter how large we increase the maxNumAttempts.
> The error logs:
> {code:java}
> E1107 17:34:25.092439 7353 CatalogServiceCatalog.java:2626] Couldn't retrieve the covering topic version for catalog objects. Updated objects: [TABLE:test_ddls_with_invalidate_metadata_sync_ddl_f41e97e6.test_9_part version: 349], deleted objects: []
> I1107 17:34:25.093451 7353 jni-util.cc:288] org.apache.impala.catalog.CatalogException: Couldn't retrieve the catalog topic version for the SYNC_DDL operation after 5 attempts.The operation has been successfully executed but its effects may have not been broadcast to all the coordinators.
> at org.apache.impala.catalog.CatalogServiceCatalog.waitForSyncDdlVersion(CatalogServiceCatalog.java:2630)
> at org.apache.impala.service.CatalogOpExecutor.execDdlRequest(CatalogOpExecutor.java:414)
> at org.apache.impala.service.JniCatalog.execDdl(JniCatalog.java:167)
> I1107 17:34:25.142006 6389 catalog-server.cc:337] A catalog update with 2 entries is assembled. Catalog version: 356 Last sent catalog version: 355
> I1107 17:34:25.142168 6381 catalog-server.cc:641] Collected update: 1:TABLE:test_ddls_with_invalidate_metadata_sync_ddl_f41e97e6.test_15_part, version=357, original size=101, compressed size=98
> I1107 17:34:25.142215 6381 catalog-server.cc:641] Collected update: 1:CATALOG_SERVICE_ID, version=357, original size=49, compressed size=52
> I1107 17:34:25.142287 7356 CatalogServiceCatalog.java:2642] Operation using SYNC_DDL is waiting for catalog topic version: 357. Time to identify topic version (msec): 19
> I1107 17:34:25.192239 6389 catalog-server.cc:337] A catalog update with 2 entries is assembled. Catalog version: 357 Last sent catalog version: 356
> I1107 17:34:25.192428 6381 catalog-server.cc:641] Collected update: 1:TABLE:test_ddls_with_invalidate_metadata_sync_ddl_f41e97e6.test_16_part, version=358, original size=101, compressed size=98
> I1107 17:34:25.192462 6381 catalog-server.cc:641] Collected update: 1:TABLE:test_ddls_with_invalidate_metadata_sync_ddl_f41e97e6.test_11_part, version=359, original size=101, compressed size=98
> I1107 17:34:25.192484 6381 catalog-server.cc:641] Collected update: 1:TABLE:test_ddls_with_invalidate_metadata_sync_ddl_f41e97e6.test_12_part, version=360, original size=101, compressed size=98
> I1107 17:34:25.192535 6381 catalog-server.cc:641] Collected update: 1:CATALOG_SERVICE_ID, version=360, original size=49, compressed size=52
> I1107 17:34:25.192613 7355 CatalogServiceCatalog.java:2642] Operation using SYNC_DDL is waiting for catalog topic version: 360. Time to identify topic version (msec): 13
> I1107 17:34:25.192695 7351 CatalogServiceCatalog.java:2642] Operation using SYNC_DDL is waiting for catalog topic version: 360. Time to identify topic version (msec): 45
> I1107 17:34:25.192734 7350 CatalogServiceCatalog.java:2642] Operation using SYNC_DDL is waiting for catalog topic version: 360. Time to identify topic version (msec): 29
> I1107 17:34:25.222911 7353 status.cc:126] CatalogException: Couldn't retrieve the catalog topic version for the SYNC_DDL operation after 5 attempts.The operation has been successfully executed but its effects may have not been broadcast to all the coordinators.
> @ 0x1c5ae50 impala::Status::Status()
> @ 0x24f7ad2 impala::JniUtil::GetJniExceptionMsg()
> @ 0x1c41987 impala::JniCall::Call<>()
> @ 0x1c3fec9 impala::JniUtil::CallJniMethod<>()
> @ 0x1c3e0e6 impala::Catalog::ExecDdl()
> @ 0x1c1ed17 CatalogServiceThriftIf::ExecDdl()
> @ 0x1cb3047 impala::CatalogServiceProcessor::process_ExecDdl()
> @ 0x1cb2d95 impala::CatalogServiceProcessor::dispatchCall()
> @ 0x1c08d65 apache::thrift::TDispatchProcessor::process()
> @ 0x20e8a0d apache::thrift::server::TAcceptQueueServer::Task::run()
> @ 0x20de040 impala::ThriftThread::RunRunnable()
> @ 0x20df766 boost::_mfi::mf2<>::operator()()
> @ 0x20df5fc boost::_bi::list3<>::operator()<>()
> @ 0x20df348 boost::_bi::bind_t<>::operator()()
> @ 0x20df25b boost::detail::function::void_function_obj_invoker0<>::invoke()
> @ 0x1ffb6e9 boost::function0<>::operator()()
> @ 0x2573dea impala::Thread::SuperviseThread()
> @ 0x257c16e boost::_bi::list5<>::operator()<>()
> @ 0x257c092 boost::_bi::bind_t<>::operator()()
> @ 0x257c055 boost::detail::thread_data<>::run()
> @ 0x3d61599 thread_proxy
> @ 0x7f1ce6ca46b9 start_thread
> @ 0x7f1ce343f41c clone
> E1107 17:34:25.222932 7353 catalog-server.cc:112] CatalogException: Couldn't retrieve the catalog topic version for the SYNC_DDL operation after 5 attempts.The operation has been successfully executed but its effects may have not been broadcast to all the coordinators.
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org