You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@impala.apache.org by 汲广熙 <28...@qq.com> on 2026/01/05 12:35:03 UTC
Impala 4.5 Iceberg table on Tencent Cloud COS fails with "failed to load 1 paths" during metadata refresh.
Subject: Re: [Bug?] Impala 4.5 Iceberg table loading failure on Tencent Cloud COS (cosn://)
Hi,
Thank you for the suggestions. I have some critical updates regarding the TableLoadingException: failed to load 1 paths issue on Tencent Cloud COS (cosn://).
I have already enabled Local Catalog Mode (--use_local_catalog=true and minimal topic mode), but the issue persists.
Based on an analysis provided by Gemini (Google's AI model) regarding the stack trace, it was suggested that the error in ParallelFileMetadataLoader.loadInternal might be caused by concurrency issues when handling object storage paths. Following this logic, I tried to apply a workaround by setting metadata_loader_parallelism=1 via ALTER TABLE.
However, the command failed with the same AnalysisException and TableLoadingException. It seems Impala falls into a deadlock state: I cannot modify the table properties to fix the loading logic because Impala fails to load the table metadata even during the ALTER TABLE analysis phase.
Observations:
The error originates from ParallelFileMetadataLoader.loadInternal.
Even with Local Catalog enabled, the loader seems unable to handle the missing block location info from the cosn:// driver.
Standard Parquet tables work fine; this appears specific to the Iceberg metadata loading path in Impala 4.5.
Question: Is there a way to globally set metadata_loader_parallelism=1 for the entire Catalogd, or perhaps a flag to force Impala to ignore block location errors for object storage during the initial load?
I am happy to provide a full stack trace or open a JIRA ticket, as this currently makes Iceberg on COS unusable in our environment.
============================================================================================================
When I run a simple SELECT *or INSERT INTOquery on this table, I get the following error:
AnalysisException: Failed to load metadata for table: 'iceberg_cos_employee_test'
...
IcebergTableLoadingException: Error loading metadata for Iceberg table cosn://cdnlogtest-1252412955/impala_test_db/iceberg_cos_employee_test
...
Loading file and block metadata for 1 paths for table ...: failed to load 1 paths.
============================================================================================================
Best regards,
原始邮件
发件人：Zoltán Borók-Nagy <boroknagyz@apache.org>
发件时间：2025年12月23日 23:52
收件人：dev <dev@impala.apache.org>
主题：Re: Advice and Considerations for Building an Impala Compute-StorageSeparated Architecture
Hi,
Just a few tips off the top of my head:
- use dedicated coordinators and executors, rule of thumb for
coordinator:executor ratio is 1:50. Though for HA you probably want >1
coordinators.
- use local catalog mode (aka On-demand metadata):
https://impala.apache.org/docs/build/html/topics/impala_metadata.html
- enabling remote data cache (with SSD disks) is essential in
compute-storage separated setup:
https://impala.apache.org/docs/build/html/topics/impala_data_cache.html
What table format / file format are you planning to use?
If table format is Iceberg, make sure you use the latest Impala as we
continuously improve Impala's performance on Iceberg.
File format: Impala most efficiently works on Parquet files.
Avoid small file issues:
- choose proper partitioning for your data, i.e. avoid too coarse-grained
and too fine-grained partitioning. I.e. you probably want more than 200 MB
data per partition, but probably less than 20 GB.
- compact your tables regularly, for Iceberg tables Impala has the
OPTIMIZE statement:
https://impala.apache.org/docs/build/html/topics/impala_iceberg.html
I hope others chime in as well.
We would love to hear back about your experiences, and feel free to open
tickets for Impala if you run into any issue:
https://issues.apache.org/jira/projects/IMPALA/issues
Cheers,
Zoltan
On Tue, Dec 23, 2025 at 8:23 AM 汲广熙 <2870811759@qq.com> wrote:
> Dear Impala Team,
>
> I hope this message finds you well.
>
> I am currently planning to build a compute-storage separated architecture
> based on Apache Impala. In this setup:
>
>
> Compute layer: Apache Impala will be used for SQL query execution.
>
>
>
> Storage layer: Tencent Cloud Object Storage (COS) will serve as the
> backend storage.
>
>
>
> Data ingestion: Kafka will be used to stream data into the system.
>
>
>
> Monitoring &amp; visualization: Grafana will be used to display
> operational and performance metrics.
>
>
> Could you please provide some recommendations and key considerations for
> such an architecture? Specifically, I would appreciate guidance on:
>
>
> Best practices for integrating Impala with cloud object storage (COS).
>
>
>
> Performance tuning tips for Impala in a disaggregated environment.
>
>
>
> Any known limitations or compatibility issues when using COS as storage.
>
>
>
> Recommended configurations for Kafka-to-Impala data pipelines.
>
>
>
> Monitoring strategies for tracking query performance and resource usage in
> Grafana.
>
>
> Thank you very much for your support and advice. I look forward to your
> reply.
>
> Best regards
Re: Impala 4.5 Iceberg table on Tencent Cloud COS fails with"failedto load 1 paths" during metadata refresh.
Posted by 汲广熙 <28...@qq.com>.
Subject: Re: [Bug?] Impala 4.5 Iceberg table loading failure on Tencent Cloud COS (cosn://)
Hi Zoltan,
Thank you for the reminder. I have just filed a Jira ticket to track this issue:https://issues.apache.org/jira/browse/IMPALA-14659
For the convenience of the mailing list, I am copy-pasting the core stack trace from the catalogd logs below. As mentioned earlier, this exception occurs even when parallelism is restricted to 1:
I0105 22:28:30.371518 45789 status.cc:129] IcebergTableLoadingException: Error loading metadata for Iceberg table cosn://cdnlogtest-1252412955/impala_test_db/iceberg_cos_employee_test
CAUSED BY: TableLoadingException: Loading file and block metadata for 1 paths for table db_on_cos_test.iceberg_cos_employee_test: failed to load 1 paths. Check the catalog server log for more details.
@ 0x10a31d4
@ 0x1c1a17a
@ 0x104e830
@ 0xff5fd7
@ 0xfa8511
@ 0xf83d33
@ 0xf8b7fb
@ 0x14a8a42
@ 0x1cff659
@ 0x264d407
@ 0x7faf31128ea5
@ 0x7faf2dfb29fd
E0105 22:28:30.371537 45789 catalog-server.cc:377] IcebergTableLoadingException: Error loading metadata for Iceberg table cosn://cdnlogtest-1252412955/impala_test_db/iceberg_cos_employee_test
CAUSED BY: TableLoadingException: Loading file and block metadata for 1 paths for table db_on_cos_test.iceberg_cos_employee_test: failed to load 1 paths. Check the catalog server log for more details.
I0105 22:28:33.177121 45789 jni-util.cc:321] org.apache.impala.catalog.IcebergTableLoadingException: Error loading metadata for Iceberg table cosn://cdnlogtest-1252412955/impala_test_db/iceberg_cos_employee_test
at org.apache.impala.catalog.IcebergTable.load(IcebergTable.java:460)
at org.apache.impala.catalog.TableLoader.load(TableLoader.java:145)
at org.apache.impala.catalog.TableLoadingMgr$2.call(TableLoadingMgr.java:251)
at org.apache.impala.catalog.TableLoadingMgr$2.call(TableLoadingMgr.java:247)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.impala.catalog.TableLoadingException: Loading file and block metadata for 1 paths for table db_on_cos_test.iceberg_cos_employee_test: failed to load 1 paths. Check the catalog server log for more details.
at org.apache.impala.catalog.ParallelFileMetadataLoader.loadInternal(ParallelFileMetadataLoader.java:211)
at org.apache.impala.catalog.ParallelFileMetadataLoader.load(ParallelFileMetadataLoader.java:140)
at org.apache.impala.catalog.HdfsTable.loadFileMetadataForPartitions(HdfsTable.java:827)
at org.apache.impala.catalog.HdfsTable.loadFileMetadataForPartitions(HdfsTable.java:780)
at org.apache.impala.catalog.HdfsTable.loadAllPartitions(HdfsTable.java:754)
at org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1371)
at org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1262)
at org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1251)
at org.apache.impala.catalog.IcebergTable.load(IcebergTable.java:449)
... 7 more
I0105 22:28:33.177153 45789 status.cc:129] IcebergTableLoadingException: Error loading metadata for Iceberg table cosn://cdnlogtest-1252412955/impala_test_db/iceberg_cos_employee_test
CAUSED BY: TableLoadingException: Loading file and block metadata for 1 paths for table db_on_cos_test.iceberg_cos_employee_test: failed to load 1 paths. Check the catalog server log for more details.
@ 0x10a31d4
@ 0x1c1a17a
@ 0x104e830
@ 0xff5fd7
@ 0xfa8511
@ 0xf83d33
@ 0xf8b7fb
@ 0x14a8a42
@ 0x1cff659
@ 0x264d407
@ 0x7faf31128ea5
@ 0x7faf2dfb29fd
E0105 22:28:33.177165 45789 catalog-server.cc:377] IcebergTableLoadingException: Error loading metadata for Iceberg table cosn://cdnlogtest-1252412955/impala_test_db/iceberg_cos_employee_test
CAUSED BY: TableLoadingException: Loading file and block metadata for 1 paths for table db_on_cos_test.iceberg_cos_employee_test: failed to load 1 paths. Check the catalog server log for more details.
W0105 22:28:35.111603 25279 minidump.cc:202] Failed to open file "/tmp/minidumps/catalogd/7e4ed6a5-0c24-4a24-4c0147b7-ab48e83f.dmp". Removing it.
W0105 22:28:35.111624 25279 minidump.cc:202] Failed to open file "/tmp/minidumps/catalogd/931cac7e-3669-48bc-bc68b185-77bd28e4.dmp". Removing it.
W0105 22:28:35.111640 25279 minidump.cc:202] Failed to open file "/tmp/minidumps/catalogd/9a535c7e-c762-4df8-0a6e1689-de0fbea0.dmp". Removing it.
I0105 22:28:58.031069 45789 jni-util.cc:321] org.apache.impala.catalog.IcebergTableLoadingException: Error loading metadata for Iceberg table cosn://cdnlogtest-1252412955/impala_test_db/iceberg_cos_employee_test
at org.apache.impala.catalog.IcebergTable.load(IcebergTable.java:460)
at org.apache.impala.catalog.TableLoader.load(TableLoader.java:145)
at org.apache.impala.catalog.TableLoadingMgr$2.call(TableLoadingMgr.java:251)
at org.apache.impala.catalog.TableLoadingMgr$2.call(TableLoadingMgr.java:247)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.impala.catalog.TableLoadingException: Loading file and block metadata for 1 paths for table db_on_cos_test.iceberg_cos_employee_test: failed to load 1 paths. Check the catalog server log for more details.
at org.apache.impala.catalog.ParallelFileMetadataLoader.loadInternal(ParallelFileMetadataLoader.java:211)
at org.apache.impala.catalog.ParallelFileMetadataLoader.load(ParallelFileMetadataLoader.java:140)
at org.apache.impala.catalog.HdfsTable.loadFileMetadataForPartitions(HdfsTable.java:827)
at org.apache.impala.catalog.HdfsTable.loadFileMetadataForPartitions(HdfsTable.java:780)
at org.apache.impala.catalog.HdfsTable.loadAllPartitions(HdfsTable.java:754)
at org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1371)
at org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1262)
at org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1251)
at org.apache.impala.catalog.IcebergTable.load(IcebergTable.java:449)
... 7 more
I0105 22:28:58.031085 45789 status.cc:129] IcebergTableLoadingException: Error loading metadata for Iceberg table cosn://cdnlogtest-1252412955/impala_test_db/iceberg_cos_employee_test
CAUSED BY: TableLoadingException: Loading file and block metadata for 1 paths for table db_on_cos_test.iceberg_cos_employee_test: failed to load 1 paths. Check the catalog server log for more details.
@ 0x10a31d4
@ 0x1c1a17a
@ 0x104e830
@ 0xff5fd7
@ 0xfa8511
@ 0xf83d33
@ 0xf8b7fb
@ 0x14a8a42
@ 0x1cff659
@ 0x264d407
@ 0x7faf31128ea5
@ 0x7faf2dfb29fd
E0105 22:28:58.031098 45789 catalog-server.cc:377] IcebergTableLoadingException: Error loading metadata for Iceberg table cosn://cdnlogtest-1252412955/impala_test_db/iceberg_cos_employee_test
CAUSED BY: TableLoadingException: Loading file and block metadata for 1 paths for table db_on_cos_test.iceberg_cos_employee_test: failed to load 1 paths. Check the catalog server log for more details.
W0105 22:29:05.124416 25279 minidump.cc:202] Failed to open file "/tmp/minidumps/catalogd/7e4ed6a5-0c24-4a24-4c0147b7-ab48e83f.dmp". Removing it.
W0105 22:29:05.124442 25279 minidump.cc:202] Failed to open file "/tmp/minidumps/catalogd/931cac7e-3669-48bc-bc68b185-77bd28e4.dmp". Removing it.
W0105 22:29:05.124451 25279 minidump.cc:202] Failed to open file "/tmp/minidumps/catalogd/9a535c7e-c762-4df8-0a6e1689-de0fbea0.dmp". Removing it.
I0105 22:29:08.762989 45789 jni-util.cc:321] org.apache.impala.catalog.IcebergTableLoadingException: Error loading metadata for Iceberg table cosn://cdnlogtest-1252412955/impala_test_db/iceberg_cos_employee_test
at org.apache.impala.catalog.IcebergTable.load(IcebergTable.java:460)
at org.apache.impala.catalog.TableLoader.load(TableLoader.java:145)
at org.apache.impala.catalog.TableLoadingMgr$2.call(TableLoadingMgr.java:251)
at org.apache.impala.catalog.TableLoadingMgr$2.call(TableLoadingMgr.java:247)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.impala.catalog.TableLoadingException: Loading file and block metadata for 1 paths for table db_on_cos_test.iceberg_cos_employee_test: failed to load 1 paths. Check the catalog server log for more details.
at org.apache.impala.catalog.ParallelFileMetadataLoader.loadInternal(ParallelFileMetadataLoader.java:211)
at org.apache.impala.catalog.ParallelFileMetadataLoader.load(ParallelFileMetadataLoader.java:140)
at org.apache.impala.catalog.HdfsTable.loadFileMetadataForPartitions(HdfsTable.java:827)
at org.apache.impala.catalog.HdfsTable.loadFileMetadataForPartitions(HdfsTable.java:780)
at org.apache.impala.catalog.HdfsTable.loadAllPartitions(HdfsTable.java:754)
at org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1371)
at org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1262)
at org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1251)
at org.apache.impala.catalog.IcebergTable.load(IcebergTable.java:449)
... 7 more
I0105 22:29:08.763005 45789 status.cc:129] IcebergTableLoadingException: Error loading metadata for Iceberg table cosn://cdnlogtest-1252412955/impala_test_db/iceberg_cos_employee_test
CAUSED BY: TableLoadingException: Loading file and block metadata for 1 paths for table db_on_cos_test.iceberg_cos_employee_test: failed to load 1 paths. Check the catalog server log for more details.
@ 0x10a31d4
@ 0x1c1a17a
@ 0x104e830
@ 0xff5fd7
@ 0xfa8511
@ 0xf83d33
@ 0xf8b7fb
@ 0x14a8a42
@ 0x1cff659
@ 0x264d407
@ 0x7faf31128ea5
@ 0x7faf2dfb29fd
E0105 22:29:08.763021 45789 catalog-server.cc:377] IcebergTableLoadingException: Error loading metadata for Iceberg table cosn://cdnlogtest-1252412955/impala_test_db/iceberg_cos_employee_test
CAUSED BY: TableLoadingException: Loading file and block metadata for 1 paths for table db_on_cos_test.iceberg_cos_employee_test: failed to load 1 paths. Check the catalog server log for more details.
W0105 22:29:35.137226 25279 minidump.cc:202] Failed to open file "/tmp/minidumps/catalogd/7e4ed6a5-0c24-4a24-4c0147b7-ab48e83f.dmp". Removing it.
W0105 22:29:35.137250 25279 minidump.cc:202] Failed to open file "/tmp/minidumps/catalogd/931cac7e-3669-48bc-bc68b185-77bd28e4.dmp". Removing it.
W0105 22:29:35.137259 25279 minidump.cc:202] Failed to open file "/tmp/minidumps/catalogd/9a535c7e-c762-4df8-0a6e1689-de0fbea0.dmp". Removing it.
W0105 22:30:05.150115 25279 minidump.cc:202] Failed to open file "/tmp/minidumps/catalogd/7e4ed6a5-0c24-4a24-4c0147b7-ab48e83f.dmp". Removing it.
W0105 22:30:05.150141 25279 minidump.cc:202] Failed to open file "/tmp/minidumps/catalogd/931cac7e-3669-48bc-bc68b185-77bd28e4.dmp". Removing it.
W0105 22:30:05.150151 25279 minidump.cc:202] Failed to open file "/tmp/minidumps/catalogd/9a535c7e-c762-4df8-0a6e1689-de0fbea0.dmp". Removing it.
W0105 22:30:35.162889 25279 minidump.cc:202] Failed to open file "/tmp/minidumps/catalogd/7e4ed6a5-0c24-4a24-4c0147b7-ab48e83f.dmp". Removing it.
W0105 22:30:35.162916 25279 minidump.cc:202] Failed to open file "/tmp/minidumps/catalogd/931cac7e-3669-48bc-bc68b185-77bd28e4.dmp". Removing it.
W0105 22:30:35.162925 25279 minidump.cc:202] Failed to open file "/tmp/minidumps/catalogd/9a535c7e-c762-4df8-0a6e1689-de0fbea0.dmp". Removing it.
I0105 22:30:42.056529 45789 jni-util.cc:321] org.apache.impala.catalog.IcebergTableLoadingException: Error loading metadata for Iceberg table cosn://cdnlogtest-1252412955/impala_test_db/iceberg_cos_employee_test
at org.apache.impala.catalog.IcebergTable.load(IcebergTable.java:460)
at org.apache.impala.catalog.TableLoader.load(TableLoader.java:145)
at org.apache.impala.catalog.TableLoadingMgr$2.call(TableLoadingMgr.java:251)
at org.apache.impala.catalog.TableLoadingMgr$2.call(TableLoadingMgr.java:247)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.impala.catalog.TableLoadingException: Loading file and block metadata for 1 paths for table db_on_cos_test.iceberg_cos_employee_test: failed to load 1 paths. Check the catalog server log for more details.
at org.apache.impala.catalog.ParallelFileMetadataLoader.loadInternal(ParallelFileMetadataLoader.java:211)
at org.apache.impala.catalog.ParallelFileMetadataLoader.load(ParallelFileMetadataLoader.java:140)
at org.apache.impala.catalog.HdfsTable.loadFileMetadataForPartitions(HdfsTable.java:827)
at org.apache.impala.catalog.HdfsTable.loadFileMetadataForPartitions(HdfsTable.java:780)
at org.apache.impala.catalog.HdfsTable.loadAllPartitions(HdfsTable.java:754)
at org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1371)
at org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1262)
at org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1251)
at org.apache.impala.catalog.IcebergTable.load(IcebergTable.java:449)
... 7 more
I0105 22:30:42.056545 45789 status.cc:129] IcebergTableLoadingException: Error loading metadata for Iceberg table cosn://cdnlogtest-1252412955/impala_test_db/iceberg_cos_employee_test
CAUSED BY: TableLoadingException: Loading file and block metadata for 1 paths for table db_on_cos_test.iceberg_cos_employee_test: failed to load 1 paths. Check the catalog server log for more details.
@ 0x10a31d4
@ 0x1c1a17a
@ 0x104e830
@ 0xff5fd7
@ 0xfa8511
@ 0xf83d33
@ 0xf8b7fb
@ 0x14a8a42
@ 0x1cff659
@ 0x264d407
@ 0x7faf31128ea5
@ 0x7faf2dfb29fd
E0105 22:30:42.056563 45789 catalog-server.cc:377] IcebergTableLoadingException: Error loading metadata for Iceberg table cosn://cdnlogtest-1252412955/impala_test_db/iceberg_cos_employee_test
CAUSED BY: TableLoadingException: Loading file and block metadata for 1 paths for table db_on_cos_test.iceberg_cos_employee_test: failed to load 1 paths. Check the catalog server log for more details.
I0105 22:31:00.136520 45789 jni-util.cc:321] org.apache.impala.catalog.IcebergTableLoadingException: Error loading metadata for Iceberg table cosn://cdnlogtest-1252412955/impala_test_db/iceberg_cos_employee_test
at org.apache.impala.catalog.IcebergTable.load(IcebergTable.java:460)
at org.apache.impala.catalog.TableLoader.load(TableLoader.java:145)
at org.apache.impala.catalog.TableLoadingMgr$2.call(TableLoadingMgr.java:251)
at org.apache.impala.catalog.TableLoadingMgr$2.call(TableLoadingMgr.java:247)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.impala.catalog.TableLoadingException: Loading file and block metadata for 1 paths for table db_on_cos_test.iceberg_cos_employee_test: failed to load 1 paths. Check the catalog server log for more details.
at org.apache.impala.catalog.ParallelFileMetadataLoader.loadInternal(ParallelFileMetadataLoader.java:211)
at org.apache.impala.catalog.ParallelFileMetadataLoader.load(ParallelFileMetadataLoader.java:140)
at org.apache.impala.catalog.HdfsTable.loadFileMetadataForPartitions(HdfsTable.java:827)
at org.apache.impala.catalog.HdfsTable.loadFileMetadataForPartitions(HdfsTable.java:780)
at org.apache.impala.catalog.HdfsTable.loadAllPartitions(HdfsTable.java:754)
at org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1371)
at org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1262)
at org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1251)
at org.apache.impala.catalog.IcebergTable.load(IcebergTable.java:449)
... 7 more
I0105 22:31:00.136550 45789 status.cc:129] IcebergTableLoadingException: Error loading metadata for Iceberg table cosn://cdnlogtest-1252412955/impala_test_db/iceberg_cos_employee_test
CAUSED BY: TableLoadingException: Loading file and block metadata for 1 paths for table db_on_cos_test.iceberg_cos_employee_test: failed to load 1 paths. Check the catalog server log for more details.
@ 0x10a31d4
@ 0x1c1a17a
@ 0x104e830
@ 0xff5fd7
@ 0xfa8511
@ 0xf83d33
@ 0xf8b7fb
@ 0x14a8a42
@ 0x1cff659
@ 0x264d407
@ 0x7faf31128ea5
@ 0x7faf2dfb29fd
E0105 22:31:00.136565 45789 catalog-server.cc:377] IcebergTableLoadingException: Error loading metadata for Iceberg table cosn://cdnlogtest-1252412955/impala_test_db/iceberg_cos_employee_test
CAUSED BY: TableLoadingException: Loading file and block metadata for 1 paths for table db_on_cos_test.iceberg_cos_employee_test: failed to load 1 paths. Check the catalog server log for more details.
W0105 22:31:05.175676 25279 minidump.cc:202] Failed to open file "/tmp/minidumps/catalogd/7e4ed6a5-0c24-4a24-4c0147b7-ab48e83f.dmp". Removing it.
W0105 22:31:05.175701 25279 minidump.cc:202] Failed to open file "/tmp/minidumps/catalogd/931cac7e-3669-48bc-bc68b185-77bd28e4.dmp". Removing it.
W0105 22:31:05.175710 25279 minidump.cc:202] Failed to open file "/tmp/minidumps/catalogd/9a535c7e-c762-4df8-0a6e1689-de0fbea0.dmp". Removing it.
I0105 22:31:14.446766 45789 jni-util.cc:321] org.apache.impala.catalog.IcebergTableLoadingException: Error loading metadata for Iceberg table cosn://cdnlogtest-1252412955/impala_test_db/iceberg_cos_employee_test
at org.apache.impala.catalog.IcebergTable.load(IcebergTable.java:460)
at org.apache.impala.catalog.TableLoader.load(TableLoader.java:145)
at org.apache.impala.catalog.TableLoadingMgr$2.call(TableLoadingMgr.java:251)
at org.apache.impala.catalog.TableLoadingMgr$2.call(TableLoadingMgr.java:247)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.impala.catalog.TableLoadingException: Loading file and block metadata for 1 paths for table db_on_cos_test.iceberg_cos_employee_test: failed to load 1 paths. Check the catalog server log for more details.
at org.apache.impala.catalog.ParallelFileMetadataLoader.loadInternal(ParallelFileMetadataLoader.java:211)
at org.apache.impala.catalog.ParallelFileMetadataLoader.load(ParallelFileMetadataLoader.java:140)
at org.apache.impala.catalog.HdfsTable.loadFileMetadataForPartitions(HdfsTable.java:827)
at org.apache.impala.catalog.HdfsTable.loadFileMetadataForPartitions(HdfsTable.java:780)
at org.apache.impala.catalog.HdfsTable.loadAllPartitions(HdfsTable.java:754)
at org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1371)
at org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1262)
at org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1251)
at org.apache.impala.catalog.IcebergTable.load(IcebergTable.java:449)
... 7 more
I0105 22:31:14.446784 45789 status.cc:129] IcebergTableLoadingException: Error loading metadata for Iceberg table cosn://cdnlogtest-1252412955/impala_test_db/iceberg_cos_employee_test
CAUSED BY: TableLoadingException: Loading file and block metadata for 1 paths for table db_on_cos_test.iceberg_cos_employee_test: failed to load 1 paths. Check the catalog server log for more details.
@ 0x10a31d4
@ 0x1c1a17a
@ 0x104e830
@ 0xff5fd7
@ 0xfa8511
@ 0xf83d33
@ 0xf8b7fb
@ 0x14a8a42
@ 0x1cff659
@ 0x264d407
@ 0x7faf31128ea5
@ 0x7faf2dfb29fd
E0105 22:31:14.446801 45789 catalog-server.cc:377] IcebergTableLoadingException: Error loading metadata for Iceberg table cosn://cdnlogtest-1252412955/impala_test_db/iceberg_cos_employee_test
CAUSED BY: TableLoadingException: Loading file and block metadata for 1 paths for table db_on_cos_test.iceberg_cos_employee_test: failed to load 1 paths. Check the catalog server log for more details.
W0105 22:31:35.188406 25279 minidump.cc:202] Failed to open file "/tmp/minidumps/catalogd/7e4ed6a5-0c24-4a24-4c0147b7-ab48e83f.dmp". Removing it.
W0105 22:31:35.188434 25279 minidump.cc:202] Failed to open file "/tmp/minidumps/catalogd/931cac7e-3669-48bc-bc68b185-77bd28e4.dmp". Removing it.
W0105 22:31:35.188443 25279 minidump.cc:202] Failed to open file "/tmp/minidumps/catalogd/9a535c7e-c762-4df8-0a6e1689-de0fbea0.dmp". Removing it.
原始邮件
发件人：Zoltán Borók-Nagy <boroknagyz@apache.org>
发件时间：2026年1月5日 23:19
收件人：dev <dev@impala.apache.org>
主题：Re: Impala 4.5 Iceberg table on Tencent Cloud COS fails with"failedto load 1 paths" during metadata refresh.
Thanks for your reply. Yeah, if Impala cannot load a table for some reason,
it cannot really do anything with it.
Please note that attachments don't get through Apache mailing lists.
Please copy-paste the stack trace of the exception from the CatalogD logs.
Or, feel free to file a Jira ticket about this issue.
Cheers,
Zoltan
On Mon, Jan 5, 2026 at 3:51 PM 汲广熙 <2870811759@qq.com> wrote:
> *Subject:* Re: [Bug?] Impala 4.5 Iceberg table loading failure on Tencent
> Cloud COS (cosn://)
>
> *Hi,*
>
> Thank you for the guidance. I have an important update regarding the
> backend flags you suggested.
>
> I applied --max_nonhdfs_partitions_parallel_load=1 and
> --max_hdfs_partitions_parallel_load=1 to my catalogd_flags and restarted
> the service. Unfortunately, the issue persists even with single-threaded
> loading.
>
> It seems that in Impala 4.5, ParallelFileMetadataLoader encounters
> difficulties when processing metadata returned by the cosn:// driver,
> regardless of the thread count. This suggests there might be a specific
> compatibility issue with how file/block metadata is interpreted for this
> storage scheme.
>
> Below are the raw logs from my session showing that *SHOW METADATA*,
> *INSERT*, and *SELECT* all trigger the same metadata loading error:
>
> ======================================
> =======================================================================================================
> [9.192.163.17:21050] db_on_cos_test> SHOW METADATA TABLES IN
> iceberg_cos_employee_test;
> Query: SHOW METADATA TABLES IN iceberg_cos_employee_test
> 2026-01-05 22:28:58 [Exception] ERROR: Query
> 9349f0ef1c788ec2:8548051e00000000 failed:
> AnalysisException: Could not load table
> db_on_cos_test.iceberg_cos_employee_test from catalog
> CAUSED BY: TableLoadingException: Could not load table
> db_on_cos_test.iceberg_cos_employee_test from catalog
> CAUSED BY: TException:
> TGetPartialCatalogObjectResponse(status:TStatus(status_code:GENERAL,
> error_msgs:[IcebergTableLoadingException: Error loading metadata for
> Iceberg table
> cosn://cdnlogtest-1252412955/impala_test_db/iceberg_cos_employee_test
> CAUSED BY: TableLoadingException: Loading file and block metadata for 1
> paths for table db_on_cos_test.iceberg_cos_employee_test: failed to load 1
> paths. Check the catalog server log for more details.]), lookup_status:OK)
>
> [9.192.163.17:21050] db_on_cos_test> INSERT INTO
> iceberg_cos_employee_test VALUES (1, '张三', '研发部', 15001.5, now());
> Query: INSERT INTO iceberg_cos_employee_test VALUES (1, '张三', '研发部',
> 15001.5, now())
> Query submitted at: 2026-01-05 22:29:08 (Coordinator:
> http://9.192.163.17:8080)
> 2026-01-05 22:29:08 [Exception] ERROR: Query
> ad4f18bf73bf1157:772c7c7600000000 failed:
> AnalysisException: org.apache.impala.catalog.TableLoadingException: Could
> not load table db_on_cos_test.iceberg_cos_employee_test from catalog
> CAUSED BY: TableLoadingException: Could not load table
> db_on_cos_test.iceberg_cos_employee_test from catalog
> CAUSED BY: TException:
> TGetPartialCatalogObjectResponse(status:TStatus(status_code:GENERAL,
> error_msgs:[IcebergTableLoadingException: Error loading metadata for
> Iceberg table
> cosn://cdnlogtest-1252412955/impala_test_db/iceberg_cos_employee_test
> CAUSED BY: TableLoadingException: Loading file and block metadata for 1
> paths for table db_on_cos_test.iceberg_cos_employee_test: failed to load 1
> paths. Check the catalog server log for more details.]), lookup_status:OK)
>
> [9.192.163.17:21050] db_on_cos_test> select * from
> iceberg_cos_employee_test;
> Query: select * from iceberg_cos_employee_test
> Query submitted at: 2026-01-05 22:30:42 (Coordinator:
> http://9.192.163.17:8080)
> 2026-01-05 22:30:42 [Exception] ERROR: Query
> 274fea8afd3f9dde:8d9847ab00000000 failed:
> AnalysisException: Failed to load metadata for table:
> 'iceberg_cos_employee_test'
> CAUSED BY: TableLoadingException: Could not load table
> db_on_cos_test.iceberg_cos_employee_test from catalog
> CAUSED BY: TException:
> TGetPartialCatalogObjectResponse(status:TStatus(status_code:GENERAL,
> error_msgs:[IcebergTableLoadingException: Error loading metadata for
> Iceberg table
> cosn://cdnlogtest-1252412955/impala_test_db/iceberg_cos_employee_test
> CAUSED BY: TableLoadingException: Loading file and block metadata for 1
> paths for table db_on_cos_test.iceberg_cos_employee_test: failed to load 1
> paths. Check the catalog server log for more details.]), lookup_status:OK)
> ======================================
> =======================================================================================================
>
> I have attached the full *catalogd logs* to this email for further
> investigation
>
>
> Best regards,
> 原始邮件
> ------------------------------
> 发件人：Zoltán Borók-Nagy <boroknagyz@apache.org>
> 发件时间：2026年1月5日 21:10
> 收件人：dev <dev@impala.apache.org>
> 主题：Re: Impala 4.5 Iceberg table on Tencent Cloud COS fails with "failedto
> load 1 paths" during metadata refresh.
>
> Thanks for reaching out.
>
> Please file a ticket about the error with stack traces and all
> necessary information (snippets from cagalogd logs) to further debug this.
>
> > I tried to apply a workaround by setting metadata_loader_parallelism=1
> via ALTER TABLE
> We don't have such a table property. Was it suggested by Gemini?
>
> > Question: Is there a way to globally set metadata_loader_parallelism=1
> for the entire Catalogd, or perhaps a flag to force Impala to ignore block
> location errors for object storage during the initial load?
> I suspect a different issue than a concurrency bug, but FWIW Catalogd has
> backend flags "max_hdfs_partitions_parallel_load" and
> "max_nonhdfs_partitions_parallel_load" which you can set:
>
> https://github.com/apache/impala/blob/master/be/src/catalog/catalog.cc#L39-L45
> This affects file metadata loading of non-Iceberg tables as well, but it
> can be useful to validate the bug.
>
> We improved Iceberg table loading since 4.5, and fixed a few issues, so
> there's a chance that the issue you ran into is already fixed.
>
> Would it be possible for you to try out the current Impala master?
> I think it is time for us to do another upstream release, but it would be
> nice if we can make sure your use case will work well with the newest
> release.
>
> Cheers,
> Zoltan
>
>
> On Mon, Jan 5, 2026 at 1:35 PM 汲广熙 <2870811759@qq.com> wrote:
>
> > Subject:&nbsp;Re: [Bug?] Impala 4.5 Iceberg table loading failure on
> > Tencent Cloud COS (cosn://)
> >
> > Hi,
> >
> > Thank you for the suggestions. I have some critical updates regarding the
> > TableLoadingException: failed to load 1 paths&nbsp;issue on Tencent Cloud
> > COS (cosn://).
> >
> > I have already enabled Local Catalog
>
> > Mode&nbsp;(--use_local_catalog=true&nbsp;and minimal&nbsp;topic mode), but
> > the issue persists.
> >
>
> > Based on an analysis provided by Gemini (Google's AI model)&nbsp;regarding
> > the stack trace, it was suggested that the error in
>
> > ParallelFileMetadataLoader.loadInternal&nbsp;might be caused by concurrency
>
> > issues when handling object storage paths. Following this logic, I tried to
>
> > apply a workaround by setting metadata_loader_parallelism=1&nbsp;via ALTER
> > TABLE.
> >
> > However, the command failed with the same AnalysisException&nbsp;and
> > TableLoadingException. It seems Impala falls into a deadlock state: I
>
> > cannot modify the table properties to fix the loading logic because Impala
>
> > fails to load the table metadata even during the ALTER TABLE&nbsp;analysis
> > phase.
> >
> > Observations:
> >
> >
> > The error originates from ParallelFileMetadataLoader.loadInternal.
> >
> >
> >
> > Even with Local Catalog enabled, the loader seems unable to handle the
> > missing block location info from the cosn://&nbsp;driver.
> >
> >
> >
> > Standard Parquet tables work fine; this appears specific to the
> > Iceberg&nbsp;metadata loading path in Impala 4.5.
> >
> >
> > Question: Is there a way to globally set
> > metadata_loader_parallelism=1&nbsp;for the entire Catalogd, or perhaps a
> > flag to force Impala to ignore block location errors for object storage
> > during the initial load?
> >
> > I am happy to provide a full stack trace or open a JIRA ticket, as this
> > currently makes Iceberg on COS unusable in our environment.
> >
> >
> >
> >
> >
>
> > ============================================================================================================
> > When I run a simple SELECT *or INSERT INTOquery on this table, I get the
> > following error:
> > AnalysisException: Failed to load metadata for table:
> > 'iceberg_cos_employee_test'
> > ...
> > IcebergTableLoadingException: Error loading metadata for Iceberg table
> > cosn://cdnlogtest-1252412955/impala_test_db/iceberg_cos_employee_test
> > ...
> > Loading file and block metadata for 1 paths for table ...: failed to load
> > 1 paths.
> >
>
> > ============================================================================================================
> >
> >
> > Best regards,
> > 原始邮件
> >
> >
> > 发件人：Zoltán Borók-Nagy <boroknagyz@apache.org&gt;
> > 发件时间：2025年12月23日 23:52
> > 收件人：dev <dev@impala.apache.org&gt;
> > 主题：Re: Advice and Considerations for Building an Impala
> > Compute-StorageSeparated Architecture
> >
> >
> >
> > Hi,
> >
> >
>
> > Just&nbsp;a&nbsp;few&nbsp;tips&nbsp;off&nbsp;the&nbsp;top&nbsp;of&nbsp;my&nbsp;head:
> >
>
> > &nbsp;-&nbsp;use&nbsp;dedicated&nbsp;coordinators&nbsp;and&nbsp;executors,&nbsp;rule&nbsp;of&nbsp;thumb&nbsp;for
> >
>
> > coordinator:executor&nbsp;ratio&nbsp;is&nbsp;1:50.&nbsp;Though&nbsp;for&nbsp;HA&nbsp;you&nbsp;probably&nbsp;want&nbsp;&gt;1
> > coordinators.
> >
>
> > &nbsp;-&nbsp;use&nbsp;local&nbsp;catalog&nbsp;mode&nbsp;(aka&nbsp;On-demand&nbsp;metadata):
> > https://impala.apache.org/docs/build/html/topics/impala_metadata.html
> >
>
> > &nbsp;-&nbsp;enabling&nbsp;remote&nbsp;data&nbsp;cache&nbsp;(with&nbsp;SSD&nbsp;disks)&nbsp;is&nbsp;essential&nbsp;in
> > compute-storage&nbsp;separated&nbsp;setup:
> > https://impala.apache.org/docs/build/html/topics/impala_data_cache.html
> >
> >
>
> > What&nbsp;table&nbsp;format&nbsp;/&nbsp;file&nbsp;format&nbsp;are&nbsp;you&nbsp;planning&nbsp;to&nbsp;use?
> >
>
> > If&nbsp;table&nbsp;format&nbsp;is&nbsp;Iceberg,&nbsp;make&nbsp;sure&nbsp;you&nbsp;use&nbsp;the&nbsp;latest&nbsp;Impala&nbsp;as&nbsp;we
> >
>
> > continuously&nbsp;improve&nbsp;Impala's&nbsp;performance&nbsp;on&nbsp;Iceberg.
> >
>
> > File&nbsp;format:&nbsp;Impala&nbsp;most&nbsp;efficiently&nbsp;works&nbsp;on&nbsp;Parquet&nbsp;files.
> >
> > Avoid&nbsp;small&nbsp;file issues:
> >
>
> > &nbsp;-&nbsp;choose&nbsp;proper&nbsp;partitioning&nbsp;for&nbsp;your&nbsp;data,&nbsp;i.e.&nbsp;avoid&nbsp;too&nbsp;coarse-grained
> >
>
> > and&nbsp;too&nbsp;fine-grained&nbsp;partitioning.&nbsp;I.e.&nbsp;you&nbsp;probably&nbsp;want&nbsp;more&nbsp;than&nbsp;200&nbsp;MB
> >
>
> > data&nbsp;per&nbsp;partition,&nbsp;but&nbsp;probably&nbsp;less&nbsp;than&nbsp;20&nbsp;GB.
> >
>
> > &nbsp;-&nbsp;compact&nbsp;your&nbsp;tables&nbsp;regularly,&nbsp;for&nbsp;Iceberg&nbsp;tables&nbsp;Impala&nbsp;has&nbsp;the
> > OPTIMIZE&nbsp;statement:
> > https://impala.apache.org/docs/build/html/topics/impala_iceberg.html
> >
> > I&nbsp;hope&nbsp;others&nbsp;chime&nbsp;in&nbsp;as&nbsp;well.
> >
> >
>
> > We&nbsp;would&nbsp;love&nbsp;to&nbsp;hear&nbsp;back&nbsp;about&nbsp;your&nbsp;experiences,&nbsp;and&nbsp;feel&nbsp;free&nbsp;to&nbsp;open
> >
>
> > tickets&nbsp;for&nbsp;Impala&nbsp;if&nbsp;you&nbsp;run&nbsp;into&nbsp;any&nbsp;issue:
> > https://issues.apache.org/jira/projects/IMPALA/issues
> >
> > Cheers,
> > &nbsp;&nbsp;&nbsp;&nbsp;Zoltan
> >
> >
>
> > On&nbsp;Tue,&nbsp;Dec&nbsp;23,&nbsp;2025&nbsp;at&nbsp;8:23 AM&nbsp;汲广熙&nbsp;<
> > 2870811759@qq.com&gt;&nbsp;wrote:
> >
> > &gt;&nbsp;Dear&nbsp;Impala&nbsp;Team,
> > &gt;
>
> > &gt;&nbsp;I&nbsp;hope&nbsp;this&nbsp;message&nbsp;finds&nbsp;you&nbsp;well.
> > &gt;
> >
>
> > &gt;&nbsp;I&nbsp;am&nbsp;currently&nbsp;planning&nbsp;to&nbsp;build&nbsp;a&nbsp;compute-storage&nbsp;separated&nbsp;architecture
> >
>
> > &gt;&nbsp;based&nbsp;on&nbsp;Apache&nbsp;Impala.&nbsp;In&nbsp;this&nbsp;setup:
> > &gt;
> > &gt;
> >
>
> > &gt;&nbsp;Compute&nbsp;layer:&nbsp;Apache&nbsp;Impala&nbsp;will&nbsp;be&nbsp;used&nbsp;for&nbsp;SQL&nbsp;query&nbsp;execution.
> > &gt;
> > &gt;
> > &gt;
> >
>
> > &gt;&nbsp;Storage&nbsp;layer:&nbsp;Tencent&nbsp;Cloud&nbsp;Object&nbsp;Storage&nbsp;(COS)&nbsp;will&nbsp;serve&nbsp;as&nbsp;the
> > &gt;&nbsp;backend&nbsp;storage.
> > &gt;
> > &gt;
> > &gt;
> >
>
> > &gt;&nbsp;Data&nbsp;ingestion:&nbsp;Kafka&nbsp;will&nbsp;be&nbsp;used&nbsp;to&nbsp;stream&nbsp;data&nbsp;into&nbsp;the&nbsp;system.
> > &gt;
> > &gt;
> > &gt;
> >
>
> > &gt;&nbsp;Monitoring&nbsp;&amp;amp;&nbsp;visualization:&nbsp;Grafana&nbsp;will&nbsp;be&nbsp;used&nbsp;to&nbsp;display
> > &gt;&nbsp;operational&nbsp;and&nbsp;performance&nbsp;metrics.
> > &gt;
> > &gt;
> >
>
> > &gt;&nbsp;Could&nbsp;you&nbsp;please&nbsp;provide&nbsp;some&nbsp;recommendations&nbsp;and&nbsp;key&nbsp;considerations&nbsp;for
> >
>
> > &gt;&nbsp;such&nbsp;an&nbsp;architecture?&nbsp;Specifically,&nbsp;I&nbsp;would&nbsp;appreciate&nbsp;guidance&nbsp;on:
> > &gt;
> > &gt;
> >
>
> > &gt;&nbsp;Best&nbsp;practices&nbsp;for&nbsp;integrating&nbsp;Impala&nbsp;with&nbsp;cloud&nbsp;object&nbsp;storage&nbsp;(COS).
> > &gt;
> > &gt;
> > &gt;
> >
>
> > &gt;&nbsp;Performance&nbsp;tuning&nbsp;tips&nbsp;for&nbsp;Impala&nbsp;in&nbsp;a&nbsp;disaggregated&nbsp;environment.
> > &gt;
> > &gt;
> > &gt;
> >
>
> > &gt;&nbsp;Any&nbsp;known&nbsp;limitations&nbsp;or&nbsp;compatibility&nbsp;issues&nbsp;when&nbsp;using&nbsp;COS&nbsp;as&nbsp;storage.
> > &gt;
> > &gt;
> > &gt;
> >
>
> > &gt;&nbsp;Recommended&nbsp;configurations&nbsp;for&nbsp;Kafka-to-Impala&nbsp;data&nbsp;pipelines.
> > &gt;
> > &gt;
> > &gt;
> >
>
> > &gt;&nbsp;Monitoring&nbsp;strategies&nbsp;for&nbsp;tracking&nbsp;query&nbsp;performance&nbsp;and&nbsp;resource&nbsp;usage&nbsp;in
> > &gt;&nbsp;Grafana.
> > &gt;
> > &gt;
> >
>
> > &gt;&nbsp;Thank&nbsp;you&nbsp;very&nbsp;much&nbsp;for&nbsp;your&nbsp;support&nbsp;and&nbsp;advice.&nbsp;I&nbsp;look&nbsp;forward&nbsp;to&nbsp;your
> > &gt;&nbsp;reply.
> > &gt;
> > &gt;&nbsp;Best&nbsp;regards
>
>
>
Re: Impala 4.5 Iceberg table on Tencent Cloud COS fails with "failedto load 1 paths" during metadata refresh.
Posted by Zoltán Borók-Nagy <bo...@apache.org>.
Thanks for your reply. Yeah, if Impala cannot load a table for some reason,
it cannot really do anything with it.
Please note that attachments don't get through Apache mailing lists.
Please copy-paste the stack trace of the exception from the CatalogD logs.
Or, feel free to file a Jira ticket about this issue.
Cheers,
Zoltan
On Mon, Jan 5, 2026 at 3:51 PM 汲广熙 <28...@qq.com> wrote:
> *Subject:* Re: [Bug?] Impala 4.5 Iceberg table loading failure on Tencent
> Cloud COS (cosn://)
>
> *Hi,*
>
> Thank you for the guidance. I have an important update regarding the
> backend flags you suggested.
>
> I applied --max_nonhdfs_partitions_parallel_load=1 and
> --max_hdfs_partitions_parallel_load=1 to my catalogd_flags and restarted
> the service. Unfortunately, the issue persists even with single-threaded
> loading.
>
> It seems that in Impala 4.5, ParallelFileMetadataLoader encounters
> difficulties when processing metadata returned by the cosn:// driver,
> regardless of the thread count. This suggests there might be a specific
> compatibility issue with how file/block metadata is interpreted for this
> storage scheme.
>
> Below are the raw logs from my session showing that *SHOW METADATA*,
> *INSERT*, and *SELECT* all trigger the same metadata loading error:
>
> ======================================
> =======================================================================================================
> [9.192.163.17:21050] db_on_cos_test> SHOW METADATA TABLES IN
> iceberg_cos_employee_test;
> Query: SHOW METADATA TABLES IN iceberg_cos_employee_test
> 2026-01-05 22:28:58 [Exception] ERROR: Query
> 9349f0ef1c788ec2:8548051e00000000 failed:
> AnalysisException: Could not load table
> db_on_cos_test.iceberg_cos_employee_test from catalog
> CAUSED BY: TableLoadingException: Could not load table
> db_on_cos_test.iceberg_cos_employee_test from catalog
> CAUSED BY: TException:
> TGetPartialCatalogObjectResponse(status:TStatus(status_code:GENERAL,
> error_msgs:[IcebergTableLoadingException: Error loading metadata for
> Iceberg table
> cosn://cdnlogtest-1252412955/impala_test_db/iceberg_cos_employee_test
> CAUSED BY: TableLoadingException: Loading file and block metadata for 1
> paths for table db_on_cos_test.iceberg_cos_employee_test: failed to load 1
> paths. Check the catalog server log for more details.]), lookup_status:OK)
>
> [9.192.163.17:21050] db_on_cos_test> INSERT INTO
> iceberg_cos_employee_test VALUES (1, '张三', '研发部', 15001.5, now());
> Query: INSERT INTO iceberg_cos_employee_test VALUES (1, '张三', '研发部',
> 15001.5, now())
> Query submitted at: 2026-01-05 22:29:08 (Coordinator:
> http://9.192.163.17:8080)
> 2026-01-05 22:29:08 [Exception] ERROR: Query
> ad4f18bf73bf1157:772c7c7600000000 failed:
> AnalysisException: org.apache.impala.catalog.TableLoadingException: Could
> not load table db_on_cos_test.iceberg_cos_employee_test from catalog
> CAUSED BY: TableLoadingException: Could not load table
> db_on_cos_test.iceberg_cos_employee_test from catalog
> CAUSED BY: TException:
> TGetPartialCatalogObjectResponse(status:TStatus(status_code:GENERAL,
> error_msgs:[IcebergTableLoadingException: Error loading metadata for
> Iceberg table
> cosn://cdnlogtest-1252412955/impala_test_db/iceberg_cos_employee_test
> CAUSED BY: TableLoadingException: Loading file and block metadata for 1
> paths for table db_on_cos_test.iceberg_cos_employee_test: failed to load 1
> paths. Check the catalog server log for more details.]), lookup_status:OK)
>
> [9.192.163.17:21050] db_on_cos_test> select * from
> iceberg_cos_employee_test;
> Query: select * from iceberg_cos_employee_test
> Query submitted at: 2026-01-05 22:30:42 (Coordinator:
> http://9.192.163.17:8080)
> 2026-01-05 22:30:42 [Exception] ERROR: Query
> 274fea8afd3f9dde:8d9847ab00000000 failed:
> AnalysisException: Failed to load metadata for table:
> 'iceberg_cos_employee_test'
> CAUSED BY: TableLoadingException: Could not load table
> db_on_cos_test.iceberg_cos_employee_test from catalog
> CAUSED BY: TException:
> TGetPartialCatalogObjectResponse(status:TStatus(status_code:GENERAL,
> error_msgs:[IcebergTableLoadingException: Error loading metadata for
> Iceberg table
> cosn://cdnlogtest-1252412955/impala_test_db/iceberg_cos_employee_test
> CAUSED BY: TableLoadingException: Loading file and block metadata for 1
> paths for table db_on_cos_test.iceberg_cos_employee_test: failed to load 1
> paths. Check the catalog server log for more details.]), lookup_status:OK)
> ======================================
> =======================================================================================================
>
> I have attached the full *catalogd logs* to this email for further
> investigation
>
>
> Best regards,
> 原始邮件
> ------------------------------
> 发件人：Zoltán Borók-Nagy <bo...@apache.org>
> 发件时间：2026年1月5日 21:10
> 收件人：dev <de...@impala.apache.org>
> 主题：Re: Impala 4.5 Iceberg table on Tencent Cloud COS fails with "failedto
> load 1 paths" during metadata refresh.
>
> Thanks for reaching out.
>
> Please file a ticket about the error with stack traces and all
> necessary information (snippets from cagalogd logs) to further debug this.
>
> > I tried to apply a workaround by setting metadata_loader_parallelism=1
> via ALTER TABLE
> We don't have such a table property. Was it suggested by Gemini?
>
> > Question: Is there a way to globally set metadata_loader_parallelism=1
> for the entire Catalogd, or perhaps a flag to force Impala to ignore block
> location errors for object storage during the initial load?
> I suspect a different issue than a concurrency bug, but FWIW Catalogd has
> backend flags "max_hdfs_partitions_parallel_load" and
> "max_nonhdfs_partitions_parallel_load" which you can set:
>
> https://github.com/apache/impala/blob/master/be/src/catalog/catalog.cc#L39-L45
> This affects file metadata loading of non-Iceberg tables as well, but it
> can be useful to validate the bug.
>
> We improved Iceberg table loading since 4.5, and fixed a few issues, so
> there's a chance that the issue you ran into is already fixed.
>
> Would it be possible for you to try out the current Impala master?
> I think it is time for us to do another upstream release, but it would be
> nice if we can make sure your use case will work well with the newest
> release.
>
> Cheers,
> Zoltan
>
>
> On Mon, Jan 5, 2026 at 1:35 PM 汲广熙 <28...@qq.com> wrote:
>
> > Subject: Re: [Bug?] Impala 4.5 Iceberg table loading failure on
> > Tencent Cloud COS (cosn://)
> >
> > Hi,
> >
> > Thank you for the suggestions. I have some critical updates regarding the
> > TableLoadingException: failed to load 1 paths issue on Tencent Cloud
> > COS (cosn://).
> >
> > I have already enabled Local Catalog
>
> > Mode (--use_local_catalog=true and minimal topic mode), but
> > the issue persists.
> >
>
> > Based on an analysis provided by Gemini (Google's AI model) regarding
> > the stack trace, it was suggested that the error in
>
> > ParallelFileMetadataLoader.loadInternal might be caused by concurrency
>
> > issues when handling object storage paths. Following this logic, I tried to
>
> > apply a workaround by setting metadata_loader_parallelism=1 via ALTER
> > TABLE.
> >
> > However, the command failed with the same AnalysisException and
> > TableLoadingException. It seems Impala falls into a deadlock state: I
>
> > cannot modify the table properties to fix the loading logic because Impala
>
> > fails to load the table metadata even during the ALTER TABLE analysis
> > phase.
> >
> > Observations:
> >
> >
> > The error originates from ParallelFileMetadataLoader.loadInternal.
> >
> >
> >
> > Even with Local Catalog enabled, the loader seems unable to handle the
> > missing block location info from the cosn:// driver.
> >
> >
> >
> > Standard Parquet tables work fine; this appears specific to the
> > Iceberg metadata loading path in Impala 4.5.
> >
> >
> > Question: Is there a way to globally set
> > metadata_loader_parallelism=1 for the entire Catalogd, or perhaps a
> > flag to force Impala to ignore block location errors for object storage
> > during the initial load?
> >
> > I am happy to provide a full stack trace or open a JIRA ticket, as this
> > currently makes Iceberg on COS unusable in our environment.
> >
> >
> >
> >
> >
>
> > ============================================================================================================
> > When I run a simple SELECT *or INSERT INTOquery on this table, I get the
> > following error:
> > AnalysisException: Failed to load metadata for table:
> > 'iceberg_cos_employee_test'
> > ...
> > IcebergTableLoadingException: Error loading metadata for Iceberg table
> > cosn://cdnlogtest-1252412955/impala_test_db/iceberg_cos_employee_test
> > ...
> > Loading file and block metadata for 1 paths for table ...: failed to load
> > 1 paths.
> >
>
> > ============================================================================================================
> >
> >
> > Best regards,
> > 原始邮件
> >
> >
> > 发件人：Zoltán Borók-Nagy <boroknagyz@apache.org>
> > 发件时间：2025年12月23日 23:52
> > 收件人：dev <dev@impala.apache.org>
> > 主题：Re: Advice and Considerations for Building an Impala
> > Compute-StorageSeparated Architecture
> >
> >
> >
> > Hi,
> >
> >
>
> > Just a few tips off the top of my head:
> >
>
> > - use dedicated coordinators and executors, rule of thumb for
> >
>
> > coordinator:executor ratio is 1:50. Though for HA you probably want >1
> > coordinators.
> >
>
> > - use local catalog mode (aka On-demand metadata):
> > https://impala.apache.org/docs/build/html/topics/impala_metadata.html
> >
>
> > - enabling remote data cache (with SSD disks) is essential in
> > compute-storage separated setup:
> > https://impala.apache.org/docs/build/html/topics/impala_data_cache.html
> >
> >
>
> > What table format / file format are you planning to use?
> >
>
> > If table format is Iceberg, make sure you use the latest Impala as we
> >
>
> > continuously improve Impala's performance on Iceberg.
> >
>
> > File format: Impala most efficiently works on Parquet files.
> >
> > Avoid small file issues:
> >
>
> > - choose proper partitioning for your data, i.e. avoid too coarse-grained
> >
>
> > and too fine-grained partitioning. I.e. you probably want more than 200 MB
> >
>
> > data per partition, but probably less than 20 GB.
> >
>
> > - compact your tables regularly, for Iceberg tables Impala has the
> > OPTIMIZE statement:
> > https://impala.apache.org/docs/build/html/topics/impala_iceberg.html
> >
> > I hope others chime in as well.
> >
> >
>
> > We would love to hear back about your experiences, and feel free to open
> >
>
> > tickets for Impala if you run into any issue:
> > https://issues.apache.org/jira/projects/IMPALA/issues
> >
> > Cheers,
> > Zoltan
> >
> >
>
> > On Tue, Dec 23, 2025 at 8:23 AM 汲广熙 <
> > 2870811759@qq.com> wrote:
> >
> > > Dear Impala Team,
> > >
>
> > > I hope this message finds you well.
> > >
> >
>
> > > I am currently planning to build a compute-storage separated architecture
> >
>
> > > based on Apache Impala. In this setup:
> > >
> > >
> >
>
> > > Compute layer: Apache Impala will be used for SQL query execution.
> > >
> > >
> > >
> >
>
> > > Storage layer: Tencent Cloud Object Storage (COS) will serve as the
> > > backend storage.
> > >
> > >
> > >
> >
>
> > > Data ingestion: Kafka will be used to stream data into the system.
> > >
> > >
> > >
> >
>
> > > Monitoring &amp; visualization: Grafana will be used to display
> > > operational and performance metrics.
> > >
> > >
> >
>
> > > Could you please provide some recommendations and key considerations for
> >
>
> > > such an architecture? Specifically, I would appreciate guidance on:
> > >
> > >
> >
>
> > > Best practices for integrating Impala with cloud object storage (COS).
> > >
> > >
> > >
> >
>
> > > Performance tuning tips for Impala in a disaggregated environment.
> > >
> > >
> > >
> >
>
> > > Any known limitations or compatibility issues when using COS as storage.
> > >
> > >
> > >
> >
>
> > > Recommended configurations for Kafka-to-Impala data pipelines.
> > >
> > >
> > >
> >
>
> > > Monitoring strategies for tracking query performance and resource usage in
> > > Grafana.
> > >
> > >
> >
>
> > > Thank you very much for your support and advice. I look forward to your
> > > reply.
> > >
> > > Best regards
>
>
>
Re: Impala 4.5 Iceberg table on Tencent Cloud COS fails with "failedto load 1 paths" during metadata refresh.
Posted by 汲广熙 <28...@qq.com>.
Subject: Re: [Bug?] Impala 4.5 Iceberg table loading failure on Tencent Cloud COS (cosn://)
Hi,
Thank you for the guidance. I have an important update regarding the backend flags you suggested.
I applied --max_nonhdfs_partitions_parallel_load=1 and --max_hdfs_partitions_parallel_load=1 to my catalogd_flags and restarted the service. Unfortunately, the issue persists even with single-threaded loading.
It seems that in Impala 4.5, ParallelFileMetadataLoader encounters difficulties when processing metadata returned by the cosn:// driver, regardless of the thread count. This suggests there might be a specific compatibility issue with how file/block metadata is interpreted for this storage scheme.
Below are the raw logs from my session showing that SHOW METADATA, INSERT, and SELECT all trigger the same metadata loading error:
=============================================================================================================================================
[9.192.163.17:21050] db_on_cos_test> SHOW METADATA TABLES IN iceberg_cos_employee_test;
Query: SHOW METADATA TABLES IN iceberg_cos_employee_test
2026-01-05 22:28:58 [Exception] ERROR: Query 9349f0ef1c788ec2:8548051e00000000 failed:
AnalysisException: Could not load table db_on_cos_test.iceberg_cos_employee_test from catalog
CAUSED BY: TableLoadingException: Could not load table db_on_cos_test.iceberg_cos_employee_test from catalog
CAUSED BY: TException: TGetPartialCatalogObjectResponse(status:TStatus(status_code:GENERAL, error_msgs:[IcebergTableLoadingException: Error loading metadata for Iceberg table cosn://cdnlogtest-1252412955/impala_test_db/iceberg_cos_employee_test
CAUSED BY: TableLoadingException: Loading file and block metadata for 1 paths for table db_on_cos_test.iceberg_cos_employee_test: failed to load 1 paths. Check the catalog server log for more details.]), lookup_status:OK)
[9.192.163.17:21050] db_on_cos_test> INSERT INTO iceberg_cos_employee_test VALUES (1, '张三', '研发部', 15001.5, now());
Query: INSERT INTO iceberg_cos_employee_test VALUES (1, '张三', '研发部', 15001.5, now())
Query submitted at: 2026-01-05 22:29:08 (Coordinator: http://9.192.163.17:8080)
2026-01-05 22:29:08 [Exception] ERROR: Query ad4f18bf73bf1157:772c7c7600000000 failed:
AnalysisException: org.apache.impala.catalog.TableLoadingException: Could not load table db_on_cos_test.iceberg_cos_employee_test from catalog
CAUSED BY: TableLoadingException: Could not load table db_on_cos_test.iceberg_cos_employee_test from catalog
CAUSED BY: TException: TGetPartialCatalogObjectResponse(status:TStatus(status_code:GENERAL, error_msgs:[IcebergTableLoadingException: Error loading metadata for Iceberg table cosn://cdnlogtest-1252412955/impala_test_db/iceberg_cos_employee_test
CAUSED BY: TableLoadingException: Loading file and block metadata for 1 paths for table db_on_cos_test.iceberg_cos_employee_test: failed to load 1 paths. Check the catalog server log for more details.]), lookup_status:OK)
[9.192.163.17:21050] db_on_cos_test> select * from iceberg_cos_employee_test;
Query: select * from iceberg_cos_employee_test
Query submitted at: 2026-01-05 22:30:42 (Coordinator: http://9.192.163.17:8080)
2026-01-05 22:30:42 [Exception] ERROR: Query 274fea8afd3f9dde:8d9847ab00000000 failed:
AnalysisException: Failed to load metadata for table: 'iceberg_cos_employee_test'
CAUSED BY: TableLoadingException: Could not load table db_on_cos_test.iceberg_cos_employee_test from catalog
CAUSED BY: TException: TGetPartialCatalogObjectResponse(status:TStatus(status_code:GENERAL, error_msgs:[IcebergTableLoadingException: Error loading metadata for Iceberg table cosn://cdnlogtest-1252412955/impala_test_db/iceberg_cos_employee_test
CAUSED BY: TableLoadingException: Loading file and block metadata for 1 paths for table db_on_cos_test.iceberg_cos_employee_test: failed to load 1 paths. Check the catalog server log for more details.]), lookup_status:OK)
=============================================================================================================================================
I have attached the full catalogd logs to this email for further investigation
Best regards,
原始邮件
发件人：Zoltán Borók-Nagy <boroknagyz@apache.org>
发件时间：2026年1月5日 21:10
收件人：dev <dev@impala.apache.org>
主题：Re: Impala 4.5 Iceberg table on Tencent Cloud COS fails with "failedto load 1 paths" during metadata refresh.
Thanks for reaching out.
Please file a ticket about the error with stack traces and all
necessary information (snippets from cagalogd logs) to further debug this.
> I tried to apply a workaround by setting metadata_loader_parallelism=1
via ALTER TABLE
We don't have such a table property. Was it suggested by Gemini?
> Question: Is there a way to globally set metadata_loader_parallelism=1
for the entire Catalogd, or perhaps a flag to force Impala to ignore block
location errors for object storage during the initial load?
I suspect a different issue than a concurrency bug, but FWIW Catalogd has
backend flags "max_hdfs_partitions_parallel_load" and
"max_nonhdfs_partitions_parallel_load" which you can set:
https://github.com/apache/impala/blob/master/be/src/catalog/catalog.cc#L39-L45
This affects file metadata loading of non-Iceberg tables as well, but it
can be useful to validate the bug.
We improved Iceberg table loading since 4.5, and fixed a few issues, so
there's a chance that the issue you ran into is already fixed.
Would it be possible for you to try out the current Impala master?
I think it is time for us to do another upstream release, but it would be
nice if we can make sure your use case will work well with the newest
release.
Cheers,
Zoltan
On Mon, Jan 5, 2026 at 1:35 PM 汲广熙 <2870811759@qq.com> wrote:
> Subject:&nbsp;Re: [Bug?] Impala 4.5 Iceberg table loading failure on
> Tencent Cloud COS (cosn://)
>
> Hi,
>
> Thank you for the suggestions. I have some critical updates regarding the
> TableLoadingException: failed to load 1 paths&nbsp;issue on Tencent Cloud
> COS (cosn://).
>
> I have already enabled Local Catalog
> Mode&nbsp;(--use_local_catalog=true&nbsp;and minimal&nbsp;topic mode), but
> the issue persists.
>
> Based on an analysis provided by Gemini (Google's AI model)&nbsp;regarding
> the stack trace, it was suggested that the error in
> ParallelFileMetadataLoader.loadInternal&nbsp;might be caused by concurrency
> issues when handling object storage paths. Following this logic, I tried to
> apply a workaround by setting metadata_loader_parallelism=1&nbsp;via ALTER
> TABLE.
>
> However, the command failed with the same AnalysisException&nbsp;and
> TableLoadingException. It seems Impala falls into a deadlock state: I
> cannot modify the table properties to fix the loading logic because Impala
> fails to load the table metadata even during the ALTER TABLE&nbsp;analysis
> phase.
>
> Observations:
>
>
> The error originates from ParallelFileMetadataLoader.loadInternal.
>
>
>
> Even with Local Catalog enabled, the loader seems unable to handle the
> missing block location info from the cosn://&nbsp;driver.
>
>
>
> Standard Parquet tables work fine; this appears specific to the
> Iceberg&nbsp;metadata loading path in Impala 4.5.
>
>
> Question: Is there a way to globally set
> metadata_loader_parallelism=1&nbsp;for the entire Catalogd, or perhaps a
> flag to force Impala to ignore block location errors for object storage
> during the initial load?
>
> I am happy to provide a full stack trace or open a JIRA ticket, as this
> currently makes Iceberg on COS unusable in our environment.
>
>
>
>
>
> ============================================================================================================
> When I run a simple SELECT *or INSERT INTOquery on this table, I get the
> following error:
> AnalysisException: Failed to load metadata for table:
> 'iceberg_cos_employee_test'
> ...
> IcebergTableLoadingException: Error loading metadata for Iceberg table
> cosn://cdnlogtest-1252412955/impala_test_db/iceberg_cos_employee_test
> ...
> Loading file and block metadata for 1 paths for table ...: failed to load
> 1 paths.
>
> ============================================================================================================
>
>
> Best regards,
> 原始邮件
>
>
> 发件人：Zoltán Borók-Nagy <boroknagyz@apache.org&gt;
> 发件时间：2025年12月23日 23:52
> 收件人：dev <dev@impala.apache.org&gt;
> 主题：Re: Advice and Considerations for Building an Impala
> Compute-StorageSeparated Architecture
>
>
>
> Hi,
>
>
> Just&nbsp;a&nbsp;few&nbsp;tips&nbsp;off&nbsp;the&nbsp;top&nbsp;of&nbsp;my&nbsp;head:
>
> &nbsp;-&nbsp;use&nbsp;dedicated&nbsp;coordinators&nbsp;and&nbsp;executors,&nbsp;rule&nbsp;of&nbsp;thumb&nbsp;for
>
> coordinator:executor&nbsp;ratio&nbsp;is&nbsp;1:50.&nbsp;Though&nbsp;for&nbsp;HA&nbsp;you&nbsp;probably&nbsp;want&nbsp;&gt;1
> coordinators.
>
> &nbsp;-&nbsp;use&nbsp;local&nbsp;catalog&nbsp;mode&nbsp;(aka&nbsp;On-demand&nbsp;metadata):
> https://impala.apache.org/docs/build/html/topics/impala_metadata.html
>
> &nbsp;-&nbsp;enabling&nbsp;remote&nbsp;data&nbsp;cache&nbsp;(with&nbsp;SSD&nbsp;disks)&nbsp;is&nbsp;essential&nbsp;in
> compute-storage&nbsp;separated&nbsp;setup:
> https://impala.apache.org/docs/build/html/topics/impala_data_cache.html
>
>
> What&nbsp;table&nbsp;format&nbsp;/&nbsp;file&nbsp;format&nbsp;are&nbsp;you&nbsp;planning&nbsp;to&nbsp;use?
>
> If&nbsp;table&nbsp;format&nbsp;is&nbsp;Iceberg,&nbsp;make&nbsp;sure&nbsp;you&nbsp;use&nbsp;the&nbsp;latest&nbsp;Impala&nbsp;as&nbsp;we
>
> continuously&nbsp;improve&nbsp;Impala's&nbsp;performance&nbsp;on&nbsp;Iceberg.
>
> File&nbsp;format:&nbsp;Impala&nbsp;most&nbsp;efficiently&nbsp;works&nbsp;on&nbsp;Parquet&nbsp;files.
>
> Avoid&nbsp;small&nbsp;file issues:
>
> &nbsp;-&nbsp;choose&nbsp;proper&nbsp;partitioning&nbsp;for&nbsp;your&nbsp;data,&nbsp;i.e.&nbsp;avoid&nbsp;too&nbsp;coarse-grained
>
> and&nbsp;too&nbsp;fine-grained&nbsp;partitioning.&nbsp;I.e.&nbsp;you&nbsp;probably&nbsp;want&nbsp;more&nbsp;than&nbsp;200&nbsp;MB
>
> data&nbsp;per&nbsp;partition,&nbsp;but&nbsp;probably&nbsp;less&nbsp;than&nbsp;20&nbsp;GB.
>
> &nbsp;-&nbsp;compact&nbsp;your&nbsp;tables&nbsp;regularly,&nbsp;for&nbsp;Iceberg&nbsp;tables&nbsp;Impala&nbsp;has&nbsp;the
> OPTIMIZE&nbsp;statement:
> https://impala.apache.org/docs/build/html/topics/impala_iceberg.html
>
> I&nbsp;hope&nbsp;others&nbsp;chime&nbsp;in&nbsp;as&nbsp;well.
>
>
> We&nbsp;would&nbsp;love&nbsp;to&nbsp;hear&nbsp;back&nbsp;about&nbsp;your&nbsp;experiences,&nbsp;and&nbsp;feel&nbsp;free&nbsp;to&nbsp;open
>
> tickets&nbsp;for&nbsp;Impala&nbsp;if&nbsp;you&nbsp;run&nbsp;into&nbsp;any&nbsp;issue:
> https://issues.apache.org/jira/projects/IMPALA/issues
>
> Cheers,
> &nbsp;&nbsp;&nbsp;&nbsp;Zoltan
>
>
> On&nbsp;Tue,&nbsp;Dec&nbsp;23,&nbsp;2025&nbsp;at&nbsp;8:23 AM&nbsp;汲广熙&nbsp;<
> 2870811759@qq.com&gt;&nbsp;wrote:
>
> &gt;&nbsp;Dear&nbsp;Impala&nbsp;Team,
> &gt;
> &gt;&nbsp;I&nbsp;hope&nbsp;this&nbsp;message&nbsp;finds&nbsp;you&nbsp;well.
> &gt;
>
> &gt;&nbsp;I&nbsp;am&nbsp;currently&nbsp;planning&nbsp;to&nbsp;build&nbsp;a&nbsp;compute-storage&nbsp;separated&nbsp;architecture
>
> &gt;&nbsp;based&nbsp;on&nbsp;Apache&nbsp;Impala.&nbsp;In&nbsp;this&nbsp;setup:
> &gt;
> &gt;
>
> &gt;&nbsp;Compute&nbsp;layer:&nbsp;Apache&nbsp;Impala&nbsp;will&nbsp;be&nbsp;used&nbsp;for&nbsp;SQL&nbsp;query&nbsp;execution.
> &gt;
> &gt;
> &gt;
>
> &gt;&nbsp;Storage&nbsp;layer:&nbsp;Tencent&nbsp;Cloud&nbsp;Object&nbsp;Storage&nbsp;(COS)&nbsp;will&nbsp;serve&nbsp;as&nbsp;the
> &gt;&nbsp;backend&nbsp;storage.
> &gt;
> &gt;
> &gt;
>
> &gt;&nbsp;Data&nbsp;ingestion:&nbsp;Kafka&nbsp;will&nbsp;be&nbsp;used&nbsp;to&nbsp;stream&nbsp;data&nbsp;into&nbsp;the&nbsp;system.
> &gt;
> &gt;
> &gt;
>
> &gt;&nbsp;Monitoring&nbsp;&amp;amp;&nbsp;visualization:&nbsp;Grafana&nbsp;will&nbsp;be&nbsp;used&nbsp;to&nbsp;display
> &gt;&nbsp;operational&nbsp;and&nbsp;performance&nbsp;metrics.
> &gt;
> &gt;
>
> &gt;&nbsp;Could&nbsp;you&nbsp;please&nbsp;provide&nbsp;some&nbsp;recommendations&nbsp;and&nbsp;key&nbsp;considerations&nbsp;for
>
> &gt;&nbsp;such&nbsp;an&nbsp;architecture?&nbsp;Specifically,&nbsp;I&nbsp;would&nbsp;appreciate&nbsp;guidance&nbsp;on:
> &gt;
> &gt;
>
> &gt;&nbsp;Best&nbsp;practices&nbsp;for&nbsp;integrating&nbsp;Impala&nbsp;with&nbsp;cloud&nbsp;object&nbsp;storage&nbsp;(COS).
> &gt;
> &gt;
> &gt;
>
> &gt;&nbsp;Performance&nbsp;tuning&nbsp;tips&nbsp;for&nbsp;Impala&nbsp;in&nbsp;a&nbsp;disaggregated&nbsp;environment.
> &gt;
> &gt;
> &gt;
>
> &gt;&nbsp;Any&nbsp;known&nbsp;limitations&nbsp;or&nbsp;compatibility&nbsp;issues&nbsp;when&nbsp;using&nbsp;COS&nbsp;as&nbsp;storage.
> &gt;
> &gt;
> &gt;
>
> &gt;&nbsp;Recommended&nbsp;configurations&nbsp;for&nbsp;Kafka-to-Impala&nbsp;data&nbsp;pipelines.
> &gt;
> &gt;
> &gt;
>
> &gt;&nbsp;Monitoring&nbsp;strategies&nbsp;for&nbsp;tracking&nbsp;query&nbsp;performance&nbsp;and&nbsp;resource&nbsp;usage&nbsp;in
> &gt;&nbsp;Grafana.
> &gt;
> &gt;
>
> &gt;&nbsp;Thank&nbsp;you&nbsp;very&nbsp;much&nbsp;for&nbsp;your&nbsp;support&nbsp;and&nbsp;advice.&nbsp;I&nbsp;look&nbsp;forward&nbsp;to&nbsp;your
> &gt;&nbsp;reply.
> &gt;
> &gt;&nbsp;Best&nbsp;regards
Re: Impala 4.5 Iceberg table on Tencent Cloud COS fails with "failed to load 1 paths" during metadata refresh.
Posted by Zoltán Borók-Nagy <bo...@apache.org>.
Thanks for reaching out.
Please file a ticket about the error with stack traces and all
necessary information (snippets from cagalogd logs) to further debug this.
> I tried to apply a workaround by setting metadata_loader_parallelism=1
via ALTER TABLE
We don't have such a table property. Was it suggested by Gemini?
> Question: Is there a way to globally set metadata_loader_parallelism=1
for the entire Catalogd, or perhaps a flag to force Impala to ignore block
location errors for object storage during the initial load?
I suspect a different issue than a concurrency bug, but FWIW Catalogd has
backend flags "max_hdfs_partitions_parallel_load" and
"max_nonhdfs_partitions_parallel_load" which you can set:
https://github.com/apache/impala/blob/master/be/src/catalog/catalog.cc#L39-L45
This affects file metadata loading of non-Iceberg tables as well, but it
can be useful to validate the bug.
We improved Iceberg table loading since 4.5, and fixed a few issues, so
there's a chance that the issue you ran into is already fixed.
Would it be possible for you to try out the current Impala master?
I think it is time for us to do another upstream release, but it would be
nice if we can make sure your use case will work well with the newest
release.
Cheers,
Zoltan
On Mon, Jan 5, 2026 at 1:35 PM 汲广熙 <28...@qq.com> wrote:
> Subject: Re: [Bug?] Impala 4.5 Iceberg table loading failure on
> Tencent Cloud COS (cosn://)
>
> Hi,
>
> Thank you for the suggestions. I have some critical updates regarding the
> TableLoadingException: failed to load 1 paths issue on Tencent Cloud
> COS (cosn://).
>
> I have already enabled Local Catalog
> Mode (--use_local_catalog=true and minimal topic mode), but
> the issue persists.
>
> Based on an analysis provided by Gemini (Google's AI model) regarding
> the stack trace, it was suggested that the error in
> ParallelFileMetadataLoader.loadInternal might be caused by concurrency
> issues when handling object storage paths. Following this logic, I tried to
> apply a workaround by setting metadata_loader_parallelism=1 via ALTER
> TABLE.
>
> However, the command failed with the same AnalysisException and
> TableLoadingException. It seems Impala falls into a deadlock state: I
> cannot modify the table properties to fix the loading logic because Impala
> fails to load the table metadata even during the ALTER TABLE analysis
> phase.
>
> Observations:
>
>
> The error originates from ParallelFileMetadataLoader.loadInternal.
>
>
>
> Even with Local Catalog enabled, the loader seems unable to handle the
> missing block location info from the cosn:// driver.
>
>
>
> Standard Parquet tables work fine; this appears specific to the
> Iceberg metadata loading path in Impala 4.5.
>
>
> Question: Is there a way to globally set
> metadata_loader_parallelism=1 for the entire Catalogd, or perhaps a
> flag to force Impala to ignore block location errors for object storage
> during the initial load?
>
> I am happy to provide a full stack trace or open a JIRA ticket, as this
> currently makes Iceberg on COS unusable in our environment.
>
>
>
>
>
> ============================================================================================================
> When I run a simple SELECT *or INSERT INTOquery on this table, I get the
> following error:
> AnalysisException: Failed to load metadata for table:
> 'iceberg_cos_employee_test'
> ...
> IcebergTableLoadingException: Error loading metadata for Iceberg table
> cosn://cdnlogtest-1252412955/impala_test_db/iceberg_cos_employee_test
> ...
> Loading file and block metadata for 1 paths for table ...: failed to load
> 1 paths.
>
> ============================================================================================================
>
>
> Best regards,
> 原始邮件
>
>
> 发件人：Zoltán Borók-Nagy <boroknagyz@apache.org>
> 发件时间：2025年12月23日 23:52
> 收件人：dev <dev@impala.apache.org>
> 主题：Re: Advice and Considerations for Building an Impala
> Compute-StorageSeparated Architecture
>
>
>
> Hi,
>
>
> Just a few tips off the top of my head:
>
> - use dedicated coordinators and executors, rule of thumb for
>
> coordinator:executor ratio is 1:50. Though for HA you probably want >1
> coordinators.
>
> - use local catalog mode (aka On-demand metadata):
> https://impala.apache.org/docs/build/html/topics/impala_metadata.html
>
> - enabling remote data cache (with SSD disks) is essential in
> compute-storage separated setup:
> https://impala.apache.org/docs/build/html/topics/impala_data_cache.html
>
>
> What table format / file format are you planning to use?
>
> If table format is Iceberg, make sure you use the latest Impala as we
>
> continuously improve Impala's performance on Iceberg.
>
> File format: Impala most efficiently works on Parquet files.
>
> Avoid small file issues:
>
> - choose proper partitioning for your data, i.e. avoid too coarse-grained
>
> and too fine-grained partitioning. I.e. you probably want more than 200 MB
>
> data per partition, but probably less than 20 GB.
>
> - compact your tables regularly, for Iceberg tables Impala has the
> OPTIMIZE statement:
> https://impala.apache.org/docs/build/html/topics/impala_iceberg.html
>
> I hope others chime in as well.
>
>
> We would love to hear back about your experiences, and feel free to open
>
> tickets for Impala if you run into any issue:
> https://issues.apache.org/jira/projects/IMPALA/issues
>
> Cheers,
> Zoltan
>
>
> On Tue, Dec 23, 2025 at 8:23 AM 汲广熙 <
> 2870811759@qq.com> wrote:
>
> > Dear Impala Team,
> >
> > I hope this message finds you well.
> >
>
> > I am currently planning to build a compute-storage separated architecture
>
> > based on Apache Impala. In this setup:
> >
> >
>
> > Compute layer: Apache Impala will be used for SQL query execution.
> >
> >
> >
>
> > Storage layer: Tencent Cloud Object Storage (COS) will serve as the
> > backend storage.
> >
> >
> >
>
> > Data ingestion: Kafka will be used to stream data into the system.
> >
> >
> >
>
> > Monitoring &amp; visualization: Grafana will be used to display
> > operational and performance metrics.
> >
> >
>
> > Could you please provide some recommendations and key considerations for
>
> > such an architecture? Specifically, I would appreciate guidance on:
> >
> >
>
> > Best practices for integrating Impala with cloud object storage (COS).
> >
> >
> >
>
> > Performance tuning tips for Impala in a disaggregated environment.
> >
> >
> >
>
> > Any known limitations or compatibility issues when using COS as storage.
> >
> >
> >
>
> > Recommended configurations for Kafka-to-Impala data pipelines.
> >
> >
> >
>
> > Monitoring strategies for tracking query performance and resource usage in
> > Grafana.
> >
> >
>
> > Thank you very much for your support and advice. I look forward to your
> > reply.
> >
> > Best regards