You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by st...@apache.org on 2026/05/10 08:01:48 UTC
(impala) 03/03: IMPALA-14899: [DOCS] Document max_stmt_metadata_loader_threads for parallel metadata loading
This is an automated email from the ASF dual-hosted git repository.
asf-gitbox-commits pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit 739f9cd6a3939a3d2eaede88e6901429a2b0201c
Author: m-sanjana19 <ma...@gmail.com>
AuthorDate: Sat May 2 14:08:54 2026 +0530
IMPALA-14899: [DOCS] Document max_stmt_metadata_loader_threads for parallel metadata loading
This explains how to configure the thread pool size for loading
statement metadata to improve startup performance on queries
accessing multiple tables for the first time,
as tracked in IMPALA-14447.
Change-Id: Ic347c3a4628ab4d0c22be689d3a8ff3dfdd22adf
Reviewed-on: http://gerrit.cloudera.org:8080/24262
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Quanlong Huang <hu...@gmail.com>
---
docs/topics/impala_metadata.xml | 28 +++++++++++++++++++---------
1 file changed, 19 insertions(+), 9 deletions(-)
diff --git a/docs/topics/impala_metadata.xml b/docs/topics/impala_metadata.xml
index 9ee72db7d..e967257fc 100644
--- a/docs/topics/impala_metadata.xml
+++ b/docs/topics/impala_metadata.xml
@@ -122,14 +122,14 @@ under the License.
on-demand:
<codeblock>--use_local_catalog=true </codeblock>
</dd>
-
</dlentry>
<dlentry>
<dt>Flags related to <codeph>use_local_catalog</codeph></dt>
<dd>When <codeph>use_local_catalog</codeph> is enabled or set to <codeph>True</codeph>
on the impalad coordinators the following list of flags configure various parameters
- as described below. It is not recommended to change the default values on these
- flags.</dd>
+ as described below. It is recommended to maintain the default values for these flags;
+ however, you can modify the <codeph>max_stmt_metadata_loader_threads</codeph> flag as
+ needed for specific environment requirements. </dd>
<dd>
<ul>
<li>The flag <codeph>local_catalog_cache_mb</codeph> (defaults to -1) configures the
@@ -137,19 +137,29 @@ under the License.
cache is auto-configured to 60% of the configured Java heap size. Note that the
Java heap size is distinct from and typically smaller than the overall Impala
memory limit.</li>
- <li>The flag <codeph>local_catalog_cache_expiration_s</codeph> (defaults to 3600) configures the
- expiration time of the catalog cache within each impalad. Even if the configured
- cache capacity has not been reached, items are removed from the cache if they have not
- been accessed in the defined amount of time.</li>
+ <li>The flag <codeph>local_catalog_cache_expiration_s</codeph> (defaults to 3600)
+ configures the expiration time of the catalog cache within each impalad. Even if
+ the configured cache capacity has not been reached, items are removed from the
+ cache if they have not been accessed in the defined amount of time.</li>
<li>The flag <codeph>local_catalog_max_fetch_retries</codeph> (defaults to 40) configures
the maximum number of retries needed for queries to fetch a metadata object from the impalad
- coordinator's local catalog cache.</li>
+ coordinator's local catalog cache.
+ </li>
+ <li id="IMPALA-14447">The flag <codeph>max_stmt_metadata_loader_threads</codeph>
+ (defaults to 8) controls the maximum number of threads used per-query to load
+ table metadata during query compilation. In local catalog mode, you can
+ parallelize this process to reduce waiting time due to sequential loading. This
+ flag allows you to parallelize that process, improving performance for complex
+ queries that involve many tables. If a query requires metadata for only one table,
+ or if you set this flag to 1, coordinator loads the metadata sequentially. Impala also
+ falls back to sequential loading if the system cannot successfully start the
+ requested number of threads.
+ </li>
</ul>
</dd>
</dlentry>
</dl>
</p>
-
</conbody>
</concept>