You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Danny Chen (Jira)" <ji...@apache.org> on 2023/11/15 08:11:00 UTC
[jira] [Closed] (HUDI-7069) Optimize metaclient construction and include table config in write config for multi-table services.
[ https://issues.apache.org/jira/browse/HUDI-7069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Danny Chen closed HUDI-7069.
----------------------------
Resolution: Fixed
Fixed via master branch: dcd5a8182a11faab8bfc1ce8aa7787fa590dd395
> Optimize metaclient construction and include table config in write config for multi-table services.
> ---------------------------------------------------------------------------------------------------
>
> Key: HUDI-7069
> URL: https://issues.apache.org/jira/browse/HUDI-7069
> Project: Apache Hudi
> Issue Type: Bug
> Reporter: Ma Jian
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.0.0
>
>
> In the current implementation of run multi tables services, the clustering task and compaction task both build metaclient repeatedly for each table, causing additional overhead. To reduce this overhead, we extract the construction of metaclient and only construct it once for each table, passing it as a parameter to the corresponding task.
> At the same time, when running multi tables services, the write config lacks some information from the table config, such as the table name. This leads to empty strings when retrieving the table name in certain situations. For example, when configuring the prefix for metrics, if not specified, the table name is used as the prefix. However, in the current situation, without the table config, it's impossible to differentiate the metrics of different tables, resulting in an empty prefix. By adding the table config to the write config beforehand, we can obtain all the configuration information in the subsequent write config step.
> Additionally, we made a small modification by removing the redundant construction of metaclient in the clusteringJob's constructor.
> !https://intranetproxy.alipay.com/skylark/lark/0/2023/png/62256341/1699595853317-03416d85-8a25-4fbd-96af-f351f0ac6ec7.png?x-oss-process=image%2Fresize%2Cw_1322%2Climit_0|width=470,height=355!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)