You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kylin.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2023/07/20 08:42:00 UTC
[jira] [Commented] (KYLIN-5650) In the cloud environment, there is a probability that the dictionary metadata file will be read abnormally during building job, resulting in incorrect query results.
[ https://issues.apache.org/jira/browse/KYLIN-5650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17744965#comment-17744965 ]
ASF subversion and git services commented on KYLIN-5650:
--------------------------------------------------------
Commit a7531287d6d3733bcadec935f5f2225b413a9e18 in kylin's branch refs/heads/kylin5_beta from huangsheng
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=a7531287d6 ]
KYLIN-5650 fix Building global dict can't read meta file on S3
> In the cloud environment, there is a probability that the dictionary metadata file will be read abnormally during building job, resulting in incorrect query results.
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: KYLIN-5650
> URL: https://issues.apache.org/jira/browse/KYLIN-5650
> Project: Kylin
> Issue Type: Bug
> Components: Tools, Build and Test
> Affects Versions: 5.0-alpha
> Reporter: Zhiting Guo
> Assignee: Zhiting Guo
> Priority: Major
> Fix For: 5.0-beta
>
> Attachments: In the cloud environment, there is a probability that the dictionary metadata file will be read abnormally during building job, resulting in incorrect query results..pdf
>
>
> Checked the dictionary, there are no duplicate values. Checked the execution plan of the build dictionary step, there is no problem. Checked the steps of building a flat table and found that there was a problem in the step of flat table encoding dictionary.
> The reason for the error is that the encoding is not performed after repartition according to the dictionary column. As shown in the figure, there is no repartition, and the encode column appears in the plan.
> There are also the following logs:
> {code:java}
> 2023-03-26T20:26:30,868 INFO [logger-thread-0] dict.NGlobalDictHDFSStore : Commit from s3a://datalake-kc-s3-prd-bj/kylin/kcprodYcHG_kylin/datalake_kylin/dict/global_dict/GDT.GDT_CMPLYA_FCT_DIST_RESLT/IS_STAT/working to s3a://datalake-kc-s3-prd-bj/kylin/kcprodYcHG_kylin/datalake_kylin/dict/global_dict/GDT.GDT_CMPLYA_FCT_DIST_RESLT/IS_STAT/version_1679862387539
> 2023-03-26T20:31:14,501 INFO [logger-thread-0] dict.NGlobalDictionaryV2 : getMetaInfo versions.length is 12
> 2023-03-26T20:31:14,547 INFO [logger-thread-0] dict.NGlobalDictHDFSStore : because metaFiles.length is 0, metaInfo is null
> 2023-03-26T20:31:14,547 INFO [logger-thread-0] dict.NGlobalDictionaryV2 : getMetaInfo metadata is null : [true]{code}
> This is on s3, after renaming the dictionary directory, no metadata file is queried. However, if the meta is not obtained in the code and no error is reported, it is not reasonable to encode directly without repartition. In short, the result is that the encoding of the dictionary column on the flat table fails.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)