[jira] [Created] (KYLIN-5650) In the cloud environment, there is a probability that the dictionary metadata file will be read abnormally during building job, resulting in incorrect query results.

["Zhiting Guo (Jira)" <ji...@apache.org>]

Zhiting Guo created KYLIN-5650:
----------------------------------

             Summary: In the cloud environment, there is a probability that the dictionary metadata file will be read abnormally during building job, resulting in incorrect query results.
                 Key: KYLIN-5650
                 URL: https://issues.apache.org/jira/browse/KYLIN-5650
             Project: Kylin
          Issue Type: Bug
          Components: Tools, Build and Test
    Affects Versions: 5.0-alpha
            Reporter: Zhiting Guo
             Fix For: 5.0-alpha
         Attachments: In the cloud environment, there is a probability that the dictionary metadata file will be read abnormally during building job, resulting in incorrect query results..pdf

Checked the dictionary, there are no duplicate values. Checked the execution plan of the build dictionary step, there is no problem. Checked the steps of building a flat table and found that there was a problem in the step of flat table encoding dictionary.

The reason for the error is that the encoding is not performed after repartition according to the dictionary column. As shown in the figure, there is no repartition, and the encode column appears in the plan.

There are also the following logs:
{code:java}
2023-03-26T20:26:30,868 INFO  [logger-thread-0] dict.NGlobalDictHDFSStore : Commit from s3a://datalake-kc-s3-prd-bj/kylin/kcprodYcHG_kylin/datalake_kylin/dict/global_dict/GDT.GDT_CMPLYA_FCT_DIST_RESLT/IS_STAT/working to s3a://datalake-kc-s3-prd-bj/kylin/kcprodYcHG_kylin/datalake_kylin/dict/global_dict/GDT.GDT_CMPLYA_FCT_DIST_RESLT/IS_STAT/version_1679862387539

2023-03-26T20:31:14,501 INFO  [logger-thread-0] dict.NGlobalDictionaryV2 : getMetaInfo versions.length is 12
2023-03-26T20:31:14,547 INFO  [logger-thread-0] dict.NGlobalDictHDFSStore : because metaFiles.length is 0, metaInfo is null
2023-03-26T20:31:14,547 INFO  [logger-thread-0] dict.NGlobalDictionaryV2 : getMetaInfo metadata is null : [true]{code}
This is on s3, after renaming the dictionary directory, no metadata file is queried. However, if the meta is not obtained in the code and no error is reported, it is not reasonable to encode directly without repartition. In short, the result is that the encoding of the dictionary column on the flat table fails.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)