[jira] [Comment Edited] (KYLIN-4896) 构建过程中, cube metadata 丢失

["Linghui Zeng (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KYLIN-4896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17278707#comment-17278707 ] 

Linghui Zeng edited comment on KYLIN-4896 at 2/4/21, 9:40 AM:
--------------------------------------------------------------

This issue is related to Kylin-4153.

 

Before the versions of Kylin 2.6.4 and Kylin 3.0.0-beta, It could happen that the metadata marked that the big resource (dictionaries, screenshots, etc.) file was written to HDFS successfully, while the process failed the HDFS did not have the data. In Kylin-4153, to fix the problem, we introduced the step of deleteResourceImpl(resPathStr) in rollback() when HDFS files fail to write. 

 
{code:java}
public void rollback() {
    if (hasRollback)
        return;

    hasRollback = true;

    try {
        logger.error("Rollback {} from {}", realPath, hasOldFile ? backPath.toString() : "<empty>");

        if (fs.exists(realPath))
            fs.delete(realPath, true);

        if (hasOldFile) {
            fs.rename(backPath, realPath);
        } else {
            logger.warn("Try delete empty entry {}", resPathStr);
            deleteResourceImpl(resPathStr);
        }
    } catch (IOException ex2) {
        logger.error("Rollback failed", ex2);
    }
}{code}
 

A file is defined to be a big resource file when its size exceeds kvSizeLimit, which is configured by hbase.client.keyvalue.maxsize, and the default is 10MB.

In this case,  hbase.client.keyvalue.maxsize is set to 1MB, CubeInstance exceeds the limit as the grows of segments, thus CubeInstance.json was written to HDFS as a big resource file.
{code:java}
if (content.length > kvSizeLimit) {
    pushdown = writePushdown(resPath, ContentWriter.create(content));
    content = BytesUtil.EMPTY_BYTE_ARRAY;
}
{code}
However, when writing CubeInstance.json into HDFS, there were some unknown issues and an exception was caught, then rollback() was called.
{code:java}
catch (Exception ex) {
    if (pushdown != null)
        pushdown.rollback();
    throw ex;
{code}
When calling rollback, there was no old CubeInstance.json file in HDFS, then the resource path was deleted from metastore and so the cube metadata lost.

 

 

 

In the design of Kylin, CubeInstance is supposed to be stored in HBase, as the size of CubeInstance should not exceed the limit of 10MB theoretically. It should not be written into HDFS.

To fix the problem, we do the following:

To avoid CubeInstance.json being pushed down to HDFS, we set the minimum of hbase.client.keyvalue.maxsize to be 10MB; if user defines it to be smaller than the minimum, it will be set to 10MB.
{code:java}
int kvSizeLimitActual = Integer.parseInt(getConnection().getConfiguration().get("hbase.client.keyvalue.maxsize", "10485760"));
kvSizeLimit = kvSizeLimitActual < 10485760 ? kvSizeLimitActual : 10485760;{code}
We print an info in log to remind the user that a file is going to be pushed down.
{code:java}
if (content.length > kvSizeLimit) {
    logger.info("Length of content exceeds the limit of {} bytes, push down {} to HDFS", kvSizeLimit, resPath);
    pushdown = writePushdown(resPath, ContentWriter.create(content));
    content = BytesUtil.EMPTY_BYTE_ARRAY;
}
{code}
!1Dvn5TLm3mYAAAAASUVORK5CYII=!


was (Author: helen_zeng):
This issue is related to [Kylin-4153|https://issues.apache.org/jira/browse/KYLIN-4153].

 

Before the versions of Kylin 2.6.4 and Kylin 3.0.0-beta, It could happen that the metadata marked that the big resource (dictionaries, screenshots, etc.) file was written to HDFS successfully, while the process failed the HDFS did not have the data. In Kylin-4153, to fix the problem, we introduced the step of deleteResourceImpl(resPathStr) in rollback() when HDFS files fail to write. 

!AUXmWWIRHGiVAAAAAElFTkSuQmCC|width=745,height=417!

A file is defined to be a big resource file when its size exceeds kvSizeLimit, which is configured by hbase.client.keyvalue.maxsize, and the default is 10MB.

In this case,  hbase.client.keyvalue.maxsize is set to 1MB, CubeInstance exceeds the limit as the grows of segments, thus CubeInstance.json was written to HDFS as a big resource file.

!FAAAAAElFTkSuQmCC|width=600,height=111!

However, when writing CubeInstance.json into HDFS, there were some unknown issues and an exception was caught, then rollback() was called.

!U0h 422R1gAAAAABJRU5ErkJggg==|width=335,height=116!

When calling rollback, there was no old CubeInstance.json file in HDFS, then the resource path was deleted from metastore and so the cube metadata lost.

 

 

 

In the design of Kylin, CubeInstance is supposed to be stored in HBase, as the size of CubeInstance should not exceed the limit of 10MB theoretically. It should not be written into HDFS.

To fix the problem, we do the following:

To avoid CubeInstance.json being pushed down to HDFS, we set the minimum of hbase.client.keyvalue.maxsize to be 10MB; if user defines it to be smaller than the minimum, it will be set to 10MB.

!H8AAAAASUVORK5CYII=|width=1084,height=98!

We print an info in log to remind the user that a file is going to be pushed down.

!1Dvn5TLm3mYAAAAASUVORK5CYII=|width=960,height=131!

> 构建过程中, cube metadata 丢失
> -----------------------
>
>                 Key: KYLIN-4896
>                 URL: https://issues.apache.org/jira/browse/KYLIN-4896
>             Project: Kylin
>          Issue Type: Bug
>    Affects Versions: v3.1.1
>            Reporter: hejian
>            Priority: Major
>         Attachments: image-2021-02-03-19-11-09-261.png
>
>
> {quote}今天又出现了在cube使用分布式构建过程中，cube metadata丢失的问题了，
> 构建到第四步（Build Dimension Dictionary）的时候出现了这个cube的metedata丢失的问题。
> 错误日志如下，
> !image-2021-02-03-19-11-09-261.png!
> kylin版本3.1.1采用的是  kylin.job.scheduler.default=2，
> 其余的配置均为正确的。{quote}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)