You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@kylin.apache.org by 240902713 <24...@qq.com> on 2019/07/31 07:42:27 UTC

kylin doesn't drop intermediate lookup table after job finished

Hello :


I have a  question about why kylin doesn't drop the intermediate lookup table after job finished.


My kylin version: 2.6.0


I have a cube with a fact table join a lookup table  in hive, and both are hive view. I submit a job one time per hour.


Kylin can drop the intermediate fact table , but doesn't drop the intermediate lookup table .


I check the source code , and find out that , at '13 step: Hive Cleanup' ,  the source code has been commented. as the 


following picture shows:





After several days, in hive, there are a lot of intermediate lookup tables which are not droped.


Why can this step result some concurrent issue ? 




Looking forward your reply.


Thanks !

回复：kylin doesn't drop intermediate lookup table after job finished

Posted by 240902713 <24...@qq.com>.

StorageCleanupJob tool can result some unkown error , so I just drop the table manually .


Thanks .




------------------ 原始邮件 ------------------
发件人: "Xiaoxiang Yu"<hi...@126.com>;
发送时间: 2019年7月31日(星期三) 下午4:27
收件人: "user"<us...@qq.com>;

主题: Re:kylin doesn't drop intermediate lookup table after job finished



Hi,
    I think these tables could be cleaned by StorageCleanupJob. Please refer to http://kylin.apache.org/docs/howto/howto_cleanup_storage.html to clean these intermediate table.




-----------------
-----------------
Best wishes to you ! 
From ：Xiaoxiang Yu




At 2019-07-31 15:42:27, "240902713" <24...@qq.com> wrote:
 Hello :


I have a  question about why kylin doesn't drop the intermediate lookup table after job finished.


My kylin version: 2.6.0


I have a cube with a fact table join a lookup table  in hive, and both are hive view. I submit a job one time per hour.


Kylin can drop the intermediate fact table , but doesn't drop the intermediate lookup table .


I check the source code , and find out that , at '13 step: Hive Cleanup' ,  the source code has been commented. as the 


following picture shows:





After several days, in hive, there are a lot of intermediate lookup tables which are not droped.


Why can this step result some concurrent issue ? 




Looking forward your reply.


Thanks !

回复： kylin doesn't drop intermediate lookup table after job finished

Posted by 240902713 <24...@qq.com>.

Thanks, ShaoFeng

------------------ 原始邮件 ------------------
发件人: "ShaoFeng Shi"<sh...@apache.org>;
发送时间: 2019年7月31日(星期三) 晚上10:12
收件人: "user"<us...@kylin.apache.org>;
抄送: "240902713"<24...@qq.com>;
主题: Re: kylin doesn't drop intermediate lookup table after job finished

Hello,

Good feedback. I think it is a legacy issue. At the very beginning, the intermediate table name for the lookup table in view is likely to be the same. So if a job dropped the table on the finish, it may cause another job failed. As a temporary solution, we disabled the cleanup for Hive view tables.

Now since KYLIN-3515 has solved the conflict problem, I think this cleanup can be uncommented. JIRA is created for this: https://issues.apache.org/jira/browse/KYLIN-4121 If you want to contribute, welcome to raise a PR. Thanks!

Best regards,

Shaofeng Shi 史少锋
Apache Kylin PMC
Email: shaofengshi@apache.org

Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
Join Kylin user mail group: user-subscribe@kylin.apache.org
Join Kylin dev mail group: dev-subscribe@kylin.apache.org

Xiaoxiang Yu <hi...@126.com> 于2019年7月31日周三 上午1:27写道：

Hi,
I think these tables could be cleaned by StorageCleanupJob. Please refer to http://kylin.apache.org/docs/howto/howto_cleanup_storage.html to clean these intermediate table.

-----------------
-----------------
Best wishes to you !
From ：Xiaoxiang Yu

At 2019-07-31 15:42:27, "240902713" <24...@qq.com> wrote:
Hello :

I have a question about why kylin doesn't drop the intermediate lookup table after job finished.

My kylin version: 2.6.0

I have a cube with a fact table join a lookup table in hive, and both are hive view. I submit a job one time per hour.

Kylin can drop the intermediate fact table , but doesn't drop the intermediate lookup table .

I check the source code , and find out that , at '13 step: Hive Cleanup' , the source code has been commented. as the

following picture shows:

After several days, in hive, there are a lot of intermediate lookup tables which are not droped.

Why can this step result some concurrent issue ?

Looking forward your reply.

Thanks !

Re: kylin doesn't drop intermediate lookup table after job finished

Posted by ShaoFeng Shi <sh...@apache.org>.

Hello,

Good feedback. I think it is a legacy issue. At the very beginning, the
intermediate table name for the lookup table in view is likely to be the
same. So if a job dropped the table on the finish, it may cause another job
failed. As a temporary solution, we disabled the cleanup for Hive view
tables.

Now since KYLIN-3515 has solved the conflict problem, I think this cleanup
can be uncommented. JIRA is created for this:
https://issues.apache.org/jira/browse/KYLIN-4121 If you want to contribute,
welcome to raise a PR. Thanks!

Best regards,

Shaofeng Shi 史少锋
Apache Kylin PMC
Email: shaofengshi@apache.org

Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
Join Kylin user mail group: user-subscribe@kylin.apache.org
Join Kylin dev mail group: dev-subscribe@kylin.apache.org




Xiaoxiang Yu <hi...@126.com> 于2019年7月31日周三 上午1:27写道：

> Hi,
>     I think these tables could be cleaned by StorageCleanupJob. Please
> refer to http://kylin.apache.org/docs/howto/howto_cleanup_storage.html to
> clean these intermediate table.
>
>
> *-----------------*
> *-----------------*
> *Best wishes to you ! *
> *From ：**Xiaoxiang Yu*
>
> At 2019-07-31 15:42:27, "240902713" <24...@qq.com> wrote:
>
> Hello :
>
> I have a  question about why kylin doesn't drop the intermediate lookup
> table after job finished.
>
> My kylin version: 2.6.0
>
> I have a cube with a fact table join a lookup table  in hive, and both are
> hive view. I submit a job one time per hour.
>
> Kylin can drop the intermediate fact table , but doesn't drop the intermediate
> lookup table .
>
> I check the source code , and find out that , at '13 step: Hive Cleanup'
> ,  the source code has been commented. as the
>
> following picture shows:
>
>
> After several days, in hive, there are a lot of intermediate lookup
> tables which are not droped.
>
> Why can this step result some concurrent issue ?
>
>
> Looking forward your reply.
>
> Thanks !
>
>

Re:kylin doesn't drop intermediate lookup table after job finished

Posted by Xiaoxiang Yu <hi...@126.com>.

Hi,
    I think these tables could be cleaned by StorageCleanupJob. Please refer to http://kylin.apache.org/docs/howto/howto_cleanup_storage.html to clean these intermediate table.




-----------------
-----------------
Best wishes to you ! 
From ：Xiaoxiang Yu

At 2019-07-31 15:42:27, "240902713" <24...@qq.com> wrote:

Hello :


I have a  question about why kylin doesn't drop the intermediate lookup table after job finished.


My kylin version: 2.6.0


I have a cube with a fact table join a lookup table  in hive, and both are hive view. I submit a job one time per hour.


Kylin can drop the intermediate fact table , but doesn't drop the intermediate lookup table .


I check the source code , and find out that , at '13 step: Hive Cleanup' ,  the source code has been commented. as the 


following picture shows:




After several days, in hive, there are a lot of intermediate lookup tables which are not droped.


Why can this step result some concurrent issue ? 




Looking forward your reply.


Thanks !