You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kylin.apache.org by "Gabor Arki (Jira)" <ji...@apache.org> on 2021/07/09 16:02:00 UTC
[jira] [Comment Edited] (KYLIN-5022) kylin升级新版本-/mnt/tmp/hbase-hbase/local/jars/tmp产生大量的kylin-coprocessor文件
[ https://issues.apache.org/jira/browse/KYLIN-5022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17378143#comment-17378143 ]
Gabor Arki edited comment on KYLIN-5022 at 7/9/21, 4:01 PM:
------------------------------------------------------------
We are encountering a similar issue with v3.1.0. Apart from the {{kylin-coprocessor-.jar}} files present in {{/mnt/tmp}}, this is also causing a disk space leakage. There are a lot of deleted files still referenced by the HBase region server process thus the disk space cannot be actually freed. This causes a significant discrepancy between the {{du}} and {{df}} calculations. In our case, when running the EMR for Kylin for a few months now, 50+ GB of such deleted *{{kylin-coprocessor-3.1.0-}}*{{.jar}} files are still occupying disk space on each core node:
{code:java}
[hadoop@ip-23-0-3-131 mnt]$ sudo lsof| grep "/mnt" | grep delete | more
java 16611 hbase 666r REG 259,3 5592785 143780992 /mnt/tmp/hbase-hbase/local/jars/tmp/.1369857329.kylin-coprocessor-3.1.0-SNAPSHOT-1597.jar
.1602171232484.jar (deleted)
java 16611 hbase 679r REG 259,3 5592785 144773588 /mnt/tmp/hbase-hbase/local/jars/tmp/.1170685576.kylin-coprocessor-3.1.0-SNAPSHOT-129.jar.
1602182959858.jar (deleted)
java 16611 hbase 680r REG 259,3 5592785 144321908 /mnt/tmp/hbase-hbase/local/jars/tmp/.-1329141342.kylin-coprocessor-3.1.0-SNAPSHOT-3653.ja
r.1602180128061.jar (deleted)
java 16611 hbase 681r REG 259,3 5592785 144248531 /mnt/tmp/hbase-hbase/local/jars/tmp/.-832621882.kylin-coprocessor-3.1.0-SNAPSHOT-3651.jar
.1602179699713.jar (deleted)
...{code}
was (Author: arkigabor):
We are encountering a similar issue with v3.1.0. Apart from the {{kylin-coprocessor-.jar}} files present in {{/mnt/tmp}}, this is also causing a disk space leakage. There are a lot of deleted files still referenced by the HBase region server process thus the disk space cannot be actually freed. This causes a significant discrepancy between the {{du}} and {{df}} calculations. In our case, when running the EMR for Kylin for a few months now, 50+ GB of such deleted *{{kylin-coprocessor-3.1.0-}}*{{.jar}} files are still occupying disk space on each core node:
[hadoop@ip-23-0-3-131 mnt]$ sudo lsof| grep "/mnt" | grep delete | more
java 16611 hbase 666r REG 259,3 5592785 143780992 /mnt/tmp/hbase-hbase/local/jars/tmp/.1369857329.kylin-coprocessor-3.1.0-SNAPSHOT-1597.jar
.1602171232484.jar (deleted)
java 16611 hbase 679r REG 259,3 5592785 144773588 /mnt/tmp/hbase-hbase/local/jars/tmp/.1170685576.kylin-coprocessor-3.1.0-SNAPSHOT-129.jar.
1602182959858.jar (deleted)
java 16611 hbase 680r REG 259,3 5592785 144321908 /mnt/tmp/hbase-hbase/local/jars/tmp/.-1329141342.kylin-coprocessor-3.1.0-SNAPSHOT-3653.ja
r.1602180128061.jar (deleted)
java 16611 hbase 681r REG 259,3 5592785 144248531 /mnt/tmp/hbase-hbase/local/jars/tmp/.-832621882.kylin-coprocessor-3.1.0-SNAPSHOT-3651.jar
.1602179699713.jar (deleted)
...........
...........
> kylin升级新版本-/mnt/tmp/hbase-hbase/local/jars/tmp产生大量的kylin-coprocessor文件
> ----------------------------------------------------------------------
>
> Key: KYLIN-5022
> URL: https://issues.apache.org/jira/browse/KYLIN-5022
> Project: Kylin
> Issue Type: Bug
> Components: Metadata, Storage - HBase
> Affects Versions: v3.1.1
> Reporter: star_dev
> Priority: Major
> Attachments: 屏幕快照1.png, 屏幕快照2.png, 日志.log
>
>
> kylin版本从3.0.2更新到3.1.1，还是用原来的元数据。
> 发现在EMR的core节点中有大量的kylin-coprocessor文件生成，见附件屏幕快照1，占用了大量的空间，导致hdfs文件系统可用空间变少。路径为/mnt/tmp/hbase-hbase/local/jars/tmp
> 查询官方文档 [http://kylin.apache.org/docs/howto/howto_update_coprocessor.html]
> 执行如下命令仍然不好用，日志信息见附件
> -------------------------
>
> {{$KYLIN_HOME/bin/kylin.sh org.apache.kylin.storage.hbase.util.DeployCoprocessorCLI default all}}
> {{-------------------------}}
> 同时发现kylin元数据 kylin_metadata/coprocessor/下有大量的kylin-coprocessor-3.1.1-*.jar文件，见附件屏幕快照2
>
> 是什么原因导致的这种现象？
> 如何才能在/mnt/tmp/hbase-hbase/local/jars/tmp路径下不再产生大量的文件？
--
This message was sent by Atlassian Jira
(v8.3.4#803005)