You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@kylin.apache.org by 曾耀武 <ze...@immomo.com> on 2017/10/18 12:37:24 UTC
Re: [外部邮件] Re: [外部邮件] 回复：kylin drop cube segment Htable 仍然存在
Yeah I have checked it was a bug , because I used hbase namespace “kylin” with the patch "https://issues.apache.org/jira/secure/attachment/12885320/KYLIN-2846-001.patch”
And I modified the code as below and it works well:
MOMO@MOMOdeMacBook-Pro-5:~/gitworkspace/kylin2.0/kylin/dist/apache-kylin-2.1.0-bin/tool$ git diff ../../../server-base/src/main/java/org/apache/kylin/rest/job/StorageCleanJobHbaseUtil.java
diff --git a/server-base/src/main/java/org/apache/kylin/rest/job/StorageCleanJobHbaseUtil.java b/server-base/src/main/java/org/apache/kylin/rest/job/StorageCleanJobHbaseUtil.java
index 3728ea1..937d02d 100644
--- a/server-base/src/main/java/org/apache/kylin/rest/job/StorageCleanJobHbaseUtil.java
+++ b/server-base/src/main/java/org/apache/kylin/rest/job/StorageCleanJobHbaseUtil.java
@@ -46,15 +46,28 @@ public class StorageCleanJobHbaseUtil {
public static void cleanUnusedHBaseTables(boolean delete, int deleteTimeout) throws IOException {
Configuration conf = HBaseConfiguration.create();
- CubeManager cubeMgr = CubeManager.getInstance(KylinConfig.getInstanceFromEnv());
+ KylinConfig config = KylinConfig.getInstanceFromEnv();
+ CubeManager cubeMgr = CubeManager.getInstance(config);
// get all kylin hbase tables
try (HBaseAdmin hbaseAdmin = new HBaseAdmin(conf)) {
- String tableNamePrefix = IRealizationConstants.SharedHbaseStorageLocationPrefix;
+ String namespace = config.getHBaseStorageNameSpace();
+ StringBuffer sb = new StringBuffer();
+ String tableNamePrefix = null;
+ if(namespace.equals("default")){
+ tableNamePrefix = IRealizationConstants.SharedHbaseStorageLocationPrefix;
+ }else{
+ sb.append(config.getHBaseStorageNameSpace()).append(":");
+ sb.append(IRealizationConstants.SharedHbaseStorageLocationPrefix);
+ tableNamePrefix = sb.toString();
+ }
+
发件人: ShaoFeng Shi <sh...@apache.org>>
答复: user <us...@kylin.apache.org>>
日期: 2017年10月17日 星期二 下午4:16
至: user <us...@kylin.apache.org>>
主题: [外部邮件] Re: [外部邮件] 回复：kylin drop cube segment Htable 仍然存在
Hi yaowu,
The StorageCleanupJob will delete the hbase tables that has no reference; If you found it didn't work, please check the output log of that command. It should say something.
在 2017年10月17日 下午3:39，yuyong.zhai <yu...@ele.me>>写道：
sh metastore.sh clean --delete true
sh kylin.sh org.apache.kylin.storage.hbase.util.StorageCleanupJob --delete true
原始邮件
发件人: 曾耀武<ze...@immomo.com>>
收件人: user<us...@kylin.apache.org>>
发送时间: 2017年10月17日(周二) 15:23
主题: Re: [外部邮件] 回复：kylin drop cube segment Htable 仍然存在
这个我看了,执行之后发现只是删除了 hive 中间表 kylin_intermediate_* 任务失败遗留的中间表 和 hdfs://nameservice1:8020/kylin/kylin-kylin_metadata/ 目录下没用的临时目录，
hbase 里的 segment 表仍然存在的
发件人: "yuyong.zhai" <yu...@ele.me>>
答复: user <us...@kylin.apache.org>>
日期: 2017年10月17日 星期二 下午3:04
至: user <us...@kylin.apache.org>>
主题: [外部邮件] 回复：kylin drop cube segment Htable 仍然存在
cleanup storage
http://kylin.apache.org/docs15/howto/howto_cleanup_storage.html
原始邮件
发件人: 曾耀武<ze...@immomo.com>>
收件人: user<us...@kylin.apache.org>>
发送时间: 2017年10月17日(周二) 14:02
主题: kylin drop cube segment Htable 仍然存在
嗨，all ，在清理没用的cube （kylin2.1.0) 时，发现cube 被purge和drop 之后，对应的segment hbase 表仍然在
base中存在，只是在kylinmeta 表的里的元信息被清除， 这是我的特例还是为了数据安全特意让他存在，
只能人工删除相关的表呢？
Best regards
--
Best regards,
Shaofeng Shi 史少锋
Re: [外部邮件] Re: [外部邮件] Re: [外部邮件] 回复：kylin drop cube segment Htable 仍然存在
Posted by 曾耀武 <ze...@immomo.com>.
Updated ，please check it ， thanks.
发件人: Billy Liu <bi...@apache.org>>
答复: user <us...@kylin.apache.org>>
日期: 2017年10月19日 星期四 上午12:40
至: user <us...@kylin.apache.org>>
主题: [外部邮件] Re: [外部邮件] Re: [外部邮件] 回复：kylin drop cube segment Htable 仍然存在
Agree. A new patch is needed in KYLIN-2846. It's not a bug still, but an unfinished improvement I think. Let's continue to improve this issue.
2017-10-18 21:05 GMT+08:00 ShaoFeng Shi <sh...@apache.org>>:
Hi Yaowu,
Could you please update this patch in KYLIN-2846? Storage cleanup should also support customized namespace. Thanks.
2017-10-18 20:37 GMT+08:00 曾耀武 <ze...@immomo.com>>:
Yeah I have checked it was a bug , because I used hbase namespace “kylin” with the patch "https://issues.apache.org/jira/secure/attachment/12885320/KYLIN-2846-001.patch”
And I modified the code as below and it works well:
MOMO@MOMOdeMacBook-Pro-5:~/gitworkspace/kylin2.0/kylin/dist/apache-kylin-2.1.0-bin/tool$ git diff ../../../server-base/src/main/java/org/apache/kylin/rest/job/StorageCleanJobHbaseUtil.java
diff --git a/server-base/src/main/java/org/apache/kylin/rest/job/StorageCleanJobHbaseUtil.java b/server-base/src/main/java/org/apache/kylin/rest/job/StorageCleanJobHbaseUtil.java
index 3728ea1..937d02d 100644
--- a/server-base/src/main/java/org/apache/kylin/rest/job/StorageCleanJobHbaseUtil.java
+++ b/server-base/src/main/java/org/apache/kylin/rest/job/StorageCleanJobHbaseUtil.java
@@ -46,15 +46,28 @@ public class StorageCleanJobHbaseUtil {
public static void cleanUnusedHBaseTables(boolean delete, int deleteTimeout) throws IOException {
Configuration conf = HBaseConfiguration.create();
- CubeManager cubeMgr = CubeManager.getInstance(KylinConfig.getInstanceFromEnv());
+ KylinConfig config = KylinConfig.getInstanceFromEnv();
+ CubeManager cubeMgr = CubeManager.getInstance(config);
// get all kylin hbase tables
try (HBaseAdmin hbaseAdmin = new HBaseAdmin(conf)) {
- String tableNamePrefix = IRealizationConstants.SharedHbaseStorageLocationPrefix;
+ String namespace = config.getHBaseStorageNameSpace();
+ StringBuffer sb = new StringBuffer();
+ String tableNamePrefix = null;
+ if(namespace.equals("default")){
+ tableNamePrefix = IRealizationConstants.SharedHbaseStorageLocationPrefix;
+ }else{
+ sb.append(config.getHBaseStorageNameSpace()).append(":");
+ sb.append(IRealizationConstants.SharedHbaseStorageLocationPrefix);
+ tableNamePrefix = sb.toString();
+ }
+
发件人: ShaoFeng Shi <sh...@apache.org>>
答复: user <us...@kylin.apache.org>>
日期: 2017年10月17日 星期二 下午4:16
至: user <us...@kylin.apache.org>>
主题: [外部邮件] Re: [外部邮件] 回复：kylin drop cube segment Htable 仍然存在
Hi yaowu,
The StorageCleanupJob will delete the hbase tables that has no reference; If you found it didn't work, please check the output log of that command. It should say something.
在 2017年10月17日 下午3:39，yuyong.zhai <yu...@ele.me>>写道：
sh metastore.sh clean --delete true
sh kylin.sh org.apache.kylin.storage.hbase.util.StorageCleanupJob --delete true
原始邮件
发件人: 曾耀武<ze...@immomo.com>>
收件人: user<us...@kylin.apache.org>>
发送时间: 2017年10月17日(周二) 15:23
主题: Re: [外部邮件] 回复：kylin drop cube segment Htable 仍然存在
这个我看了,执行之后发现只是删除了 hive 中间表 kylin_intermediate_* 任务失败遗留的中间表 和 hdfs://nameservice1:8020/kylin/kylin-kylin_metadata/ 目录下没用的临时目录，
hbase 里的 segment 表仍然存在的
发件人: "yuyong.zhai" <yu...@ele.me>>
答复: user <us...@kylin.apache.org>>
日期: 2017年10月17日 星期二 下午3:04
至: user <us...@kylin.apache.org>>
主题: [外部邮件] 回复：kylin drop cube segment Htable 仍然存在
cleanup storage
http://kylin.apache.org/docs15/howto/howto_cleanup_storage.html
原始邮件
发件人: 曾耀武<ze...@immomo.com>>
收件人: user<us...@kylin.apache.org>>
发送时间: 2017年10月17日(周二) 14:02
主题: kylin drop cube segment Htable 仍然存在
嗨，all ，在清理没用的cube （kylin2.1.0) 时，发现cube 被purge和drop 之后，对应的segment hbase 表仍然在
base中存在，只是在kylinmeta 表的里的元信息被清除， 这是我的特例还是为了数据安全特意让他存在，
只能人工删除相关的表呢？
Best regards
--
Best regards,
Shaofeng Shi 史少锋
--
Best regards,
Shaofeng Shi 史少锋
Re: [外部邮件] Re: [外部邮件] 回复：kylin drop cube segment Htable 仍然存在
Posted by Billy Liu <bi...@apache.org>.
Agree. A new patch is needed in KYLIN-2846. It's not a bug still, but an
unfinished improvement I think. Let's continue to improve this issue.
2017-10-18 21:05 GMT+08:00 ShaoFeng Shi <sh...@apache.org>:
> Hi Yaowu,
>
> Could you please update this patch in KYLIN-2846? Storage cleanup should
> also support customized namespace. Thanks.
>
> 2017-10-18 20:37 GMT+08:00 曾耀武 <ze...@immomo.com>:
>
>>
>> Yeah I have checked it was a bug , because I used hbase namespace
>> “kylin” with the patch "https://issues.apache.org/jir
>> a/secure/attachment/12885320/KYLIN-2846-001.patch”
>>
>> And I modified the code as below and it works well:
>>
>>
>> MOMO@MOMOdeMacBook-Pro-5:~/gitworkspace/kylin2.0/kylin/dist/apache-kylin-2.1.0-bin/tool$
>> git diff ../../../server-base/src/main/java/org/apache/kylin/rest/job
>> /StorageCleanJobHbaseUtil.java
>> diff --git a/server-base/src/main/java/org/apache/kylin/rest/job/StorageCleanJobHbaseUtil.java
>> b/server-base/src/main/java/org/apache/kylin/rest/job/Storag
>> eCleanJobHbaseUtil.java
>> index 3728ea1..937d02d 100644
>> --- a/server-base/src/main/java/org/apache/kylin/rest/job/Storag
>> eCleanJobHbaseUtil.java
>> +++ b/server-base/src/main/java/org/apache/kylin/rest/job/Storag
>> eCleanJobHbaseUtil.java
>> @@ -46,15 +46,28 @@ public class StorageCleanJobHbaseUtil {
>>
>> public static void cleanUnusedHBaseTables(boolean delete, int
>> deleteTimeout) throws IOException {
>> Configuration conf = HBaseConfiguration.create();
>> - CubeManager cubeMgr = CubeManager.getInstance(KylinC
>> onfig.getInstanceFromEnv());
>> + KylinConfig config = KylinConfig.getInstanceFromEnv();
>> + CubeManager cubeMgr = CubeManager.getInstance(config);
>> // get all kylin hbase tables
>> try (HBaseAdmin hbaseAdmin = new HBaseAdmin(conf)) {
>> - String tableNamePrefix = IRealizationConstants.SharedHb
>> aseStorageLocationPrefix;
>> + String namespace = config.getHBaseStorageNameSpace();
>> + StringBuffer sb = new StringBuffer();
>> + String tableNamePrefix = null;
>> + if(namespace.equals("default")){
>> + tableNamePrefix = IRealizationConstants.SharedHb
>> aseStorageLocationPrefix;
>> + }else{
>> + sb.append(config.getHBaseStor
>> ageNameSpace()).append(":");
>> + sb.append(IRealizationConstan
>> ts.SharedHbaseStorageLocationPrefix);
>> + tableNamePrefix = sb.toString();
>> + }
>> +
>>
>>
>>
>>
>>
>> 发件人: ShaoFeng Shi <sh...@apache.org>
>> 答复: user <us...@kylin.apache.org>
>> 日期: 2017年10月17日 星期二 下午4:16
>> 至: user <us...@kylin.apache.org>
>> 主题: [外部邮件] Re: [外部邮件] 回复：kylin drop cube segment Htable 仍然存在
>>
>> Hi yaowu,
>>
>> The StorageCleanupJob will delete the hbase tables that has no reference;
>> If you found it didn't work, please check the output log of that command.
>> It should say something.
>>
>> 在 2017年10月17日 下午3:39，yuyong.zhai <yu...@ele.me>写道：
>>
>>>
>>> sh metastore.sh clean --delete true
>>>
>>> sh kylin.sh org.apache.kylin.storage.hbase.util.StorageCleanupJob
>>> --delete true
>>> 原始邮件
>>> *发件人:* 曾耀武<ze...@immomo.com>
>>> *收件人:* user<us...@kylin.apache.org>
>>> *发送时间:* 2017年10月17日(周二) 15:23
>>> *主题:* Re: [外部邮件] 回复：kylin drop cube segment Htable 仍然存在
>>>
>>> 这个我看了,执行之后发现只是删除了 hive 中间表 kylin_intermediate_* 任务失败遗留的中间表
>>> 和 hdfs://nameservice1:8020/kylin/kylin-kylin_metadata/ 目录下没用的临时目录，
>>> hbase 里的 segment 表仍然存在的
>>>
>>> 发件人: "yuyong.zhai" <yu...@ele.me>
>>> 答复: user <us...@kylin.apache.org>
>>> 日期: 2017年10月17日 星期二 下午3:04
>>> 至: user <us...@kylin.apache.org>
>>> 主题: [外部邮件] 回复：kylin drop cube segment Htable 仍然存在
>>>
>>> cleanup storage
>>>
>>> http://kylin.apache.org/docs15/howto/howto_cleanup_storage.html
>>>
>>> 原始邮件
>>> *发件人:* 曾耀武<ze...@immomo.com>
>>> *收件人:* user<us...@kylin.apache.org>
>>> *发送时间:* 2017年10月17日(周二) 14:02
>>> *主题:* kylin drop cube segment Htable 仍然存在
>>>
>>>
>>>
>>> 嗨，all ，在清理没用的cube （kylin2.1.0) 时，发现cube 被purge和drop 之后，对应的segment hbase
>>> 表仍然在
>>> base中存在，只是在kylinmeta 表的里的元信息被清除， 这是我的特例还是为了数据安全特意让他存在，
>>> 只能人工删除相关的表呢？
>>>
>>>
>>> Best regards
>>>
>>>
>>
>>
>> --
>> Best regards,
>>
>> Shaofeng Shi 史少锋
>>
>>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
>
Re: [外部邮件] Re: [外部邮件] 回复：kylin drop cube segment Htable 仍然存在
Posted by ShaoFeng Shi <sh...@apache.org>.
Hi Yaowu,
Could you please update this patch in KYLIN-2846? Storage cleanup should
also support customized namespace. Thanks.
2017-10-18 20:37 GMT+08:00 曾耀武 <ze...@immomo.com>:
>
> Yeah I have checked it was a bug , because I used hbase namespace “kylin”
> with the patch "https://issues.apache.org/jira/secure/attachment/
> 12885320/KYLIN-2846-001.patch”
>
> And I modified the code as below and it works well:
>
>
> MOMO@MOMOdeMacBook-Pro-5:~/gitworkspace/kylin2.0/kylin/
> dist/apache-kylin-2.1.0-bin/tool$ git diff ../../../server-base/src/main/
> java/org/apache/kylin/rest/job/StorageCleanJobHbaseUtil.java
> diff --git a/server-base/src/main/java/org/apache/kylin/rest/job/StorageCleanJobHbaseUtil.java
> b/server-base/src/main/java/org/apache/kylin/rest/job/
> StorageCleanJobHbaseUtil.java
> index 3728ea1..937d02d 100644
> --- a/server-base/src/main/java/org/apache/kylin/rest/job/
> StorageCleanJobHbaseUtil.java
> +++ b/server-base/src/main/java/org/apache/kylin/rest/job/
> StorageCleanJobHbaseUtil.java
> @@ -46,15 +46,28 @@ public class StorageCleanJobHbaseUtil {
>
> public static void cleanUnusedHBaseTables(boolean delete, int
> deleteTimeout) throws IOException {
> Configuration conf = HBaseConfiguration.create();
> - CubeManager cubeMgr = CubeManager.getInstance(KylinConfig.
> getInstanceFromEnv());
> + KylinConfig config = KylinConfig.getInstanceFromEnv();
> + CubeManager cubeMgr = CubeManager.getInstance(config);
> // get all kylin hbase tables
> try (HBaseAdmin hbaseAdmin = new HBaseAdmin(conf)) {
> - String tableNamePrefix = IRealizationConstants.
> SharedHbaseStorageLocationPrefix;
> + String namespace = config.getHBaseStorageNameSpace();
> + StringBuffer sb = new StringBuffer();
> + String tableNamePrefix = null;
> + if(namespace.equals("default")){
> + tableNamePrefix = IRealizationConstants.
> SharedHbaseStorageLocationPrefix;
> + }else{
> + sb.append(config.getHBaseStorageNameSpace()).append(":");
> + sb.append(IRealizationConstants.
> SharedHbaseStorageLocationPrefix);
> + tableNamePrefix = sb.toString();
> + }
> +
>
>
>
>
>
> 发件人: ShaoFeng Shi <sh...@apache.org>
> 答复: user <us...@kylin.apache.org>
> 日期: 2017年10月17日 星期二 下午4:16
> 至: user <us...@kylin.apache.org>
> 主题: [外部邮件] Re: [外部邮件] 回复：kylin drop cube segment Htable 仍然存在
>
> Hi yaowu,
>
> The StorageCleanupJob will delete the hbase tables that has no reference;
> If you found it didn't work, please check the output log of that command.
> It should say something.
>
> 在 2017年10月17日 下午3:39，yuyong.zhai <yu...@ele.me>写道：
>
>>
>> sh metastore.sh clean --delete true
>>
>> sh kylin.sh org.apache.kylin.storage.hbase.util.StorageCleanupJob
>> --delete true
>> 原始邮件
>> *发件人:* 曾耀武<ze...@immomo.com>
>> *收件人:* user<us...@kylin.apache.org>
>> *发送时间:* 2017年10月17日(周二) 15:23
>> *主题:* Re: [外部邮件] 回复：kylin drop cube segment Htable 仍然存在
>>
>> 这个我看了,执行之后发现只是删除了 hive 中间表 kylin_intermediate_* 任务失败遗留的中间表
>> 和 hdfs://nameservice1:8020/kylin/kylin-kylin_metadata/ 目录下没用的临时目录，
>> hbase 里的 segment 表仍然存在的
>>
>> 发件人: "yuyong.zhai" <yu...@ele.me>
>> 答复: user <us...@kylin.apache.org>
>> 日期: 2017年10月17日 星期二 下午3:04
>> 至: user <us...@kylin.apache.org>
>> 主题: [外部邮件] 回复：kylin drop cube segment Htable 仍然存在
>>
>> cleanup storage
>>
>> http://kylin.apache.org/docs15/howto/howto_cleanup_storage.html
>>
>> 原始邮件
>> *发件人:* 曾耀武<ze...@immomo.com>
>> *收件人:* user<us...@kylin.apache.org>
>> *发送时间:* 2017年10月17日(周二) 14:02
>> *主题:* kylin drop cube segment Htable 仍然存在
>>
>>
>>
>> 嗨，all ，在清理没用的cube （kylin2.1.0) 时，发现cube 被purge和drop 之后，对应的segment hbase
>> 表仍然在
>> base中存在，只是在kylinmeta 表的里的元信息被清除， 这是我的特例还是为了数据安全特意让他存在，
>> 只能人工删除相关的表呢？
>>
>>
>> Best regards
>>
>>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
>
--
Best regards,
Shaofeng Shi 史少锋