You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kylin.apache.org by ShaoFeng Shi <sh...@apache.org> on 2017/01/24 12:14:03 UTC
Re: 答复: more than 1 append dict for globalDict
@Yerui, do you have suggestion on this?
2017-01-24 18:19 GMT+08:00 市场中心-ZHANGDA32698 <zd...@ly.com>:
> Hi ShaoFeng,
>
>
>
> Thanks for your suggestion. However since ours is a daily build, I guess
> there’s no concurrent issue.
>
> The problem stops us from building subsequent segments. I try to remove
> one of the Dicts from hbase and try a rebuild, but seems that they are
> referenced at somewhere else and I got ‘java.lang.IllegalStateException:
> No resource found at -- /dict/APPLYDATA_DSDCADMIN.FLY_
> ZHUANGHUALV_PROC1/LOGINKEY/ae5a65ca-022f-4c81-89f2-7cacb2789888.dict’
> problem at step 3 ‘Extract Fact Table Distinct Columns’. Of course I can
> add the key back and let it continue, but without surprise it will stop at
> setp4 ‘Build Dimension Dictionary’ with same error I encountered before
> ‘GlobalDict xxxx should have 0 or 1 append dict but 2’
>
> How can I resolve this and continue my cube building? Any suggestion is
> appreciated, thanks!
>
>
>
>
>
> *发件人:* ShaoFeng Shi [mailto:shaofengshi@apache.org]
> *发送时间:* 2017年1月24日 17:01
> *收件人:* user
> *主题:* Re: more than 1 append dict for globalDict
>
>
>
> Hi zhangda,
>
>
>
> Do you have multiple segments (which has the count distinct measure)
> building concurrently? In 1.6.0 and before there is a concurrent bug I
> think; Yerui Sun fixed it in https://issues.apache.org/
> jira/browse/KYLIN-2192
>
>
>
> So, please check whether it is this problem first; If yes, you need add
> some control (lock) to avoid concurrent build for this cube.
>
>
>
> 2017-01-24 16:33 GMT+08:00 市场中心-ZHANGDA32698 <zd...@ly.com>:
>
> Hi there,
>
>
>
> I have a cube computing for some UV statistics. Since it requires global
> count distinct operation, in the advanced dictionaries setting I put UV key
> column’s builder class as ‘GlobalDictionaryBuilder’.
>
> Everyday build went on without any problem until yesterday there was a
> ‘GlobalDict xxxx should have 0 or 1 append dict but 2"’ exception. I
> checked the ‘kylin_metadata’ table in the hbase , and saw there were indeed
> 2 dict
>
> '/dict/APPLYDATA_DSDCADMIN.FLY_ZHUANGHUALV_PROC1/
> LOGINKEY/ae5a65ca-022f-4c81-89f2-7cacb2789888.dict'
>
> /dict/APPLYDATA_DSDCADMIN.FLY_ZHUANGHUALV_PROC1/LOGINKEY/
> cedce514-657d-4032-a591-7f01d984df65.dict'
>
> Fyi, APPLYDATA_DSDCADMIN.FLY_ZHUANGHUALV_PROC1 is the hive table name,
> LOGINKEY is the count distinct key
>
> I’m not sure what had gone wrong, and I suppose global dict on 1 key
> should be unique. Has anyone ever encountered similar error before, can
> share some ideas? Thanks!
>
>
>
>
>
> --
>
> Best regards,
>
>
>
> Shaofeng Shi 史少锋
>
>
>
--
Best regards,
Shaofeng Shi 史少锋
答复: 答复: more than 1 append dict for globalDict
Posted by 市场中心-ZHANGDA32698 <zd...@ly.com>.
Hi ShaoFeng,
I see, thanks for the solution. Actually I notice that the extra dictionary appeared right after the last segment were successfully built, which means all segments in the cube.json are pointing to the older dictionary except the last one. So similar to your suggestion, I wonder if I can remove the last segment together with the extra the dictionary, then do a rebuild on the last segment. Is it ok to do so?
Another thing, how do I remove an existing segment? Found this https://issues.apache.org/jira/browse/KYLIN-1540 but seem doesn’t work in v1.6.0 yet. Can I just edit the cube.json file and remove the want-to-remove segment part?
Thanks !
发件人: ShaoFeng Shi [mailto:shaofengshi@apache.org]
发送时间: 2017年1月24日 20:27
收件人: user; dev; Yerui Sun
主题: Re: 答复: more than 1 append dict for globalDict
Yerui is on vacation;
If you're urgent, you can try this way (also confirmed with yerui);
1) backup your metadata to local folder;
2) copy to backup the two global dictionaries in some other folder;
3) among the two dictionaries, pick the bigger one (it should includes all values in the smaller one), and then in the cube json, replace all the references to smaller dict with the bigger one; You can edit the Cube json from Kylin's web gui, in the "Admin" -> "Edit JSON";
4) remove the smaller one dict from kylin metadata, use kylin's command, like : "./bin/metastore.sh remove /dict/APPLYDATA_DSDCADMIN.FLY_ZHUANGHUALV_PROC1/LOGINKEY/ae5a65ca-022f-4c81-89f2-7cacb2789888.dict"
2017-01-24 20:14 GMT+08:00 ShaoFeng Shi <sh...@apache.org>>:
@Yerui, do you have suggestion on this?
2017-01-24 18:19 GMT+08:00 市场中心-ZHANGDA32698 <zd...@ly.com>>:
Hi ShaoFeng,
Thanks for your suggestion. However since ours is a daily build, I guess there’s no concurrent issue.
The problem stops us from building subsequent segments. I try to remove one of the Dicts from hbase and try a rebuild, but seems that they are referenced at somewhere else and I got ‘java.lang.IllegalStateException: No resource found at -- /dict/APPLYDATA_DSDCADMIN.FLY_ZHUANGHUALV_PROC1/LOGINKEY/ae5a65ca-022f-4c81-89f2-7cacb2789888.dict’ problem at step 3 ‘Extract Fact Table Distinct Columns’. Of course I can add the key back and let it continue, but without surprise it will stop at setp4 ‘Build Dimension Dictionary’ with same error I encountered before ‘GlobalDict xxxx should have 0 or 1 append dict but 2’
How can I resolve this and continue my cube building? Any suggestion is appreciated, thanks!
发件人: ShaoFeng Shi [mailto:shaofengshi@apache.org<ma...@apache.org>]
发送时间: 2017年1月24日 17:01
收件人: user
主题: Re: more than 1 append dict for globalDict
Hi zhangda,
Do you have multiple segments (which has the count distinct measure) building concurrently? In 1.6.0 and before there is a concurrent bug I think; Yerui Sun fixed it in https://issues.apache.org/jira/browse/KYLIN-2192
So, please check whether it is this problem first; If yes, you need add some control (lock) to avoid concurrent build for this cube.
2017-01-24 16:33 GMT+08:00 市场中心-ZHANGDA32698 <zd...@ly.com>>:
Hi there,
I have a cube computing for some UV statistics. Since it requires global count distinct operation, in the advanced dictionaries setting I put UV key column’s builder class as ‘GlobalDictionaryBuilder’.
Everyday build went on without any problem until yesterday there was a ‘GlobalDict xxxx should have 0 or 1 append dict but 2"’ exception. I checked the ‘kylin_metadata’ table in the hbase , and saw there were indeed 2 dict
'/dict/APPLYDATA_DSDCADMIN.FLY_ZHUANGHUALV_PROC1/LOGINKEY/ae5a65ca-022f-4c81-89f2-7cacb2789888.dict'
/dict/APPLYDATA_DSDCADMIN.FLY_ZHUANGHUALV_PROC1/LOGINKEY/cedce514-657d-4032-a591-7f01d984df65.dict'
Fyi, APPLYDATA_DSDCADMIN.FLY_ZHUANGHUALV_PROC1 is the hive table name, LOGINKEY is the count distinct key
I’m not sure what had gone wrong, and I suppose global dict on 1 key should be unique. Has anyone ever encountered similar error before, can share some ideas? Thanks!
--
Best regards,
Shaofeng Shi 史少锋
--
Best regards,
Shaofeng Shi 史少锋
--
Best regards,
Shaofeng Shi 史少锋
答复: 答复: more than 1 append dict for globalDict
Posted by 市场中心-ZHANGDA32698 <zd...@ly.com>.
Hi ShaoFeng,
I see, thanks for the solution. Actually I notice that the extra dictionary appeared right after the last segment were successfully built, which means all segments in the cube.json are pointing to the older dictionary except the last one. So similar to your suggestion, I wonder if I can remove the last segment together with the extra the dictionary, then do a rebuild on the last segment. Is it ok to do so?
Another thing, how do I remove an existing segment? Found this https://issues.apache.org/jira/browse/KYLIN-1540 but seem doesn’t work in v1.6.0 yet. Can I just edit the cube.json file and remove the want-to-remove segment part?
Thanks !
发件人: ShaoFeng Shi [mailto:shaofengshi@apache.org]
发送时间: 2017年1月24日 20:27
收件人: user; dev; Yerui Sun
主题: Re: 答复: more than 1 append dict for globalDict
Yerui is on vacation;
If you're urgent, you can try this way (also confirmed with yerui);
1) backup your metadata to local folder;
2) copy to backup the two global dictionaries in some other folder;
3) among the two dictionaries, pick the bigger one (it should includes all values in the smaller one), and then in the cube json, replace all the references to smaller dict with the bigger one; You can edit the Cube json from Kylin's web gui, in the "Admin" -> "Edit JSON";
4) remove the smaller one dict from kylin metadata, use kylin's command, like : "./bin/metastore.sh remove /dict/APPLYDATA_DSDCADMIN.FLY_ZHUANGHUALV_PROC1/LOGINKEY/ae5a65ca-022f-4c81-89f2-7cacb2789888.dict"
2017-01-24 20:14 GMT+08:00 ShaoFeng Shi <sh...@apache.org>>:
@Yerui, do you have suggestion on this?
2017-01-24 18:19 GMT+08:00 市场中心-ZHANGDA32698 <zd...@ly.com>>:
Hi ShaoFeng,
Thanks for your suggestion. However since ours is a daily build, I guess there’s no concurrent issue.
The problem stops us from building subsequent segments. I try to remove one of the Dicts from hbase and try a rebuild, but seems that they are referenced at somewhere else and I got ‘java.lang.IllegalStateException: No resource found at -- /dict/APPLYDATA_DSDCADMIN.FLY_ZHUANGHUALV_PROC1/LOGINKEY/ae5a65ca-022f-4c81-89f2-7cacb2789888.dict’ problem at step 3 ‘Extract Fact Table Distinct Columns’. Of course I can add the key back and let it continue, but without surprise it will stop at setp4 ‘Build Dimension Dictionary’ with same error I encountered before ‘GlobalDict xxxx should have 0 or 1 append dict but 2’
How can I resolve this and continue my cube building? Any suggestion is appreciated, thanks!
发件人: ShaoFeng Shi [mailto:shaofengshi@apache.org<ma...@apache.org>]
发送时间: 2017年1月24日 17:01
收件人: user
主题: Re: more than 1 append dict for globalDict
Hi zhangda,
Do you have multiple segments (which has the count distinct measure) building concurrently? In 1.6.0 and before there is a concurrent bug I think; Yerui Sun fixed it in https://issues.apache.org/jira/browse/KYLIN-2192
So, please check whether it is this problem first; If yes, you need add some control (lock) to avoid concurrent build for this cube.
2017-01-24 16:33 GMT+08:00 市场中心-ZHANGDA32698 <zd...@ly.com>>:
Hi there,
I have a cube computing for some UV statistics. Since it requires global count distinct operation, in the advanced dictionaries setting I put UV key column’s builder class as ‘GlobalDictionaryBuilder’.
Everyday build went on without any problem until yesterday there was a ‘GlobalDict xxxx should have 0 or 1 append dict but 2"’ exception. I checked the ‘kylin_metadata’ table in the hbase , and saw there were indeed 2 dict
'/dict/APPLYDATA_DSDCADMIN.FLY_ZHUANGHUALV_PROC1/LOGINKEY/ae5a65ca-022f-4c81-89f2-7cacb2789888.dict'
/dict/APPLYDATA_DSDCADMIN.FLY_ZHUANGHUALV_PROC1/LOGINKEY/cedce514-657d-4032-a591-7f01d984df65.dict'
Fyi, APPLYDATA_DSDCADMIN.FLY_ZHUANGHUALV_PROC1 is the hive table name, LOGINKEY is the count distinct key
I’m not sure what had gone wrong, and I suppose global dict on 1 key should be unique. Has anyone ever encountered similar error before, can share some ideas? Thanks!
--
Best regards,
Shaofeng Shi 史少锋
--
Best regards,
Shaofeng Shi 史少锋
--
Best regards,
Shaofeng Shi 史少锋
Re: 答复: more than 1 append dict for globalDict
Posted by ShaoFeng Shi <sh...@apache.org>.
Yerui is on vacation;
If you're urgent, you can try this way (also confirmed with yerui);
1) backup your metadata to local folder;
2) copy to backup the two global dictionaries in some other folder;
3) among the two dictionaries, pick the bigger one (it should includes all
values in the smaller one), and then in the cube json, replace all the
references to smaller dict with the bigger one; You can edit the Cube json
from Kylin's web gui, in the "Admin" -> "Edit JSON";
4) remove the smaller one dict from kylin metadata, use kylin's command,
like : "./bin/metastore.sh remove /dict/APPLYDATA_DSDCADMIN.FLY_
ZHUANGHUALV_PROC1/LOGINKEY/ae5a65ca-022f-4c81-89f2-7cacb2789888.dict"
2017-01-24 20:14 GMT+08:00 ShaoFeng Shi <sh...@apache.org>:
> @Yerui, do you have suggestion on this?
>
> 2017-01-24 18:19 GMT+08:00 市场中心-ZHANGDA32698 <zd...@ly.com>:
>
>> Hi ShaoFeng,
>>
>>
>>
>> Thanks for your suggestion. However since ours is a daily build, I guess
>> there’s no concurrent issue.
>>
>> The problem stops us from building subsequent segments. I try to remove
>> one of the Dicts from hbase and try a rebuild, but seems that they are
>> referenced at somewhere else and I got ‘java.lang.IllegalStateException:
>> No resource found at -- /dict/APPLYDATA_DSDCADMIN.FLY_
>> ZHUANGHUALV_PROC1/LOGINKEY/ae5a65ca-022f-4c81-89f2-7cacb2789888.dict’
>> problem at step 3 ‘Extract Fact Table Distinct Columns’. Of course I can
>> add the key back and let it continue, but without surprise it will stop at
>> setp4 ‘Build Dimension Dictionary’ with same error I encountered before
>> ‘GlobalDict xxxx should have 0 or 1 append dict but 2’
>>
>> How can I resolve this and continue my cube building? Any suggestion is
>> appreciated, thanks!
>>
>>
>>
>>
>>
>> *发件人:* ShaoFeng Shi [mailto:shaofengshi@apache.org]
>> *发送时间:* 2017年1月24日 17:01
>> *收件人:* user
>> *主题:* Re: more than 1 append dict for globalDict
>>
>>
>>
>> Hi zhangda,
>>
>>
>>
>> Do you have multiple segments (which has the count distinct measure)
>> building concurrently? In 1.6.0 and before there is a concurrent bug I
>> think; Yerui Sun fixed it in https://issues.apache.org/jira
>> /browse/KYLIN-2192
>>
>>
>>
>> So, please check whether it is this problem first; If yes, you need add
>> some control (lock) to avoid concurrent build for this cube.
>>
>>
>>
>> 2017-01-24 16:33 GMT+08:00 市场中心-ZHANGDA32698 <zd...@ly.com>:
>>
>> Hi there,
>>
>>
>>
>> I have a cube computing for some UV statistics. Since it requires global
>> count distinct operation, in the advanced dictionaries setting I put UV key
>> column’s builder class as ‘GlobalDictionaryBuilder’.
>>
>> Everyday build went on without any problem until yesterday there was a
>> ‘GlobalDict xxxx should have 0 or 1 append dict but 2"’ exception. I
>> checked the ‘kylin_metadata’ table in the hbase , and saw there were indeed
>> 2 dict
>>
>> '/dict/APPLYDATA_DSDCADMIN.FLY_ZHUANGHUALV_PROC1/LOGINKEY/
>> ae5a65ca-022f-4c81-89f2-7cacb2789888.dict'
>>
>> /dict/APPLYDATA_DSDCADMIN.FLY_ZHUANGHUALV_PROC1/LOGINKEY/ced
>> ce514-657d-4032-a591-7f01d984df65.dict'
>>
>> Fyi, APPLYDATA_DSDCADMIN.FLY_ZHUANGHUALV_PROC1 is the hive table name,
>> LOGINKEY is the count distinct key
>>
>> I’m not sure what had gone wrong, and I suppose global dict on 1 key
>> should be unique. Has anyone ever encountered similar error before, can
>> share some ideas? Thanks!
>>
>>
>>
>>
>>
>> --
>>
>> Best regards,
>>
>>
>>
>> Shaofeng Shi 史少锋
>>
>>
>>
>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
>
--
Best regards,
Shaofeng Shi 史少锋
Re: 答复: more than 1 append dict for globalDict
Posted by ShaoFeng Shi <sh...@apache.org>.
Yerui is on vacation;
If you're urgent, you can try this way (also confirmed with yerui);
1) backup your metadata to local folder;
2) copy to backup the two global dictionaries in some other folder;
3) among the two dictionaries, pick the bigger one (it should includes all
values in the smaller one), and then in the cube json, replace all the
references to smaller dict with the bigger one; You can edit the Cube json
from Kylin's web gui, in the "Admin" -> "Edit JSON";
4) remove the smaller one dict from kylin metadata, use kylin's command,
like : "./bin/metastore.sh remove /dict/APPLYDATA_DSDCADMIN.FLY_
ZHUANGHUALV_PROC1/LOGINKEY/ae5a65ca-022f-4c81-89f2-7cacb2789888.dict"
2017-01-24 20:14 GMT+08:00 ShaoFeng Shi <sh...@apache.org>:
> @Yerui, do you have suggestion on this?
>
> 2017-01-24 18:19 GMT+08:00 市场中心-ZHANGDA32698 <zd...@ly.com>:
>
>> Hi ShaoFeng,
>>
>>
>>
>> Thanks for your suggestion. However since ours is a daily build, I guess
>> there’s no concurrent issue.
>>
>> The problem stops us from building subsequent segments. I try to remove
>> one of the Dicts from hbase and try a rebuild, but seems that they are
>> referenced at somewhere else and I got ‘java.lang.IllegalStateException:
>> No resource found at -- /dict/APPLYDATA_DSDCADMIN.FLY_
>> ZHUANGHUALV_PROC1/LOGINKEY/ae5a65ca-022f-4c81-89f2-7cacb2789888.dict’
>> problem at step 3 ‘Extract Fact Table Distinct Columns’. Of course I can
>> add the key back and let it continue, but without surprise it will stop at
>> setp4 ‘Build Dimension Dictionary’ with same error I encountered before
>> ‘GlobalDict xxxx should have 0 or 1 append dict but 2’
>>
>> How can I resolve this and continue my cube building? Any suggestion is
>> appreciated, thanks!
>>
>>
>>
>>
>>
>> *发件人:* ShaoFeng Shi [mailto:shaofengshi@apache.org]
>> *发送时间:* 2017年1月24日 17:01
>> *收件人:* user
>> *主题:* Re: more than 1 append dict for globalDict
>>
>>
>>
>> Hi zhangda,
>>
>>
>>
>> Do you have multiple segments (which has the count distinct measure)
>> building concurrently? In 1.6.0 and before there is a concurrent bug I
>> think; Yerui Sun fixed it in https://issues.apache.org/jira
>> /browse/KYLIN-2192
>>
>>
>>
>> So, please check whether it is this problem first; If yes, you need add
>> some control (lock) to avoid concurrent build for this cube.
>>
>>
>>
>> 2017-01-24 16:33 GMT+08:00 市场中心-ZHANGDA32698 <zd...@ly.com>:
>>
>> Hi there,
>>
>>
>>
>> I have a cube computing for some UV statistics. Since it requires global
>> count distinct operation, in the advanced dictionaries setting I put UV key
>> column’s builder class as ‘GlobalDictionaryBuilder’.
>>
>> Everyday build went on without any problem until yesterday there was a
>> ‘GlobalDict xxxx should have 0 or 1 append dict but 2"’ exception. I
>> checked the ‘kylin_metadata’ table in the hbase , and saw there were indeed
>> 2 dict
>>
>> '/dict/APPLYDATA_DSDCADMIN.FLY_ZHUANGHUALV_PROC1/LOGINKEY/
>> ae5a65ca-022f-4c81-89f2-7cacb2789888.dict'
>>
>> /dict/APPLYDATA_DSDCADMIN.FLY_ZHUANGHUALV_PROC1/LOGINKEY/ced
>> ce514-657d-4032-a591-7f01d984df65.dict'
>>
>> Fyi, APPLYDATA_DSDCADMIN.FLY_ZHUANGHUALV_PROC1 is the hive table name,
>> LOGINKEY is the count distinct key
>>
>> I’m not sure what had gone wrong, and I suppose global dict on 1 key
>> should be unique. Has anyone ever encountered similar error before, can
>> share some ideas? Thanks!
>>
>>
>>
>>
>>
>> --
>>
>> Best regards,
>>
>>
>>
>> Shaofeng Shi 史少锋
>>
>>
>>
>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
>
--
Best regards,
Shaofeng Shi 史少锋