You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by "446463844@qq.com" <44...@qq.com> on 2017/01/05 15:21:40 UTC
where kylin global dict store?
Hi all :
I wan't to kown where kylin global dictionary stroe?
I found my cube infomation in HBase kylin table.for example:
in HBase console shell
get "kylin_metadata","/cube/dmp_user_info_cube.json"
i Get this (this data is formated )
-----------------------------------------------------------
{
"uuid": "f5db317d-32a9-4b14-8c56-837d3f9a3fae",
"last_modified": 1483607960443,
"version": "1.6.0",
"name": "dmp_user_info",
"owner": "ADMIN",
"descriptor": "dmp_user_info",
"cost": 50,
"status": "DISABLED",
"segments": [
{
"uuid": "9b187fdb-28ca-48f0-a0df-fe68cef05981",
"name": "20161001000000_20161220000000",
"storage_location_identifier": "KYLIN_3WV3VKHD99",
"date_range_start": 1475280000000,
"date_range_end": 1482192000000,
"source_offset_start": 0,
"source_offset_end": 0,
"status": "NEW",
"size_kb": 0,
"input_records": 0,
"input_records_size": 0,
"last_build_time": 0,
"last_build_job_id": null,
"create_time_utc": 1483607736083,
"cuboid_shard_nums": { },
"total_shards": 0,
"blackout_cuboids": [ ],
"binary_signature": null,
"dictionaries": {
"KYLINLABEL.USER_INFO/SEX": "/dict/KYLINLABEL.USER_INFO/SEX/e01c7973-7dd7-411a-9e6d-79613e4fb2c9.dict",
"KYLINLABEL.USER_INFO/BIRTHDAY": "/dict/KYLINLABEL.USER_INFO/BIRTHDAY/39270014-1c39-4d18-905d-43935d658a41.dict",
"KYLINLABEL.USER_INFO/NATIVE_PLACE": "/dict/KYLINLABEL.USER_INFO/NATIVE_PLACE/c964fe32-66e9-4fa3-9902-fad28167b46d.dict",
"KYLINLABEL.USER_INFO/LOCATION": "/dict/KYLINLABEL.USER_INFO/LOCATION/2155116f-6bd0-44cb-b14c- .dict"
},
"snapshots": null,
"index_path": null,
"rowkey_stats": [
[
"SEX",
3,
1
],
[
"BIRTHDAY",
9193,
2
],
[
"NATIVE_PLACE",
2268,
2
],
[
"LOCATION",
15,
1
]
]
}
],
"create_time_utc": 1483605652669,
"size_kb": 0,
"input_records_count": 0,
"input_records_size": 0
}
-------------------------------------
I find all thing begin with /cube or /cube_desc or /model_desc or /table or /table_ext related to my cube and I can't find anything with Global Dictionary with my cube
Help me !
446463844@qq.com
Re: where kylin global dict store?
Posted by Alberto Ramón <a....@gmail.com>.
(I answer to User Kylin Maillist)
Check this references, please:
http://kylin.apache.org/blog/2016/08/01/count-distinct-in-kylin/ there is
an exemple
"builder": "org.apache.kylin.dict.GlobalDictionaryBuilder"
and Kylin 1904 <https://issues.apache.org/jira/browse/KYLIN-1904>
BR, Alb
2017-01-05 16:21 GMT+01:00 446463844@qq.com <44...@qq.com>:
> Hi all :
> I wan't to kown where kylin global dictionary stroe?
> I found my cube infomation in HBase kylin table.for example:
>
> in HBase console shell
> get "kylin_metadata","/cube/dmp_user_info_cube.json"
> i Get this (this data is formated )
> -----------------------------------------------------------
> {
> "uuid": "f5db317d-32a9-4b14-8c56-837d3f9a3fae",
> "last_modified": 1483607960443,
> "version": "1.6.0",
> "name": "dmp_user_info",
> "owner": "ADMIN",
> "descriptor": "dmp_user_info",
> "cost": 50,
> "status": "DISABLED",
> "segments": [
> {
> "uuid": "9b187fdb-28ca-48f0-a0df-fe68cef05981",
> "name": "20161001000000_20161220000000",
> "storage_location_identifier": "KYLIN_3WV3VKHD99",
> "date_range_start": 1475280000000,
> "date_range_end": 1482192000000,
> "source_offset_start": 0,
> "source_offset_end": 0,
> "status": "NEW",
> "size_kb": 0,
> "input_records": 0,
> "input_records_size": 0,
> "last_build_time": 0,
> "last_build_job_id": null,
> "create_time_utc": 1483607736083,
> "cuboid_shard_nums": { },
> "total_shards": 0,
> "blackout_cuboids": [ ],
> "binary_signature": null,
> "dictionaries": {
> "KYLINLABEL.USER_INFO/SEX": "/dict/
> KYLINLABEL.USER_INFO/SEX/e01c7973-7dd7-411a-9e6d-79613e4fb2c9.dict",
> "KYLINLABEL.USER_INFO/BIRTHDAY": "/dict/
> KYLINLABEL.USER_INFO/BIRTHDAY/39270014-1c39-4d18-905d-43935d658a41.dict",
> "KYLINLABEL.USER_INFO/NATIVE_PLACE": "/
> dict/KYLINLABEL.USER_INFO/NATIVE_PLACE/c964fe32-66e9-
> 4fa3-9902-fad28167b46d.dict",
> "KYLINLABEL.USER_INFO/LOCATION": "/dict/
> KYLINLABEL.USER_INFO/LOCATION/2155116f-6bd0-44cb-b14c- .dict"
> },
> "snapshots": null,
> "index_path": null,
> "rowkey_stats": [
> [
> "SEX",
> 3,
> 1
> ],
> [
> "BIRTHDAY",
> 9193,
> 2
> ],
> [
> "NATIVE_PLACE",
> 2268,
> 2
> ],
> [
> "LOCATION",
> 15,
> 1
> ]
> ]
> }
> ],
> "create_time_utc": 1483605652669,
> "size_kb": 0,
> "input_records_count": 0,
> "input_records_size": 0
> }
> -------------------------------------
> I find all thing begin with /cube or /cube_desc or /model_desc or /table
> or /table_ext related to my cube and I can't find anything with Global
> Dictionary with my cube
> Help me !
>
> ------------------------------
> 446463844@qq.com
>
Re: Re: where kylin global dict store?
Posted by "446463844@qq.com" <44...@qq.com>.
I find the Globel Dictionary infomation in HBase table ,sorry for disturb
446463844@qq.com
From: 446463844@qq.com
Date: 2017-01-06 10:48
To: user; dev
CC: user
Subject: Re: Re: where kylin global dict store?
I am sorry for not making a point for my question
I used a Global Dictionary cloumn when I build a cube .but I can't find anything with this Global Dictionary infomation related with my cube in HBase table 'kylin_metadata'
I searched all rowkeys in HBase table 'kylin_metadata' and filter out rowkeys prefix with '/cube' ,'/model_desc' ,'/table' related with my cube name or my model name
Nothing with Global Dictionary infomation can be found.
than I searched HDFS files in directory /kylin/kylin_metadata and find nothing sadly.
I deleted the Global Dictionary rowkey in HBase table before and I resume the job to build this cube
in kylin webUI job throw Error tall me ' No resource found at -- /dict/KYLINLABEL.USER_INFO/ROWKEY/bbcf19eb-1af9-4b03-b64b-40a5d423bcd6.dict'
but /dict/KYLINLABEL.USER_INFO/ROWKEY/bbcf19eb-1af9-4b03-b64b-40a5d423bcd6.dict has been deleted by me.
finally I accoding to the stack infomation and find code in engine-mr module which is source code in kylin 1.6.0
in the class AbstracHadoopJob have a method
```
attachKylinPropsAndMatedata(CubeInstance cube,Configuration conf) throw Exception{
MetadataManager metaMgr = MetadataManager.getInstance(cube.getConfig());
// write cube / model_desc / cube_desc / dict / table
ArrayList<String> dumpList = new ArrayList<String>();
dumpList.add(cube.getResourcePath());
dumpList.add(cube.getDescriptor().getModel().getResourcePath());
dumpList.add(cube.getDescriptor().getResourcePath());
for (String tableName : cube.getDescriptor().getModel().getAllTables()) {
TableDesc table = metaMgr.getTableDesc(tableName);
dumpList.add(table.getResourcePath());
List<String> dependentResources = SourceFactory.getMRDependentResources(table);
dumpList.addAll(dependentResources);
}
for (CubeSegment segment : cube.getSegments()) {
dumpList.addAll(segment.getDictionaryPaths());
}
attachKylinPropsAndMetadata(dumpList, cube.getConfig(), conf);
}
```
the problem is happend with dumpList which contains Global Dictionary paths ,so I try to understand how dumpList be filled with data. thanks to your annotations .I searched all rowkeys in HBase table 'kylin_metadata'and filter out that which starts with prefix '/cube' or '/model_desc' or '/cube_desc' or '/table' and so on .but finally I could not find any infomation with my cube used Global Dictionary
I don't kown whether or not it clearly with my question?
446463844@qq.com
From: ShaoFeng Shi
Date: 2017-01-06 09:53
To: dev
CC: user
Subject: Re: where kylin global dict store?
Kylin metadata is stored in HBase by default; but if a file is too large, it will be saved in HDFS in the "/kylin/kylin_metadata" folder; You can do a search there.
2017-01-06 8:52 GMT+08:00 zy <34...@qq.com>:
I think you could find the more content about kylin from the internet..i give you as follow..
http://blog.csdn.net/yu616568/article/details/50570536
http://www.mamicode.com/info-detail-1015006.html
http://www.chinahadoop.cn/search?q=kylin
------------------ Original ------------------
From: "446463844@qq.com"<44...@qq.com>;
Date: 2017年1月5日(星期四) 晚上11:21
To: "dev"<de...@kylin.apache.org>; "user"<us...@hive.apache.org>;
Subject: where kylin global dict store?
Hi all :
I wan't to kown where kylin global dictionary stroe?
I found my cube infomation in HBase kylin table.for example:
in HBase console shell
get "kylin_metadata","/cube/dmp_user_info_cube.json"
i Get this (this data is formated )
-----------------------------------------------------------
{
"uuid": "f5db317d-32a9-4b14-8c56-837d3f9a3fae",
"last_modified": 1483607960443,
"version": "1.6.0",
"name": "dmp_user_info",
"owner": "ADMIN",
"descriptor": "dmp_user_info",
"cost": 50,
"status": "DISABLED",
"segments": [
{
"uuid": "9b187fdb-28ca-48f0-a0df-fe68cef05981",
"name": "20161001000000_20161220000000",
"storage_location_identifier": "KYLIN_3WV3VKHD99",
"date_range_start": 1475280000000,
"date_range_end": 1482192000000,
"source_offset_start": 0,
"source_offset_end": 0,
"status": "NEW",
"size_kb": 0,
"input_records": 0,
"input_records_size": 0,
"last_build_time": 0,
"last_build_job_id": null,
"create_time_utc": 1483607736083,
"cuboid_shard_nums": { },
"total_shards": 0,
"blackout_cuboids": [ ],
"binary_signature": null,
"dictionaries": {
"KYLINLABEL.USER_INFO/SEX": "/dict/KYLINLABEL.USER_INFO/SEX/e01c7973-7dd7-411a-9e6d-79613e4fb2c9.dict",
"KYLINLABEL.USER_INFO/BIRTHDAY": "/dict/KYLINLABEL.USER_INFO/BIRTHDAY/39270014-1c39-4d18-905d-43935d658a41.dict",
"KYLINLABEL.USER_INFO/NATIVE_PLACE": "/dict/KYLINLABEL.USER_INFO/NATIVE_PLACE/c964fe32-66e9-4fa3-9902-fad28167b46d.dict",
"KYLINLABEL.USER_INFO/LOCATION": "/dict/KYLINLABEL.USER_INFO/LOCATION/2155116f-6bd0-44cb-b14c- .dict"
},
"snapshots": null,
"index_path": null,
"rowkey_stats": [
[
"SEX",
3,
1
],
[
"BIRTHDAY",
9193,
2
],
[
"NATIVE_PLACE",
2268,
2
],
[
"LOCATION",
15,
1
]
]
}
],
"create_time_utc": 1483605652669,
"size_kb": 0,
"input_records_count": 0,
"input_records_size": 0
}
-------------------------------------
I find all thing begin with /cube or /cube_desc or /model_desc or /table or /table_ext related to my cube and I can't find anything with Global Dictionary with my cube
Help me !
446463844@qq.com
--
Best regards,
Shaofeng Shi 史少锋
Re: Re: where kylin global dict store?
Posted by "446463844@qq.com" <44...@qq.com>.
I find the Globel Dictionary infomation in HBase table ,sorry for disturb
446463844@qq.com
From: 446463844@qq.com
Date: 2017-01-06 10:48
To: user; dev
CC: user
Subject: Re: Re: where kylin global dict store?
I am sorry for not making a point for my question
I used a Global Dictionary cloumn when I build a cube .but I can't find anything with this Global Dictionary infomation related with my cube in HBase table 'kylin_metadata'
I searched all rowkeys in HBase table 'kylin_metadata' and filter out rowkeys prefix with '/cube' ,'/model_desc' ,'/table' related with my cube name or my model name
Nothing with Global Dictionary infomation can be found.
than I searched HDFS files in directory /kylin/kylin_metadata and find nothing sadly.
I deleted the Global Dictionary rowkey in HBase table before and I resume the job to build this cube
in kylin webUI job throw Error tall me ' No resource found at -- /dict/KYLINLABEL.USER_INFO/ROWKEY/bbcf19eb-1af9-4b03-b64b-40a5d423bcd6.dict'
but /dict/KYLINLABEL.USER_INFO/ROWKEY/bbcf19eb-1af9-4b03-b64b-40a5d423bcd6.dict has been deleted by me.
finally I accoding to the stack infomation and find code in engine-mr module which is source code in kylin 1.6.0
in the class AbstracHadoopJob have a method
```
attachKylinPropsAndMatedata(CubeInstance cube,Configuration conf) throw Exception{
MetadataManager metaMgr = MetadataManager.getInstance(cube.getConfig());
// write cube / model_desc / cube_desc / dict / table
ArrayList<String> dumpList = new ArrayList<String>();
dumpList.add(cube.getResourcePath());
dumpList.add(cube.getDescriptor().getModel().getResourcePath());
dumpList.add(cube.getDescriptor().getResourcePath());
for (String tableName : cube.getDescriptor().getModel().getAllTables()) {
TableDesc table = metaMgr.getTableDesc(tableName);
dumpList.add(table.getResourcePath());
List<String> dependentResources = SourceFactory.getMRDependentResources(table);
dumpList.addAll(dependentResources);
}
for (CubeSegment segment : cube.getSegments()) {
dumpList.addAll(segment.getDictionaryPaths());
}
attachKylinPropsAndMetadata(dumpList, cube.getConfig(), conf);
}
```
the problem is happend with dumpList which contains Global Dictionary paths ,so I try to understand how dumpList be filled with data. thanks to your annotations .I searched all rowkeys in HBase table 'kylin_metadata'and filter out that which starts with prefix '/cube' or '/model_desc' or '/cube_desc' or '/table' and so on .but finally I could not find any infomation with my cube used Global Dictionary
I don't kown whether or not it clearly with my question?
446463844@qq.com
From: ShaoFeng Shi
Date: 2017-01-06 09:53
To: dev
CC: user
Subject: Re: where kylin global dict store?
Kylin metadata is stored in HBase by default; but if a file is too large, it will be saved in HDFS in the "/kylin/kylin_metadata" folder; You can do a search there.
2017-01-06 8:52 GMT+08:00 zy <34...@qq.com>:
I think you could find the more content about kylin from the internet..i give you as follow..
http://blog.csdn.net/yu616568/article/details/50570536
http://www.mamicode.com/info-detail-1015006.html
http://www.chinahadoop.cn/search?q=kylin
------------------ Original ------------------
From: "446463844@qq.com"<44...@qq.com>;
Date: 2017年1月5日(星期四) 晚上11:21
To: "dev"<de...@kylin.apache.org>; "user"<us...@hive.apache.org>;
Subject: where kylin global dict store?
Hi all :
I wan't to kown where kylin global dictionary stroe?
I found my cube infomation in HBase kylin table.for example:
in HBase console shell
get "kylin_metadata","/cube/dmp_user_info_cube.json"
i Get this (this data is formated )
-----------------------------------------------------------
{
"uuid": "f5db317d-32a9-4b14-8c56-837d3f9a3fae",
"last_modified": 1483607960443,
"version": "1.6.0",
"name": "dmp_user_info",
"owner": "ADMIN",
"descriptor": "dmp_user_info",
"cost": 50,
"status": "DISABLED",
"segments": [
{
"uuid": "9b187fdb-28ca-48f0-a0df-fe68cef05981",
"name": "20161001000000_20161220000000",
"storage_location_identifier": "KYLIN_3WV3VKHD99",
"date_range_start": 1475280000000,
"date_range_end": 1482192000000,
"source_offset_start": 0,
"source_offset_end": 0,
"status": "NEW",
"size_kb": 0,
"input_records": 0,
"input_records_size": 0,
"last_build_time": 0,
"last_build_job_id": null,
"create_time_utc": 1483607736083,
"cuboid_shard_nums": { },
"total_shards": 0,
"blackout_cuboids": [ ],
"binary_signature": null,
"dictionaries": {
"KYLINLABEL.USER_INFO/SEX": "/dict/KYLINLABEL.USER_INFO/SEX/e01c7973-7dd7-411a-9e6d-79613e4fb2c9.dict",
"KYLINLABEL.USER_INFO/BIRTHDAY": "/dict/KYLINLABEL.USER_INFO/BIRTHDAY/39270014-1c39-4d18-905d-43935d658a41.dict",
"KYLINLABEL.USER_INFO/NATIVE_PLACE": "/dict/KYLINLABEL.USER_INFO/NATIVE_PLACE/c964fe32-66e9-4fa3-9902-fad28167b46d.dict",
"KYLINLABEL.USER_INFO/LOCATION": "/dict/KYLINLABEL.USER_INFO/LOCATION/2155116f-6bd0-44cb-b14c- .dict"
},
"snapshots": null,
"index_path": null,
"rowkey_stats": [
[
"SEX",
3,
1
],
[
"BIRTHDAY",
9193,
2
],
[
"NATIVE_PLACE",
2268,
2
],
[
"LOCATION",
15,
1
]
]
}
],
"create_time_utc": 1483605652669,
"size_kb": 0,
"input_records_count": 0,
"input_records_size": 0
}
-------------------------------------
I find all thing begin with /cube or /cube_desc or /model_desc or /table or /table_ext related to my cube and I can't find anything with Global Dictionary with my cube
Help me !
446463844@qq.com
--
Best regards,
Shaofeng Shi 史少锋
Re: Re: where kylin global dict store?
Posted by "446463844@qq.com" <44...@qq.com>.
I am sorry for not making a point for my question
I used a Global Dictionary cloumn when I build a cube .but I can't find anything with this Global Dictionary infomation related with my cube in HBase table 'kylin_metadata'
I searched all rowkeys in HBase table 'kylin_metadata' and filter out rowkeys prefix with '/cube' ,'/model_desc' ,'/table' related with my cube name or my model name
Nothing with Global Dictionary infomation can be found.
than I searched HDFS files in directory /kylin/kylin_metadata and find nothing sadly.
I deleted the Global Dictionary rowkey in HBase table before and I resume the job to build this cube
in kylin webUI job throw Error tall me ' No resource found at -- /dict/KYLINLABEL.USER_INFO/ROWKEY/bbcf19eb-1af9-4b03-b64b-40a5d423bcd6.dict'
but /dict/KYLINLABEL.USER_INFO/ROWKEY/bbcf19eb-1af9-4b03-b64b-40a5d423bcd6.dict has been deleted by me.
finally I accoding to the stack infomation and find code in engine-mr module which is source code in kylin 1.6.0
in the class AbstracHadoopJob have a method
```
attachKylinPropsAndMatedata(CubeInstance cube,Configuration conf) throw Exception{
MetadataManager metaMgr = MetadataManager.getInstance(cube.getConfig());
// write cube / model_desc / cube_desc / dict / table
ArrayList<String> dumpList = new ArrayList<String>();
dumpList.add(cube.getResourcePath());
dumpList.add(cube.getDescriptor().getModel().getResourcePath());
dumpList.add(cube.getDescriptor().getResourcePath());
for (String tableName : cube.getDescriptor().getModel().getAllTables()) {
TableDesc table = metaMgr.getTableDesc(tableName);
dumpList.add(table.getResourcePath());
List<String> dependentResources = SourceFactory.getMRDependentResources(table);
dumpList.addAll(dependentResources);
}
for (CubeSegment segment : cube.getSegments()) {
dumpList.addAll(segment.getDictionaryPaths());
}
attachKylinPropsAndMetadata(dumpList, cube.getConfig(), conf);
}
```
the problem is happend with dumpList which contains Global Dictionary paths ,so I try to understand how dumpList be filled with data. thanks to your annotations .I searched all rowkeys in HBase table 'kylin_metadata'and filter out that which starts with prefix '/cube' or '/model_desc' or '/cube_desc' or '/table' and so on .but finally I could not find any infomation with my cube used Global Dictionary
I don't kown whether or not it clearly with my question?
446463844@qq.com
From: ShaoFeng Shi
Date: 2017-01-06 09:53
To: dev
CC: user
Subject: Re: where kylin global dict store?
Kylin metadata is stored in HBase by default; but if a file is too large, it will be saved in HDFS in the "/kylin/kylin_metadata" folder; You can do a search there.
2017-01-06 8:52 GMT+08:00 zy <34...@qq.com>:
I think you could find the more content about kylin from the internet..i give you as follow..
http://blog.csdn.net/yu616568/article/details/50570536
http://www.mamicode.com/info-detail-1015006.html
http://www.chinahadoop.cn/search?q=kylin
------------------ Original ------------------
From: "446463844@qq.com"<44...@qq.com>;
Date: 2017年1月5日(星期四) 晚上11:21
To: "dev"<de...@kylin.apache.org>; "user"<us...@hive.apache.org>;
Subject: where kylin global dict store?
Hi all :
I wan't to kown where kylin global dictionary stroe?
I found my cube infomation in HBase kylin table.for example:
in HBase console shell
get "kylin_metadata","/cube/dmp_user_info_cube.json"
i Get this (this data is formated )
-----------------------------------------------------------
{
"uuid": "f5db317d-32a9-4b14-8c56-837d3f9a3fae",
"last_modified": 1483607960443,
"version": "1.6.0",
"name": "dmp_user_info",
"owner": "ADMIN",
"descriptor": "dmp_user_info",
"cost": 50,
"status": "DISABLED",
"segments": [
{
"uuid": "9b187fdb-28ca-48f0-a0df-fe68cef05981",
"name": "20161001000000_20161220000000",
"storage_location_identifier": "KYLIN_3WV3VKHD99",
"date_range_start": 1475280000000,
"date_range_end": 1482192000000,
"source_offset_start": 0,
"source_offset_end": 0,
"status": "NEW",
"size_kb": 0,
"input_records": 0,
"input_records_size": 0,
"last_build_time": 0,
"last_build_job_id": null,
"create_time_utc": 1483607736083,
"cuboid_shard_nums": { },
"total_shards": 0,
"blackout_cuboids": [ ],
"binary_signature": null,
"dictionaries": {
"KYLINLABEL.USER_INFO/SEX": "/dict/KYLINLABEL.USER_INFO/SEX/e01c7973-7dd7-411a-9e6d-79613e4fb2c9.dict",
"KYLINLABEL.USER_INFO/BIRTHDAY": "/dict/KYLINLABEL.USER_INFO/BIRTHDAY/39270014-1c39-4d18-905d-43935d658a41.dict",
"KYLINLABEL.USER_INFO/NATIVE_PLACE": "/dict/KYLINLABEL.USER_INFO/NATIVE_PLACE/c964fe32-66e9-4fa3-9902-fad28167b46d.dict",
"KYLINLABEL.USER_INFO/LOCATION": "/dict/KYLINLABEL.USER_INFO/LOCATION/2155116f-6bd0-44cb-b14c- .dict"
},
"snapshots": null,
"index_path": null,
"rowkey_stats": [
[
"SEX",
3,
1
],
[
"BIRTHDAY",
9193,
2
],
[
"NATIVE_PLACE",
2268,
2
],
[
"LOCATION",
15,
1
]
]
}
],
"create_time_utc": 1483605652669,
"size_kb": 0,
"input_records_count": 0,
"input_records_size": 0
}
-------------------------------------
I find all thing begin with /cube or /cube_desc or /model_desc or /table or /table_ext related to my cube and I can't find anything with Global Dictionary with my cube
Help me !
446463844@qq.com
--
Best regards,
Shaofeng Shi 史少锋
Re: Re: where kylin global dict store?
Posted by "446463844@qq.com" <44...@qq.com>.
I am sorry for not making a point for my question
I used a Global Dictionary cloumn when I build a cube .but I can't find anything with this Global Dictionary infomation related with my cube in HBase table 'kylin_metadata'
I searched all rowkeys in HBase table 'kylin_metadata' and filter out rowkeys prefix with '/cube' ,'/model_desc' ,'/table' related with my cube name or my model name
Nothing with Global Dictionary infomation can be found.
than I searched HDFS files in directory /kylin/kylin_metadata and find nothing sadly.
I deleted the Global Dictionary rowkey in HBase table before and I resume the job to build this cube
in kylin webUI job throw Error tall me ' No resource found at -- /dict/KYLINLABEL.USER_INFO/ROWKEY/bbcf19eb-1af9-4b03-b64b-40a5d423bcd6.dict'
but /dict/KYLINLABEL.USER_INFO/ROWKEY/bbcf19eb-1af9-4b03-b64b-40a5d423bcd6.dict has been deleted by me.
finally I accoding to the stack infomation and find code in engine-mr module which is source code in kylin 1.6.0
in the class AbstracHadoopJob have a method
```
attachKylinPropsAndMatedata(CubeInstance cube,Configuration conf) throw Exception{
MetadataManager metaMgr = MetadataManager.getInstance(cube.getConfig());
// write cube / model_desc / cube_desc / dict / table
ArrayList<String> dumpList = new ArrayList<String>();
dumpList.add(cube.getResourcePath());
dumpList.add(cube.getDescriptor().getModel().getResourcePath());
dumpList.add(cube.getDescriptor().getResourcePath());
for (String tableName : cube.getDescriptor().getModel().getAllTables()) {
TableDesc table = metaMgr.getTableDesc(tableName);
dumpList.add(table.getResourcePath());
List<String> dependentResources = SourceFactory.getMRDependentResources(table);
dumpList.addAll(dependentResources);
}
for (CubeSegment segment : cube.getSegments()) {
dumpList.addAll(segment.getDictionaryPaths());
}
attachKylinPropsAndMetadata(dumpList, cube.getConfig(), conf);
}
```
the problem is happend with dumpList which contains Global Dictionary paths ,so I try to understand how dumpList be filled with data. thanks to your annotations .I searched all rowkeys in HBase table 'kylin_metadata'and filter out that which starts with prefix '/cube' or '/model_desc' or '/cube_desc' or '/table' and so on .but finally I could not find any infomation with my cube used Global Dictionary
I don't kown whether or not it clearly with my question?
446463844@qq.com
From: ShaoFeng Shi
Date: 2017-01-06 09:53
To: dev
CC: user
Subject: Re: where kylin global dict store?
Kylin metadata is stored in HBase by default; but if a file is too large, it will be saved in HDFS in the "/kylin/kylin_metadata" folder; You can do a search there.
2017-01-06 8:52 GMT+08:00 zy <34...@qq.com>:
I think you could find the more content about kylin from the internet..i give you as follow..
http://blog.csdn.net/yu616568/article/details/50570536
http://www.mamicode.com/info-detail-1015006.html
http://www.chinahadoop.cn/search?q=kylin
------------------ Original ------------------
From: "446463844@qq.com"<44...@qq.com>;
Date: 2017年1月5日(星期四) 晚上11:21
To: "dev"<de...@kylin.apache.org>; "user"<us...@hive.apache.org>;
Subject: where kylin global dict store?
Hi all :
I wan't to kown where kylin global dictionary stroe?
I found my cube infomation in HBase kylin table.for example:
in HBase console shell
get "kylin_metadata","/cube/dmp_user_info_cube.json"
i Get this (this data is formated )
-----------------------------------------------------------
{
"uuid": "f5db317d-32a9-4b14-8c56-837d3f9a3fae",
"last_modified": 1483607960443,
"version": "1.6.0",
"name": "dmp_user_info",
"owner": "ADMIN",
"descriptor": "dmp_user_info",
"cost": 50,
"status": "DISABLED",
"segments": [
{
"uuid": "9b187fdb-28ca-48f0-a0df-fe68cef05981",
"name": "20161001000000_20161220000000",
"storage_location_identifier": "KYLIN_3WV3VKHD99",
"date_range_start": 1475280000000,
"date_range_end": 1482192000000,
"source_offset_start": 0,
"source_offset_end": 0,
"status": "NEW",
"size_kb": 0,
"input_records": 0,
"input_records_size": 0,
"last_build_time": 0,
"last_build_job_id": null,
"create_time_utc": 1483607736083,
"cuboid_shard_nums": { },
"total_shards": 0,
"blackout_cuboids": [ ],
"binary_signature": null,
"dictionaries": {
"KYLINLABEL.USER_INFO/SEX": "/dict/KYLINLABEL.USER_INFO/SEX/e01c7973-7dd7-411a-9e6d-79613e4fb2c9.dict",
"KYLINLABEL.USER_INFO/BIRTHDAY": "/dict/KYLINLABEL.USER_INFO/BIRTHDAY/39270014-1c39-4d18-905d-43935d658a41.dict",
"KYLINLABEL.USER_INFO/NATIVE_PLACE": "/dict/KYLINLABEL.USER_INFO/NATIVE_PLACE/c964fe32-66e9-4fa3-9902-fad28167b46d.dict",
"KYLINLABEL.USER_INFO/LOCATION": "/dict/KYLINLABEL.USER_INFO/LOCATION/2155116f-6bd0-44cb-b14c- .dict"
},
"snapshots": null,
"index_path": null,
"rowkey_stats": [
[
"SEX",
3,
1
],
[
"BIRTHDAY",
9193,
2
],
[
"NATIVE_PLACE",
2268,
2
],
[
"LOCATION",
15,
1
]
]
}
],
"create_time_utc": 1483605652669,
"size_kb": 0,
"input_records_count": 0,
"input_records_size": 0
}
-------------------------------------
I find all thing begin with /cube or /cube_desc or /model_desc or /table or /table_ext related to my cube and I can't find anything with Global Dictionary with my cube
Help me !
446463844@qq.com
--
Best regards,
Shaofeng Shi 史少锋
Re: where kylin global dict store?
Posted by ShaoFeng Shi <sh...@apache.org>.
Kylin metadata is stored in HBase by default; but if a file is too large,
it will be saved in HDFS in the "/kylin/kylin_metadata" folder; You can do
a search there.
2017-01-06 8:52 GMT+08:00 zy <34...@qq.com>:
> I think you could find the more content about kylin from the internet..i
> give you as follow..
> http://blog.csdn.net/yu616568/article/details/50570536
> http://www.mamicode.com/info-detail-1015006.html
> http://www.chinahadoop.cn/search?q=kylin
>
>
>
>
> ------------------ Original ------------------
> From: "446463844@qq.com"<44...@qq.com>;
> Date: 2017年1月5日(星期四) 晚上11:21
> To: "dev"<de...@kylin.apache.org>; "user"<us...@hive.apache.org>;
> Subject: where kylin global dict store?
>
>
>
> Hi all :
> I wan't to kown where kylin global dictionary stroe?
> I found my cube infomation in HBase kylin table.for example:
>
> in HBase console shell
> get "kylin_metadata","/cube/dmp_user_info_cube.json"
> i Get this (this data is formated )
> -----------------------------------------------------------
> {
> "uuid": "f5db317d-32a9-4b14-8c56-837d3f9a3fae",
> "last_modified": 1483607960443,
> "version": "1.6.0",
> "name": "dmp_user_info",
> "owner": "ADMIN",
> "descriptor": "dmp_user_info",
> "cost": 50,
> "status": "DISABLED",
> "segments": [
> {
> "uuid": "9b187fdb-28ca-48f0-a0df-fe68cef05981",
> "name": "20161001000000_20161220000000",
> "storage_location_identifier": "KYLIN_3WV3VKHD99",
> "date_range_start": 1475280000000,
> "date_range_end": 1482192000000,
> "source_offset_start": 0,
> "source_offset_end": 0,
> "status": "NEW",
> "size_kb": 0,
> "input_records": 0,
> "input_records_size": 0,
> "last_build_time": 0,
> "last_build_job_id": null,
> "create_time_utc": 1483607736083,
> "cuboid_shard_nums": { },
> "total_shards": 0,
> "blackout_cuboids": [ ],
> "binary_signature": null,
> "dictionaries": {
> "KYLINLABEL.USER_INFO/SEX": "/dict/KYLINLABEL.USER_INFO/
> SEX/e01c7973-7dd7-411a-9e6d-79613e4fb2c9.dict",
> "KYLINLABEL.USER_INFO/BIRTHDAY":
> "/dict/KYLINLABEL.USER_INFO/BIRTHDAY/39270014-1c39-4d18-
> 905d-43935d658a41.dict",
> "KYLINLABEL.USER_INFO/NATIVE_PLACE":
> "/dict/KYLINLABEL.USER_INFO/NATIVE_PLACE/c964fe32-66e9-
> 4fa3-9902-fad28167b46d.dict",
> "KYLINLABEL.USER_INFO/LOCATION":
> "/dict/KYLINLABEL.USER_INFO/LOCATION/2155116f-6bd0-44cb-b14c- .dict"
> },
> "snapshots": null,
> "index_path": null,
> "rowkey_stats": [
> [
> "SEX",
> 3,
> 1
> ],
> [
> "BIRTHDAY",
> 9193,
> 2
> ],
> [
> "NATIVE_PLACE",
> 2268,
> 2
> ],
> [
> "LOCATION",
> 15,
> 1
> ]
> ]
> }
> ],
> "create_time_utc": 1483605652669,
> "size_kb": 0,
> "input_records_count": 0,
> "input_records_size": 0
> }
> -------------------------------------
> I find all thing begin with /cube or /cube_desc or /model_desc or /table
> or /table_ext related to my cube and I can't find anything with Global
> Dictionary with my cube
> Help me !
>
>
>
> 446463844@qq.com
>
--
Best regards,
Shaofeng Shi 史少锋
Re: where kylin global dict store?
Posted by ShaoFeng Shi <sh...@apache.org>.
Kylin metadata is stored in HBase by default; but if a file is too large,
it will be saved in HDFS in the "/kylin/kylin_metadata" folder; You can do
a search there.
2017-01-06 8:52 GMT+08:00 zy <34...@qq.com>:
> I think you could find the more content about kylin from the internet..i
> give you as follow..
> http://blog.csdn.net/yu616568/article/details/50570536
> http://www.mamicode.com/info-detail-1015006.html
> http://www.chinahadoop.cn/search?q=kylin
>
>
>
>
> ------------------ Original ------------------
> From: "446463844@qq.com"<44...@qq.com>;
> Date: 2017年1月5日(星期四) 晚上11:21
> To: "dev"<de...@kylin.apache.org>; "user"<us...@hive.apache.org>;
> Subject: where kylin global dict store?
>
>
>
> Hi all :
> I wan't to kown where kylin global dictionary stroe?
> I found my cube infomation in HBase kylin table.for example:
>
> in HBase console shell
> get "kylin_metadata","/cube/dmp_user_info_cube.json"
> i Get this (this data is formated )
> -----------------------------------------------------------
> {
> "uuid": "f5db317d-32a9-4b14-8c56-837d3f9a3fae",
> "last_modified": 1483607960443,
> "version": "1.6.0",
> "name": "dmp_user_info",
> "owner": "ADMIN",
> "descriptor": "dmp_user_info",
> "cost": 50,
> "status": "DISABLED",
> "segments": [
> {
> "uuid": "9b187fdb-28ca-48f0-a0df-fe68cef05981",
> "name": "20161001000000_20161220000000",
> "storage_location_identifier": "KYLIN_3WV3VKHD99",
> "date_range_start": 1475280000000,
> "date_range_end": 1482192000000,
> "source_offset_start": 0,
> "source_offset_end": 0,
> "status": "NEW",
> "size_kb": 0,
> "input_records": 0,
> "input_records_size": 0,
> "last_build_time": 0,
> "last_build_job_id": null,
> "create_time_utc": 1483607736083,
> "cuboid_shard_nums": { },
> "total_shards": 0,
> "blackout_cuboids": [ ],
> "binary_signature": null,
> "dictionaries": {
> "KYLINLABEL.USER_INFO/SEX": "/dict/KYLINLABEL.USER_INFO/
> SEX/e01c7973-7dd7-411a-9e6d-79613e4fb2c9.dict",
> "KYLINLABEL.USER_INFO/BIRTHDAY":
> "/dict/KYLINLABEL.USER_INFO/BIRTHDAY/39270014-1c39-4d18-
> 905d-43935d658a41.dict",
> "KYLINLABEL.USER_INFO/NATIVE_PLACE":
> "/dict/KYLINLABEL.USER_INFO/NATIVE_PLACE/c964fe32-66e9-
> 4fa3-9902-fad28167b46d.dict",
> "KYLINLABEL.USER_INFO/LOCATION":
> "/dict/KYLINLABEL.USER_INFO/LOCATION/2155116f-6bd0-44cb-b14c- .dict"
> },
> "snapshots": null,
> "index_path": null,
> "rowkey_stats": [
> [
> "SEX",
> 3,
> 1
> ],
> [
> "BIRTHDAY",
> 9193,
> 2
> ],
> [
> "NATIVE_PLACE",
> 2268,
> 2
> ],
> [
> "LOCATION",
> 15,
> 1
> ]
> ]
> }
> ],
> "create_time_utc": 1483605652669,
> "size_kb": 0,
> "input_records_count": 0,
> "input_records_size": 0
> }
> -------------------------------------
> I find all thing begin with /cube or /cube_desc or /model_desc or /table
> or /table_ext related to my cube and I can't find anything with Global
> Dictionary with my cube
> Help me !
>
>
>
> 446463844@qq.com
>
--
Best regards,
Shaofeng Shi 史少锋
Re:where kylin global dict store?
Posted by zy <34...@qq.com>.
I think you could find the more content about kylin from the internet..i give you as follow..
http://blog.csdn.net/yu616568/article/details/50570536
http://www.mamicode.com/info-detail-1015006.html
http://www.chinahadoop.cn/search?q=kylin
------------------ Original ------------------
From: "446463844@qq.com"<44...@qq.com>;
Date: 2017年1月5日(星期四) 晚上11:21
To: "dev"<de...@kylin.apache.org>; "user"<us...@hive.apache.org>;
Subject: where kylin global dict store?
Hi all :
I wan't to kown where kylin global dictionary stroe?
I found my cube infomation in HBase kylin table.for example:
in HBase console shell
get "kylin_metadata","/cube/dmp_user_info_cube.json"
i Get this (this data is formated )
-----------------------------------------------------------
{
"uuid": "f5db317d-32a9-4b14-8c56-837d3f9a3fae",
"last_modified": 1483607960443,
"version": "1.6.0",
"name": "dmp_user_info",
"owner": "ADMIN",
"descriptor": "dmp_user_info",
"cost": 50,
"status": "DISABLED",
"segments": [
{
"uuid": "9b187fdb-28ca-48f0-a0df-fe68cef05981",
"name": "20161001000000_20161220000000",
"storage_location_identifier": "KYLIN_3WV3VKHD99",
"date_range_start": 1475280000000,
"date_range_end": 1482192000000,
"source_offset_start": 0,
"source_offset_end": 0,
"status": "NEW",
"size_kb": 0,
"input_records": 0,
"input_records_size": 0,
"last_build_time": 0,
"last_build_job_id": null,
"create_time_utc": 1483607736083,
"cuboid_shard_nums": { },
"total_shards": 0,
"blackout_cuboids": [ ],
"binary_signature": null,
"dictionaries": {
"KYLINLABEL.USER_INFO/SEX": "/dict/KYLINLABEL.USER_INFO/SEX/e01c7973-7dd7-411a-9e6d-79613e4fb2c9.dict",
"KYLINLABEL.USER_INFO/BIRTHDAY": "/dict/KYLINLABEL.USER_INFO/BIRTHDAY/39270014-1c39-4d18-905d-43935d658a41.dict",
"KYLINLABEL.USER_INFO/NATIVE_PLACE": "/dict/KYLINLABEL.USER_INFO/NATIVE_PLACE/c964fe32-66e9-4fa3-9902-fad28167b46d.dict",
"KYLINLABEL.USER_INFO/LOCATION": "/dict/KYLINLABEL.USER_INFO/LOCATION/2155116f-6bd0-44cb-b14c- .dict"
},
"snapshots": null,
"index_path": null,
"rowkey_stats": [
[
"SEX",
3,
1
],
[
"BIRTHDAY",
9193,
2
],
[
"NATIVE_PLACE",
2268,
2
],
[
"LOCATION",
15,
1
]
]
}
],
"create_time_utc": 1483605652669,
"size_kb": 0,
"input_records_count": 0,
"input_records_size": 0
}
-------------------------------------
I find all thing begin with /cube or /cube_desc or /model_desc or /table or /table_ext related to my cube and I can't find anything with Global Dictionary with my cube
Help me !
446463844@qq.com