You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@kylin.apache.org by jyzheng <jy...@iflytek.com> on 2016/05/09 09:12:53 UTC
kylin query failed
When I send sql query to Kylin like this:
select app_name, count(distinct uid) as uv
from sdk_log
left join week_calendar on sdk_log.day_time = week_calendar.week_cal
where sdk_log.day_time = date '2016-05-01'
group by app_name
order by uv desc
limit 200
and failed. So I have to restart Kylin.
The dimension `app_name` is a great high cardinality. If I switch to
`app_tag` dimension , it will return the right result. So I can’t get the
top measure dimension in Kylin? If can, and what can I do to solve this?
--------------------------------
郑江雨 云平台
Phone: 15155195496
Re: 答复: kylin query failed
Posted by Li Yang <li...@apache.org>.
My best guess is out-of-memory crash. Returning 30K HLL counter may take
200MB~2GB memory depending on the count distinct precision. Suggest
increase Kylin JVM heap and try again.
On Tue, May 17, 2016 at 10:16 AM, jyzheng <jy...@iflytek.com> wrote:
> 1. The cardinality of app_name is about 30,000.
>
> 2. Yes, the Kylin has crashed after query failed.
>
> 3. The clue of kylin.log in the given below:
>
>
>
> Thanks for attention!
>
>
>
>
>
> 2016-05-17 10:13:51,531 INFO [pool-4-thread-1]
> threadpool.DefaultScheduler:106 : Job Fetcher: 0 running, 0 actual running,
> 0 ready, 17 others
>
> 2016-05-17 10:14:20,031 DEBUG [http-bio-7070-exec-5]
> service.AdminService:90 : Get Kylin Runtime Config
>
> 2016-05-17 10:14:20,037 DEBUG [http-bio-7070-exec-8]
> controller.UserController:64 : authentication.getPrincipal() is
> org.springframework.security.core.userdetails.User@3b40b2f: Username:
> ADMIN; Password: [PROTECTED]; Enabled: true; AccountNonExpired: true;
> credentialsNonExpired: true; AccountNonLocked: true; Granted Authorities:
> ROLE_ADMIN,ROLE_ANALYST,ROLE_MODELER
>
> 2016-05-17 10:14:20,321 DEBUG [http-bio-7070-exec-5]
> controller.ProjectController:97 : authentication.getPrincipal() is
> org.springframework.security.core.userdetails.User@3b40b2f: Username:
> ADMIN; Password: [PROTECTED]; Enabled: true; AccountNonExpired: true;
> credentialsNonExpired: true; AccountNonLocked: true; Granted Authorities:
> ROLE_ADMIN,ROLE_ANALYST,ROLE_MODELER
>
> 2016-05-17 10:14:25,467 DEBUG [http-bio-7070-exec-5]
> controller.UserController:64 : authentication.getPrincipal() is
> org.springframework.security.core.userdetails.User@3b40b2f: Username:
> ADMIN; Password: [PROTECTED]; Enabled: true; AccountNonExpired: true;
> credentialsNonExpired: true; AccountNonLocked: true; Granted Authorities:
> ROLE_ADMIN,ROLE_ANALYST,ROLE_MODELER
>
> 2016-05-17 10:14:25,480 DEBUG [http-bio-7070-exec-5]
> controller.UserController:64 : authentication.getPrincipal() is
> org.springframework.security.core.userdetails.User@3b40b2f: Username:
> ADMIN; Password: [PROTECTED]; Enabled: true; AccountNonExpired: true;
> credentialsNonExpired: true; AccountNonLocked: true; Granted Authorities:
> ROLE_ADMIN,ROLE_ANALYST,ROLE_MODELER
>
> 2016-05-17 10:14:25,610 INFO [http-bio-7070-exec-1]
> controller.TableController:89 : Return all table metadata in 1 seconds
>
> 2016-05-17 10:14:28,341 DEBUG [http-bio-7070-exec-5]
> controller.UserController:64 : authentication.getPrincipal() is
> org.springframework.security.core.userdetails.User@3b40b2f: Username:
> ADMIN; Password: [PROTECTED]; Enabled: true; AccountNonExpired: true;
> credentialsNonExpired: true; AccountNonLocked: true; Granted Authorities:
> ROLE_ADMIN,ROLE_ANALYST,ROLE_MODELER
>
> 2016-05-17 10:14:28,570 DEBUG [http-bio-7070-exec-5]
> service.QueryService:293 : getting table metas
>
> 2016-05-17 10:14:28,574 DEBUG [http-bio-7070-exec-5]
> service.QueryService:311 : getting column metas
>
> 2016-05-17 10:14:28,601 DEBUG [http-bio-7070-exec-5]
> service.QueryService:325 : done column metas
>
> 2016-05-17 10:14:51,542 INFO [pool-4-thread-1]
> threadpool.DefaultScheduler:106 : Job Fetcher: 0 running, 0 actual running,
> 0 ready, 17 others
>
> 2016-05-17 10:14:52,457 INFO [http-bio-7070-exec-10]
> controller.QueryController:175 : Using project: compass
>
> 2016-05-17 10:14:52,458 INFO [http-bio-7070-exec-10]
> controller.QueryController:176 : The original query: select app_name,
> count(distinct uid) as uv
>
>
>
> from sdk_log
>
>
>
> left join week_calendar on sdk_log.day_time = week_calendar.week_cal
>
>
>
> where sdk_log.day_time = date '2016-05-08'
>
>
>
> group by app_name
>
>
>
> order by uv desc
>
>
>
> limit 200
>
>
>
>
>
> 2016-05-17 10:14:52,471 INFO [http-bio-7070-exec-10]
> service.QueryService:269 : The corrected query: select app_name,
> count(distinct uid) as uv
>
>
>
> from sdk_log
>
>
>
> left join week_calendar on sdk_log.day_time = week_calendar.week_cal
>
>
>
> where sdk_log.day_time = date '2016-05-08'
>
>
>
> group by app_name
>
>
>
> order by uv desc
>
>
>
> limit 200
>
> 2016-05-17 10:14:53,938 INFO [http-bio-7070-exec-10]
> routing.QueryRouter:48 : The project manager's reference is
> org.apache.kylin.metadata.project.ProjectManager@58baceac
>
> 2016-05-17 10:14:53,941 INFO [http-bio-7070-exec-10]
> routing.QueryRouter:60 : Find candidates by table DEFAULT.SDK_LOG and
> project=COMPASS : org.apache.kylin.query.routing.Candidate@70f24f62
>
> 2016-05-17 10:14:53,947 INFO [http-bio-7070-exec-10]
> routing.QueryRouter:49 : Applying rule: class
> org.apache.kylin.query.routing.rules.RemoveUncapableRealizationsRule,
> realizations before: [sdk_cube(CUBE)], realizations after: [sdk_cube(CUBE)]
>
> 2016-05-17 10:14:53,948 INFO [http-bio-7070-exec-10]
> routing.QueryRouter:49 : Applying rule: class
> org.apache.kylin.query.routing.rules.RealizationSortRule, realizations
> before: [sdk_cube(CUBE)], realizations after: [sdk_cube(CUBE)]
>
> 2016-05-17 10:14:53,948 INFO [http-bio-7070-exec-10]
> routing.QueryRouter:72 : The realizations remaining: [sdk_cube(CUBE)] And
> the final chosen one is the first one
>
> 2016-05-17 10:14:54,662 DEBUG [http-bio-7070-exec-10]
> enumerator.OLAPEnumerator:107 : query storage...
>
> 2016-05-17 10:14:54,749 INFO [http-bio-7070-exec-10]
> cache.AbstractCacheFledgedQuery:85 : Cache for
> 95f6d938-b974-4a92-b858-c0522faa7ff4 initializing...
>
> 2016-05-17 10:14:54,797 INFO [http-bio-7070-exec-10]
> cache.CacheFledgedStaticQuery:58 : no existing cache to use
>
> 2016-05-17 10:14:54,802 INFO [http-bio-7070-exec-10]
> v2.CubeStorageQuery:252 : exactAggregation is true
>
> 2016-05-17 10:14:54,812 INFO [http-bio-7070-exec-10]
> v2.CubeStorageQuery:358 : Memory budget is set to: 49146
>
> 2016-05-17 10:14:54,834 INFO [http-bio-7070-exec-10]
> dict.DictionaryManager:401 : DictionaryManager(633099583) loading
> DictionaryInfo(loadDictObj:true) at
> /dict/DEFAULT.SDK_LOG/APP_NAME/21980b9c-a872-4fb6-aef4-3a96ed2ef1c8.dict
>
> 2016-05-17 10:14:54,959 INFO [http-bio-7070-exec-10]
> dict.DictionaryManager:401 : DictionaryManager(633099583) loading
> DictionaryInfo(loadDictObj:true) at
> /dict/DEFAULT.SDK_LOG/DAY_TIME/4b6b9a89-8666-440f-94ce-cf3b2d68fc44.dict
>
> 2016-05-17 10:14:55,037 DEBUG [http-bio-7070-exec-10]
> v2.CubeHBaseEndpointRPC:248 : New scanner for current segment
> sdk_cube[20160502000000_20170103000000] will use SCAN_FILTER_AGGR_CHECKMEM
> as endpoint's behavior
>
> 2016-05-17 10:14:55,074 DEBUG [http-bio-7070-exec-10]
> v2.CubeHBaseEndpointRPC:283 : Serialized scanRequestBytes 216 bytes,
> rawScanBytesString 81 bytes
>
> 2016-05-17 10:14:55,075 INFO [http-bio-7070-exec-10]
> v2.CubeHBaseEndpointRPC:286 : The scan(s) info for current segment is as
> below, shard part of start/end key is set to 0
>
> 2016-05-17 10:14:55,082 INFO [http-bio-7070-exec-10] v2.CubeHBaseRPC:309
> : Visiting hbase table KYLIN_F6NPB6IA0Z: cuboid exact match, from 5 to 5
> Start: \x00\x00\x00\x00\x00\x00\x00\x00\x00\x05\x00\x00\x00\x0B\x3C\xCB
> (\x00\x00\x00\x00\x00\x00\x00\x00\x00\x05\x00\x00\x00\x0B<\xCB) Stop:
> \x00\x00\x00\x00\x00\x00\x00\x00\x00\x05\xFF\xFF\xFF\x0B\x3C\xCB\x00
> (\x00\x00\x00\x00\x00\x00\x00\x00\x00\x05\xFF\xFF\xFF\x0B<\xCB\x00) Fuzzy
> key counts: 1. Fuzzy keys :
> \x00\x00\x00\x00\x00\x00\x00\x00\x00\x05\x00\x00\x00\x0B\x3C\xCB
> \x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x01\x01\x01\x00\x00\x00;
>
> 2016-05-17 10:14:55,083 DEBUG [http-bio-7070-exec-10]
> v2.CubeHBaseEndpointRPC:292 : Submitting rpc to 2 shards starting from
> shard 1, scan requests count 1
>
> 2016-05-17 10:14:55,172 INFO [http-bio-7070-exec-10]
> v2.CubeHBaseEndpointRPC:123 : Timeout for ExpectedSizeIterator is 60000
>
> 2016-05-17 10:14:55,189 DEBUG [http-bio-7070-exec-10]
> enumerator.OLAPEnumerator:127 : return TupleIterator...
>
> 2016-05-17 10:14:55,190 DEBUG [http-bio-7070-exec-10]
> enumerator.OLAPEnumerator:128 : Storage cache used for this storage
> query:null
>
> 2016-05-17 10:15:05,367 INFO [pool-6-thread-2]
> v2.CubeHBaseEndpointRPC:320 : <spawned by http-bio-7070-exec-10>Endpoint
> RPC returned from HTable KYLIN_F6NPB6IA0Z Shard
> \x4B\x59\x4C\x49\x4E\x5F\x46\x36\x4E\x50\x42\x36\x49\x41\x30\x5A\x2C\x2C\x31\x34\x36\x33\x32\x32\x35\x31\x30\x33\x35\x35\x38\x2E\x32\x32\x37\x39\x34\x66\x30\x37\x39\x34\x37\x63\x39\x34\x36\x30\x34\x64\x33\x66\x66\x36\x33\x34\x31\x62\x36\x34\x65\x34\x30\x33\x2E
> on host: hfa-pro0043.hadoop.cpcc.iflyyun.cn.Total scanned row: 128455.
> Total filtered/aggred row: 0. Time elapsed in EP: 9379(ms). Server CPU
> usage: 0.6528155468594548, server physical mem left: 1.5867633664E10,
> server swap mem left:1.00663287808E11.Etc message: 1,1183,9378,9379,.
>
> 2016-05-17 10:15:06,390 DEBUG [pool-6-thread-2] util.CompressionUtils:65 :
> Original: 36540710 bytes. Decompressed: 54563500 bytes
>
> 2016-05-17 10:15:06,401 INFO [http-bio-7070-exec-10]
> lookup.SnapshotManager:178 : Loading snapshotTable from
> /table_snapshot/week_calendar/95bd13bd-4283-4acc-8335-22e94481b1f8.snapshot,
> with loadData: true
>
> 2016-05-17 10:15:06,436 DEBUG [http-bio-7070-exec-10]
> lookup.SnapshotManager:184 : Loaded snapshot at
> /table_snapshot/week_calendar/95bd13bd-4283-4acc-8335-22e94481b1f8.snapshot
>
> 2016-05-17 10:15:08,057 INFO [pool-6-thread-1]
> v2.CubeHBaseEndpointRPC:320 : <spawned by http-bio-7070-exec-10>Endpoint
> RPC returned from HTable KYLIN_F6NPB6IA0Z Shard
> \x4B\x59\x4C\x49\x4E\x5F\x46\x36\x4E\x50\x42\x36\x49\x41\x30\x5A\x2C\x00\x01\x2C\x31\x34\x36\x33\x32\x32\x35\x31\x30\x33\x35\x35\x38\x2E\x37\x65\x39\x63\x38\x30\x37\x33\x31\x62\x33\x34\x63\x62\x63\x32\x36\x35\x33\x34\x65\x32\x39\x64\x65\x35\x38\x39\x64\x36\x38\x66\x2E
> on host: hfa-pro0042.hadoop.cpcc.iflyyun.cn.Total scanned row: 127642.
> Total filtered/aggred row: 0. Time elapsed in EP: 11935(ms). Server CPU
> usage: 0.9497198437438346, server physical mem left: 1.416450048E9, server
> swap mem left:1.00663287808E11.Etc message: 1,1278,11935,11935,.
>
> 2016-05-17 10:15:09,453 DEBUG [pool-6-thread-1] util.CompressionUtils:65 :
> Original: 35193219 bytes. Decompressed: 53229962 bytes
>
>
>
> *发件人:* user-return-560-jyzheng=iflytek.com@kylin.apache.org [mailto:
> user-return-560-jyzheng=iflytek.com@kylin.apache.org] *代表 *Li Yang
> *发送时间:* 2016年5月15日 19:32
> *收件人:* user@kylin.apache.org
> *抄送:* dev@kylin.apache.org
> *主题:* Re: kylin query failed
>
>
>
> How high is the cardinality? By saying "failed" and then have to restart,
> do you mean crash? Any clue in kylin.log?
>
> Sorry for many questions, but it's hard to help without the details. The
> new 1.5.2 release will come with a diagnosis tool that can extract
> necessary info into a zip, which you can share with community to diagnose.
>
>
>
> On Mon, May 9, 2016 at 5:12 PM, jyzheng <jy...@iflytek.com> wrote:
>
>
>
> When I send sql query to Kylin like this:
>
>
>
> select app_name, count(distinct uid) as uv
> from sdk_log
> left join week_calendar on sdk_log.day_time = week_calendar.week_cal
> where sdk_log.day_time = date '2016-05-01'
> group by app_name
> order by uv desc
> limit 200
>
>
>
>
>
> and failed. So I have to restart Kylin.
>
>
>
> The dimension `app_name` is a great high cardinality. If I switch to
> `app_tag` dimension , it will return the right result. So I can’t get the
> top measure dimension in Kylin? If can, and what can I do to solve this?
>
> --------------------------------
>
> 郑江雨 云平台
>
> Phone: 15155195496
>
>
>
>
>
Re: 答复: kylin query failed
Posted by Li Yang <li...@apache.org>.
My best guess is out-of-memory crash. Returning 30K HLL counter may take
200MB~2GB memory depending on the count distinct precision. Suggest
increase Kylin JVM heap and try again.
On Tue, May 17, 2016 at 10:16 AM, jyzheng <jy...@iflytek.com> wrote:
> 1. The cardinality of app_name is about 30,000.
>
> 2. Yes, the Kylin has crashed after query failed.
>
> 3. The clue of kylin.log in the given below:
>
>
>
> Thanks for attention!
>
>
>
>
>
> 2016-05-17 10:13:51,531 INFO [pool-4-thread-1]
> threadpool.DefaultScheduler:106 : Job Fetcher: 0 running, 0 actual running,
> 0 ready, 17 others
>
> 2016-05-17 10:14:20,031 DEBUG [http-bio-7070-exec-5]
> service.AdminService:90 : Get Kylin Runtime Config
>
> 2016-05-17 10:14:20,037 DEBUG [http-bio-7070-exec-8]
> controller.UserController:64 : authentication.getPrincipal() is
> org.springframework.security.core.userdetails.User@3b40b2f: Username:
> ADMIN; Password: [PROTECTED]; Enabled: true; AccountNonExpired: true;
> credentialsNonExpired: true; AccountNonLocked: true; Granted Authorities:
> ROLE_ADMIN,ROLE_ANALYST,ROLE_MODELER
>
> 2016-05-17 10:14:20,321 DEBUG [http-bio-7070-exec-5]
> controller.ProjectController:97 : authentication.getPrincipal() is
> org.springframework.security.core.userdetails.User@3b40b2f: Username:
> ADMIN; Password: [PROTECTED]; Enabled: true; AccountNonExpired: true;
> credentialsNonExpired: true; AccountNonLocked: true; Granted Authorities:
> ROLE_ADMIN,ROLE_ANALYST,ROLE_MODELER
>
> 2016-05-17 10:14:25,467 DEBUG [http-bio-7070-exec-5]
> controller.UserController:64 : authentication.getPrincipal() is
> org.springframework.security.core.userdetails.User@3b40b2f: Username:
> ADMIN; Password: [PROTECTED]; Enabled: true; AccountNonExpired: true;
> credentialsNonExpired: true; AccountNonLocked: true; Granted Authorities:
> ROLE_ADMIN,ROLE_ANALYST,ROLE_MODELER
>
> 2016-05-17 10:14:25,480 DEBUG [http-bio-7070-exec-5]
> controller.UserController:64 : authentication.getPrincipal() is
> org.springframework.security.core.userdetails.User@3b40b2f: Username:
> ADMIN; Password: [PROTECTED]; Enabled: true; AccountNonExpired: true;
> credentialsNonExpired: true; AccountNonLocked: true; Granted Authorities:
> ROLE_ADMIN,ROLE_ANALYST,ROLE_MODELER
>
> 2016-05-17 10:14:25,610 INFO [http-bio-7070-exec-1]
> controller.TableController:89 : Return all table metadata in 1 seconds
>
> 2016-05-17 10:14:28,341 DEBUG [http-bio-7070-exec-5]
> controller.UserController:64 : authentication.getPrincipal() is
> org.springframework.security.core.userdetails.User@3b40b2f: Username:
> ADMIN; Password: [PROTECTED]; Enabled: true; AccountNonExpired: true;
> credentialsNonExpired: true; AccountNonLocked: true; Granted Authorities:
> ROLE_ADMIN,ROLE_ANALYST,ROLE_MODELER
>
> 2016-05-17 10:14:28,570 DEBUG [http-bio-7070-exec-5]
> service.QueryService:293 : getting table metas
>
> 2016-05-17 10:14:28,574 DEBUG [http-bio-7070-exec-5]
> service.QueryService:311 : getting column metas
>
> 2016-05-17 10:14:28,601 DEBUG [http-bio-7070-exec-5]
> service.QueryService:325 : done column metas
>
> 2016-05-17 10:14:51,542 INFO [pool-4-thread-1]
> threadpool.DefaultScheduler:106 : Job Fetcher: 0 running, 0 actual running,
> 0 ready, 17 others
>
> 2016-05-17 10:14:52,457 INFO [http-bio-7070-exec-10]
> controller.QueryController:175 : Using project: compass
>
> 2016-05-17 10:14:52,458 INFO [http-bio-7070-exec-10]
> controller.QueryController:176 : The original query: select app_name,
> count(distinct uid) as uv
>
>
>
> from sdk_log
>
>
>
> left join week_calendar on sdk_log.day_time = week_calendar.week_cal
>
>
>
> where sdk_log.day_time = date '2016-05-08'
>
>
>
> group by app_name
>
>
>
> order by uv desc
>
>
>
> limit 200
>
>
>
>
>
> 2016-05-17 10:14:52,471 INFO [http-bio-7070-exec-10]
> service.QueryService:269 : The corrected query: select app_name,
> count(distinct uid) as uv
>
>
>
> from sdk_log
>
>
>
> left join week_calendar on sdk_log.day_time = week_calendar.week_cal
>
>
>
> where sdk_log.day_time = date '2016-05-08'
>
>
>
> group by app_name
>
>
>
> order by uv desc
>
>
>
> limit 200
>
> 2016-05-17 10:14:53,938 INFO [http-bio-7070-exec-10]
> routing.QueryRouter:48 : The project manager's reference is
> org.apache.kylin.metadata.project.ProjectManager@58baceac
>
> 2016-05-17 10:14:53,941 INFO [http-bio-7070-exec-10]
> routing.QueryRouter:60 : Find candidates by table DEFAULT.SDK_LOG and
> project=COMPASS : org.apache.kylin.query.routing.Candidate@70f24f62
>
> 2016-05-17 10:14:53,947 INFO [http-bio-7070-exec-10]
> routing.QueryRouter:49 : Applying rule: class
> org.apache.kylin.query.routing.rules.RemoveUncapableRealizationsRule,
> realizations before: [sdk_cube(CUBE)], realizations after: [sdk_cube(CUBE)]
>
> 2016-05-17 10:14:53,948 INFO [http-bio-7070-exec-10]
> routing.QueryRouter:49 : Applying rule: class
> org.apache.kylin.query.routing.rules.RealizationSortRule, realizations
> before: [sdk_cube(CUBE)], realizations after: [sdk_cube(CUBE)]
>
> 2016-05-17 10:14:53,948 INFO [http-bio-7070-exec-10]
> routing.QueryRouter:72 : The realizations remaining: [sdk_cube(CUBE)] And
> the final chosen one is the first one
>
> 2016-05-17 10:14:54,662 DEBUG [http-bio-7070-exec-10]
> enumerator.OLAPEnumerator:107 : query storage...
>
> 2016-05-17 10:14:54,749 INFO [http-bio-7070-exec-10]
> cache.AbstractCacheFledgedQuery:85 : Cache for
> 95f6d938-b974-4a92-b858-c0522faa7ff4 initializing...
>
> 2016-05-17 10:14:54,797 INFO [http-bio-7070-exec-10]
> cache.CacheFledgedStaticQuery:58 : no existing cache to use
>
> 2016-05-17 10:14:54,802 INFO [http-bio-7070-exec-10]
> v2.CubeStorageQuery:252 : exactAggregation is true
>
> 2016-05-17 10:14:54,812 INFO [http-bio-7070-exec-10]
> v2.CubeStorageQuery:358 : Memory budget is set to: 49146
>
> 2016-05-17 10:14:54,834 INFO [http-bio-7070-exec-10]
> dict.DictionaryManager:401 : DictionaryManager(633099583) loading
> DictionaryInfo(loadDictObj:true) at
> /dict/DEFAULT.SDK_LOG/APP_NAME/21980b9c-a872-4fb6-aef4-3a96ed2ef1c8.dict
>
> 2016-05-17 10:14:54,959 INFO [http-bio-7070-exec-10]
> dict.DictionaryManager:401 : DictionaryManager(633099583) loading
> DictionaryInfo(loadDictObj:true) at
> /dict/DEFAULT.SDK_LOG/DAY_TIME/4b6b9a89-8666-440f-94ce-cf3b2d68fc44.dict
>
> 2016-05-17 10:14:55,037 DEBUG [http-bio-7070-exec-10]
> v2.CubeHBaseEndpointRPC:248 : New scanner for current segment
> sdk_cube[20160502000000_20170103000000] will use SCAN_FILTER_AGGR_CHECKMEM
> as endpoint's behavior
>
> 2016-05-17 10:14:55,074 DEBUG [http-bio-7070-exec-10]
> v2.CubeHBaseEndpointRPC:283 : Serialized scanRequestBytes 216 bytes,
> rawScanBytesString 81 bytes
>
> 2016-05-17 10:14:55,075 INFO [http-bio-7070-exec-10]
> v2.CubeHBaseEndpointRPC:286 : The scan(s) info for current segment is as
> below, shard part of start/end key is set to 0
>
> 2016-05-17 10:14:55,082 INFO [http-bio-7070-exec-10] v2.CubeHBaseRPC:309
> : Visiting hbase table KYLIN_F6NPB6IA0Z: cuboid exact match, from 5 to 5
> Start: \x00\x00\x00\x00\x00\x00\x00\x00\x00\x05\x00\x00\x00\x0B\x3C\xCB
> (\x00\x00\x00\x00\x00\x00\x00\x00\x00\x05\x00\x00\x00\x0B<\xCB) Stop:
> \x00\x00\x00\x00\x00\x00\x00\x00\x00\x05\xFF\xFF\xFF\x0B\x3C\xCB\x00
> (\x00\x00\x00\x00\x00\x00\x00\x00\x00\x05\xFF\xFF\xFF\x0B<\xCB\x00) Fuzzy
> key counts: 1. Fuzzy keys :
> \x00\x00\x00\x00\x00\x00\x00\x00\x00\x05\x00\x00\x00\x0B\x3C\xCB
> \x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x01\x01\x01\x00\x00\x00;
>
> 2016-05-17 10:14:55,083 DEBUG [http-bio-7070-exec-10]
> v2.CubeHBaseEndpointRPC:292 : Submitting rpc to 2 shards starting from
> shard 1, scan requests count 1
>
> 2016-05-17 10:14:55,172 INFO [http-bio-7070-exec-10]
> v2.CubeHBaseEndpointRPC:123 : Timeout for ExpectedSizeIterator is 60000
>
> 2016-05-17 10:14:55,189 DEBUG [http-bio-7070-exec-10]
> enumerator.OLAPEnumerator:127 : return TupleIterator...
>
> 2016-05-17 10:14:55,190 DEBUG [http-bio-7070-exec-10]
> enumerator.OLAPEnumerator:128 : Storage cache used for this storage
> query:null
>
> 2016-05-17 10:15:05,367 INFO [pool-6-thread-2]
> v2.CubeHBaseEndpointRPC:320 : <spawned by http-bio-7070-exec-10>Endpoint
> RPC returned from HTable KYLIN_F6NPB6IA0Z Shard
> \x4B\x59\x4C\x49\x4E\x5F\x46\x36\x4E\x50\x42\x36\x49\x41\x30\x5A\x2C\x2C\x31\x34\x36\x33\x32\x32\x35\x31\x30\x33\x35\x35\x38\x2E\x32\x32\x37\x39\x34\x66\x30\x37\x39\x34\x37\x63\x39\x34\x36\x30\x34\x64\x33\x66\x66\x36\x33\x34\x31\x62\x36\x34\x65\x34\x30\x33\x2E
> on host: hfa-pro0043.hadoop.cpcc.iflyyun.cn.Total scanned row: 128455.
> Total filtered/aggred row: 0. Time elapsed in EP: 9379(ms). Server CPU
> usage: 0.6528155468594548, server physical mem left: 1.5867633664E10,
> server swap mem left:1.00663287808E11.Etc message: 1,1183,9378,9379,.
>
> 2016-05-17 10:15:06,390 DEBUG [pool-6-thread-2] util.CompressionUtils:65 :
> Original: 36540710 bytes. Decompressed: 54563500 bytes
>
> 2016-05-17 10:15:06,401 INFO [http-bio-7070-exec-10]
> lookup.SnapshotManager:178 : Loading snapshotTable from
> /table_snapshot/week_calendar/95bd13bd-4283-4acc-8335-22e94481b1f8.snapshot,
> with loadData: true
>
> 2016-05-17 10:15:06,436 DEBUG [http-bio-7070-exec-10]
> lookup.SnapshotManager:184 : Loaded snapshot at
> /table_snapshot/week_calendar/95bd13bd-4283-4acc-8335-22e94481b1f8.snapshot
>
> 2016-05-17 10:15:08,057 INFO [pool-6-thread-1]
> v2.CubeHBaseEndpointRPC:320 : <spawned by http-bio-7070-exec-10>Endpoint
> RPC returned from HTable KYLIN_F6NPB6IA0Z Shard
> \x4B\x59\x4C\x49\x4E\x5F\x46\x36\x4E\x50\x42\x36\x49\x41\x30\x5A\x2C\x00\x01\x2C\x31\x34\x36\x33\x32\x32\x35\x31\x30\x33\x35\x35\x38\x2E\x37\x65\x39\x63\x38\x30\x37\x33\x31\x62\x33\x34\x63\x62\x63\x32\x36\x35\x33\x34\x65\x32\x39\x64\x65\x35\x38\x39\x64\x36\x38\x66\x2E
> on host: hfa-pro0042.hadoop.cpcc.iflyyun.cn.Total scanned row: 127642.
> Total filtered/aggred row: 0. Time elapsed in EP: 11935(ms). Server CPU
> usage: 0.9497198437438346, server physical mem left: 1.416450048E9, server
> swap mem left:1.00663287808E11.Etc message: 1,1278,11935,11935,.
>
> 2016-05-17 10:15:09,453 DEBUG [pool-6-thread-1] util.CompressionUtils:65 :
> Original: 35193219 bytes. Decompressed: 53229962 bytes
>
>
>
> *发件人:* user-return-560-jyzheng=iflytek.com@kylin.apache.org [mailto:
> user-return-560-jyzheng=iflytek.com@kylin.apache.org] *代表 *Li Yang
> *发送时间:* 2016年5月15日 19:32
> *收件人:* user@kylin.apache.org
> *抄送:* dev@kylin.apache.org
> *主题:* Re: kylin query failed
>
>
>
> How high is the cardinality? By saying "failed" and then have to restart,
> do you mean crash? Any clue in kylin.log?
>
> Sorry for many questions, but it's hard to help without the details. The
> new 1.5.2 release will come with a diagnosis tool that can extract
> necessary info into a zip, which you can share with community to diagnose.
>
>
>
> On Mon, May 9, 2016 at 5:12 PM, jyzheng <jy...@iflytek.com> wrote:
>
>
>
> When I send sql query to Kylin like this:
>
>
>
> select app_name, count(distinct uid) as uv
> from sdk_log
> left join week_calendar on sdk_log.day_time = week_calendar.week_cal
> where sdk_log.day_time = date '2016-05-01'
> group by app_name
> order by uv desc
> limit 200
>
>
>
>
>
> and failed. So I have to restart Kylin.
>
>
>
> The dimension `app_name` is a great high cardinality. If I switch to
> `app_tag` dimension , it will return the right result. So I can’t get the
> top measure dimension in Kylin? If can, and what can I do to solve this?
>
> --------------------------------
>
> 郑江雨 云平台
>
> Phone: 15155195496
>
>
>
>
>
答复: kylin query failed
Posted by jyzheng <jy...@iflytek.com>.
1. The cardinality of app_name is about 30,000.
2. Yes, the Kylin has crashed after query failed.
3. The clue of kylin.log in the given below:
Thanks for attention!
2016-05-17 10:13:51,531 INFO [pool-4-thread-1] threadpool.DefaultScheduler:106 : Job Fetcher: 0 running, 0 actual running, 0 ready, 17 others
2016-05-17 10:14:20,031 DEBUG [http-bio-7070-exec-5] service.AdminService:90 : Get Kylin Runtime Config
2016-05-17 10:14:20,037 DEBUG [http-bio-7070-exec-8] controller.UserController:64 : authentication.getPrincipal() is org.springframework.security.core.userdetails.User@3b40b2f: Username: ADMIN; Password: [PROTECTED]; Enabled: true; AccountNonExpired: true; credentialsNonExpired: true; AccountNonLocked: true; Granted Authorities: ROLE_ADMIN,ROLE_ANALYST,ROLE_MODELER
2016-05-17 10:14:20,321 DEBUG [http-bio-7070-exec-5] controller.ProjectController:97 : authentication.getPrincipal() is org.springframework.security.core.userdetails.User@3b40b2f: Username: ADMIN; Password: [PROTECTED]; Enabled: true; AccountNonExpired: true; credentialsNonExpired: true; AccountNonLocked: true; Granted Authorities: ROLE_ADMIN,ROLE_ANALYST,ROLE_MODELER
2016-05-17 10:14:25,467 DEBUG [http-bio-7070-exec-5] controller.UserController:64 : authentication.getPrincipal() is org.springframework.security.core.userdetails.User@3b40b2f: Username: ADMIN; Password: [PROTECTED]; Enabled: true; AccountNonExpired: true; credentialsNonExpired: true; AccountNonLocked: true; Granted Authorities: ROLE_ADMIN,ROLE_ANALYST,ROLE_MODELER
2016-05-17 10:14:25,480 DEBUG [http-bio-7070-exec-5] controller.UserController:64 : authentication.getPrincipal() is org.springframework.security.core.userdetails.User@3b40b2f: Username: ADMIN; Password: [PROTECTED]; Enabled: true; AccountNonExpired: true; credentialsNonExpired: true; AccountNonLocked: true; Granted Authorities: ROLE_ADMIN,ROLE_ANALYST,ROLE_MODELER
2016-05-17 10:14:25,610 INFO [http-bio-7070-exec-1] controller.TableController:89 : Return all table metadata in 1 seconds
2016-05-17 10:14:28,341 DEBUG [http-bio-7070-exec-5] controller.UserController:64 : authentication.getPrincipal() is org.springframework.security.core.userdetails.User@3b40b2f: Username: ADMIN; Password: [PROTECTED]; Enabled: true; AccountNonExpired: true; credentialsNonExpired: true; AccountNonLocked: true; Granted Authorities: ROLE_ADMIN,ROLE_ANALYST,ROLE_MODELER
2016-05-17 10:14:28,570 DEBUG [http-bio-7070-exec-5] service.QueryService:293 : getting table metas
2016-05-17 10:14:28,574 DEBUG [http-bio-7070-exec-5] service.QueryService:311 : getting column metas
2016-05-17 10:14:28,601 DEBUG [http-bio-7070-exec-5] service.QueryService:325 : done column metas
2016-05-17 10:14:51,542 INFO [pool-4-thread-1] threadpool.DefaultScheduler:106 : Job Fetcher: 0 running, 0 actual running, 0 ready, 17 others
2016-05-17 10:14:52,457 INFO [http-bio-7070-exec-10] controller.QueryController:175 : Using project: compass
2016-05-17 10:14:52,458 INFO [http-bio-7070-exec-10] controller.QueryController:176 : The original query: select app_name, count(distinct uid) as uv
from sdk_log
left join week_calendar on sdk_log.day_time = week_calendar.week_cal
where sdk_log.day_time = date '2016-05-08'
group by app_name
order by uv desc
limit 200
2016-05-17 10:14:52,471 INFO [http-bio-7070-exec-10] service.QueryService:269 : The corrected query: select app_name, count(distinct uid) as uv
from sdk_log
left join week_calendar on sdk_log.day_time = week_calendar.week_cal
where sdk_log.day_time = date '2016-05-08'
group by app_name
order by uv desc
limit 200
2016-05-17 10:14:53,938 INFO [http-bio-7070-exec-10] routing.QueryRouter:48 : The project manager's reference is org.apache.kylin.metadata.project.ProjectManager@58baceac
2016-05-17 10:14:53,941 INFO [http-bio-7070-exec-10] routing.QueryRouter:60 : Find candidates by table DEFAULT.SDK_LOG and project=COMPASS : org.apache.kylin.query.routing.Candidate@70f24f62
2016-05-17 10:14:53,947 INFO [http-bio-7070-exec-10] routing.QueryRouter:49 : Applying rule: class org.apache.kylin.query.routing.rules.RemoveUncapableRealizationsRule, realizations before: [sdk_cube(CUBE)], realizations after: [sdk_cube(CUBE)]
2016-05-17 10:14:53,948 INFO [http-bio-7070-exec-10] routing.QueryRouter:49 : Applying rule: class org.apache.kylin.query.routing.rules.RealizationSortRule, realizations before: [sdk_cube(CUBE)], realizations after: [sdk_cube(CUBE)]
2016-05-17 10:14:53,948 INFO [http-bio-7070-exec-10] routing.QueryRouter:72 : The realizations remaining: [sdk_cube(CUBE)] And the final chosen one is the first one
2016-05-17 10:14:54,662 DEBUG [http-bio-7070-exec-10] enumerator.OLAPEnumerator:107 : query storage...
2016-05-17 10:14:54,749 INFO [http-bio-7070-exec-10] cache.AbstractCacheFledgedQuery:85 : Cache for 95f6d938-b974-4a92-b858-c0522faa7ff4 initializing...
2016-05-17 10:14:54,797 INFO [http-bio-7070-exec-10] cache.CacheFledgedStaticQuery:58 : no existing cache to use
2016-05-17 10:14:54,802 INFO [http-bio-7070-exec-10] v2.CubeStorageQuery:252 : exactAggregation is true
2016-05-17 10:14:54,812 INFO [http-bio-7070-exec-10] v2.CubeStorageQuery:358 : Memory budget is set to: 49146
2016-05-17 10:14:54,834 INFO [http-bio-7070-exec-10] dict.DictionaryManager:401 : DictionaryManager(633099583) loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.SDK_LOG/APP_NAME/21980b9c-a872-4fb6-aef4-3a96ed2ef1c8.dict
2016-05-17 10:14:54,959 INFO [http-bio-7070-exec-10] dict.DictionaryManager:401 : DictionaryManager(633099583) loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.SDK_LOG/DAY_TIME/4b6b9a89-8666-440f-94ce-cf3b2d68fc44.dict
2016-05-17 10:14:55,037 DEBUG [http-bio-7070-exec-10] v2.CubeHBaseEndpointRPC:248 : New scanner for current segment sdk_cube[20160502000000_20170103000000] will use SCAN_FILTER_AGGR_CHECKMEM as endpoint's behavior
2016-05-17 10:14:55,074 DEBUG [http-bio-7070-exec-10] v2.CubeHBaseEndpointRPC:283 : Serialized scanRequestBytes 216 bytes, rawScanBytesString 81 bytes
2016-05-17 10:14:55,075 INFO [http-bio-7070-exec-10] v2.CubeHBaseEndpointRPC:286 : The scan(s) info for current segment is as below, shard part of start/end key is set to 0
2016-05-17 10:14:55,082 INFO [http-bio-7070-exec-10] v2.CubeHBaseRPC:309 : Visiting hbase table KYLIN_F6NPB6IA0Z: cuboid exact match, from 5 to 5 Start: \x00\x00\x00\x00\x00\x00\x00\x00\x00\x05\x00\x00\x00\x0B\x3C\xCB (\x00\x00\x00\x00\x00\x00\x00\x00\x00\x05\x00\x00\x00\x0B<\xCB) Stop: \x00\x00\x00\x00\x00\x00\x00\x00\x00\x05\xFF\xFF\xFF\x0B\x3C\xCB\x00 (\x00\x00\x00\x00\x00\x00\x00\x00\x00\x05\xFF\xFF\xFF\x0B<\xCB\x00) Fuzzy key counts: 1. Fuzzy keys : \x00\x00\x00\x00\x00\x00\x00\x00\x00\x05\x00\x00\x00\x0B\x3C\xCB \x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x01\x01\x01\x00\x00\x00;
2016-05-17 10:14:55,083 DEBUG [http-bio-7070-exec-10] v2.CubeHBaseEndpointRPC:292 : Submitting rpc to 2 shards starting from shard 1, scan requests count 1
2016-05-17 10:14:55,172 INFO [http-bio-7070-exec-10] v2.CubeHBaseEndpointRPC:123 : Timeout for ExpectedSizeIterator is 60000
2016-05-17 10:14:55,189 DEBUG [http-bio-7070-exec-10] enumerator.OLAPEnumerator:127 : return TupleIterator...
2016-05-17 10:14:55,190 DEBUG [http-bio-7070-exec-10] enumerator.OLAPEnumerator:128 : Storage cache used for this storage query:null
2016-05-17 10:15:05,367 INFO [pool-6-thread-2] v2.CubeHBaseEndpointRPC:320 : <spawned by http-bio-7070-exec-10>Endpoint RPC returned from HTable KYLIN_F6NPB6IA0Z Shard \x4B\x59\x4C\x49\x4E\x5F\x46\x36\x4E\x50\x42\x36\x49\x41\x30\x5A\x2C\x2C\x31\x34\x36\x33\x32\x32\x35\x31\x30\x33\x35\x35\x38\x2E\x32\x32\x37\x39\x34\x66\x30\x37\x39\x34\x37\x63\x39\x34\x36\x30\x34\x64\x33\x66\x66\x36\x33\x34\x31\x62\x36\x34\x65\x34\x30\x33\x2E on host: hfa-pro0043.hadoop.cpcc.iflyyun.cn.Total scanned row: 128455. Total filtered/aggred row: 0. Time elapsed in EP: 9379(ms). Server CPU usage: 0.6528155468594548, server physical mem left: 1.5867633664E10, server swap mem left:1.00663287808E11.Etc message: 1,1183,9378,9379,.
2016-05-17 10:15:06,390 DEBUG [pool-6-thread-2] util.CompressionUtils:65 : Original: 36540710 bytes. Decompressed: 54563500 bytes
2016-05-17 10:15:06,401 INFO [http-bio-7070-exec-10] lookup.SnapshotManager:178 : Loading snapshotTable from /table_snapshot/week_calendar/95bd13bd-4283-4acc-8335-22e94481b1f8.snapshot, with loadData: true
2016-05-17 10:15:06,436 DEBUG [http-bio-7070-exec-10] lookup.SnapshotManager:184 : Loaded snapshot at /table_snapshot/week_calendar/95bd13bd-4283-4acc-8335-22e94481b1f8.snapshot
2016-05-17 10:15:08,057 INFO [pool-6-thread-1] v2.CubeHBaseEndpointRPC:320 : <spawned by http-bio-7070-exec-10>Endpoint RPC returned from HTable KYLIN_F6NPB6IA0Z Shard \x4B\x59\x4C\x49\x4E\x5F\x46\x36\x4E\x50\x42\x36\x49\x41\x30\x5A\x2C\x00\x01\x2C\x31\x34\x36\x33\x32\x32\x35\x31\x30\x33\x35\x35\x38\x2E\x37\x65\x39\x63\x38\x30\x37\x33\x31\x62\x33\x34\x63\x62\x63\x32\x36\x35\x33\x34\x65\x32\x39\x64\x65\x35\x38\x39\x64\x36\x38\x66\x2E on host: hfa-pro0042.hadoop.cpcc.iflyyun.cn.Total scanned row: 127642. Total filtered/aggred row: 0. Time elapsed in EP: 11935(ms). Server CPU usage: 0.9497198437438346, server physical mem left: 1.416450048E9, server swap mem left:1.00663287808E11.Etc message: 1,1278,11935,11935,.
2016-05-17 10:15:09,453 DEBUG [pool-6-thread-1] util.CompressionUtils:65 : Original: 35193219 bytes. Decompressed: 53229962 bytes
发件人: user-return-560-jyzheng=iflytek.com@kylin.apache.org [mailto:user-return-560-jyzheng=iflytek.com@kylin.apache.org] 代表 Li Yang
发送时间: 2016年5月15日 19:32
收件人: user@kylin.apache.org
抄送: dev@kylin.apache.org
主题: Re: kylin query failed
How high is the cardinality? By saying "failed" and then have to restart, do you mean crash? Any clue in kylin.log?
Sorry for many questions, but it's hard to help without the details. The new 1.5.2 release will come with a diagnosis tool that can extract necessary info into a zip, which you can share with community to diagnose.
On Mon, May 9, 2016 at 5:12 PM, jyzheng <jyzheng@iflytek.com <ma...@iflytek.com> > wrote:
When I send sql query to Kylin like this:
select app_name, count(distinct uid) as uv
from sdk_log
left join week_calendar on sdk_log.day_time = week_calendar.week_cal
where sdk_log.day_time = date '2016-05-01'
group by app_name
order by uv desc
limit 200
and failed. So I have to restart Kylin.
The dimension `app_name` is a great high cardinality. If I switch to `app_tag` dimension , it will return the right result. So I can’t get the top measure dimension in Kylin? If can, and what can I do to solve this?
--------------------------------
郑江雨 云平台
Phone: 15155195496 <tel:15155195496>
答复: kylin query failed
Posted by jyzheng <jy...@iflytek.com>.
1. The cardinality of app_name is about 30,000.
2. Yes, the Kylin has crashed after query failed.
3. The clue of kylin.log in the given below:
Thanks for attention!
2016-05-17 10:13:51,531 INFO [pool-4-thread-1] threadpool.DefaultScheduler:106 : Job Fetcher: 0 running, 0 actual running, 0 ready, 17 others
2016-05-17 10:14:20,031 DEBUG [http-bio-7070-exec-5] service.AdminService:90 : Get Kylin Runtime Config
2016-05-17 10:14:20,037 DEBUG [http-bio-7070-exec-8] controller.UserController:64 : authentication.getPrincipal() is org.springframework.security.core.userdetails.User@3b40b2f: Username: ADMIN; Password: [PROTECTED]; Enabled: true; AccountNonExpired: true; credentialsNonExpired: true; AccountNonLocked: true; Granted Authorities: ROLE_ADMIN,ROLE_ANALYST,ROLE_MODELER
2016-05-17 10:14:20,321 DEBUG [http-bio-7070-exec-5] controller.ProjectController:97 : authentication.getPrincipal() is org.springframework.security.core.userdetails.User@3b40b2f: Username: ADMIN; Password: [PROTECTED]; Enabled: true; AccountNonExpired: true; credentialsNonExpired: true; AccountNonLocked: true; Granted Authorities: ROLE_ADMIN,ROLE_ANALYST,ROLE_MODELER
2016-05-17 10:14:25,467 DEBUG [http-bio-7070-exec-5] controller.UserController:64 : authentication.getPrincipal() is org.springframework.security.core.userdetails.User@3b40b2f: Username: ADMIN; Password: [PROTECTED]; Enabled: true; AccountNonExpired: true; credentialsNonExpired: true; AccountNonLocked: true; Granted Authorities: ROLE_ADMIN,ROLE_ANALYST,ROLE_MODELER
2016-05-17 10:14:25,480 DEBUG [http-bio-7070-exec-5] controller.UserController:64 : authentication.getPrincipal() is org.springframework.security.core.userdetails.User@3b40b2f: Username: ADMIN; Password: [PROTECTED]; Enabled: true; AccountNonExpired: true; credentialsNonExpired: true; AccountNonLocked: true; Granted Authorities: ROLE_ADMIN,ROLE_ANALYST,ROLE_MODELER
2016-05-17 10:14:25,610 INFO [http-bio-7070-exec-1] controller.TableController:89 : Return all table metadata in 1 seconds
2016-05-17 10:14:28,341 DEBUG [http-bio-7070-exec-5] controller.UserController:64 : authentication.getPrincipal() is org.springframework.security.core.userdetails.User@3b40b2f: Username: ADMIN; Password: [PROTECTED]; Enabled: true; AccountNonExpired: true; credentialsNonExpired: true; AccountNonLocked: true; Granted Authorities: ROLE_ADMIN,ROLE_ANALYST,ROLE_MODELER
2016-05-17 10:14:28,570 DEBUG [http-bio-7070-exec-5] service.QueryService:293 : getting table metas
2016-05-17 10:14:28,574 DEBUG [http-bio-7070-exec-5] service.QueryService:311 : getting column metas
2016-05-17 10:14:28,601 DEBUG [http-bio-7070-exec-5] service.QueryService:325 : done column metas
2016-05-17 10:14:51,542 INFO [pool-4-thread-1] threadpool.DefaultScheduler:106 : Job Fetcher: 0 running, 0 actual running, 0 ready, 17 others
2016-05-17 10:14:52,457 INFO [http-bio-7070-exec-10] controller.QueryController:175 : Using project: compass
2016-05-17 10:14:52,458 INFO [http-bio-7070-exec-10] controller.QueryController:176 : The original query: select app_name, count(distinct uid) as uv
from sdk_log
left join week_calendar on sdk_log.day_time = week_calendar.week_cal
where sdk_log.day_time = date '2016-05-08'
group by app_name
order by uv desc
limit 200
2016-05-17 10:14:52,471 INFO [http-bio-7070-exec-10] service.QueryService:269 : The corrected query: select app_name, count(distinct uid) as uv
from sdk_log
left join week_calendar on sdk_log.day_time = week_calendar.week_cal
where sdk_log.day_time = date '2016-05-08'
group by app_name
order by uv desc
limit 200
2016-05-17 10:14:53,938 INFO [http-bio-7070-exec-10] routing.QueryRouter:48 : The project manager's reference is org.apache.kylin.metadata.project.ProjectManager@58baceac
2016-05-17 10:14:53,941 INFO [http-bio-7070-exec-10] routing.QueryRouter:60 : Find candidates by table DEFAULT.SDK_LOG and project=COMPASS : org.apache.kylin.query.routing.Candidate@70f24f62
2016-05-17 10:14:53,947 INFO [http-bio-7070-exec-10] routing.QueryRouter:49 : Applying rule: class org.apache.kylin.query.routing.rules.RemoveUncapableRealizationsRule, realizations before: [sdk_cube(CUBE)], realizations after: [sdk_cube(CUBE)]
2016-05-17 10:14:53,948 INFO [http-bio-7070-exec-10] routing.QueryRouter:49 : Applying rule: class org.apache.kylin.query.routing.rules.RealizationSortRule, realizations before: [sdk_cube(CUBE)], realizations after: [sdk_cube(CUBE)]
2016-05-17 10:14:53,948 INFO [http-bio-7070-exec-10] routing.QueryRouter:72 : The realizations remaining: [sdk_cube(CUBE)] And the final chosen one is the first one
2016-05-17 10:14:54,662 DEBUG [http-bio-7070-exec-10] enumerator.OLAPEnumerator:107 : query storage...
2016-05-17 10:14:54,749 INFO [http-bio-7070-exec-10] cache.AbstractCacheFledgedQuery:85 : Cache for 95f6d938-b974-4a92-b858-c0522faa7ff4 initializing...
2016-05-17 10:14:54,797 INFO [http-bio-7070-exec-10] cache.CacheFledgedStaticQuery:58 : no existing cache to use
2016-05-17 10:14:54,802 INFO [http-bio-7070-exec-10] v2.CubeStorageQuery:252 : exactAggregation is true
2016-05-17 10:14:54,812 INFO [http-bio-7070-exec-10] v2.CubeStorageQuery:358 : Memory budget is set to: 49146
2016-05-17 10:14:54,834 INFO [http-bio-7070-exec-10] dict.DictionaryManager:401 : DictionaryManager(633099583) loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.SDK_LOG/APP_NAME/21980b9c-a872-4fb6-aef4-3a96ed2ef1c8.dict
2016-05-17 10:14:54,959 INFO [http-bio-7070-exec-10] dict.DictionaryManager:401 : DictionaryManager(633099583) loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.SDK_LOG/DAY_TIME/4b6b9a89-8666-440f-94ce-cf3b2d68fc44.dict
2016-05-17 10:14:55,037 DEBUG [http-bio-7070-exec-10] v2.CubeHBaseEndpointRPC:248 : New scanner for current segment sdk_cube[20160502000000_20170103000000] will use SCAN_FILTER_AGGR_CHECKMEM as endpoint's behavior
2016-05-17 10:14:55,074 DEBUG [http-bio-7070-exec-10] v2.CubeHBaseEndpointRPC:283 : Serialized scanRequestBytes 216 bytes, rawScanBytesString 81 bytes
2016-05-17 10:14:55,075 INFO [http-bio-7070-exec-10] v2.CubeHBaseEndpointRPC:286 : The scan(s) info for current segment is as below, shard part of start/end key is set to 0
2016-05-17 10:14:55,082 INFO [http-bio-7070-exec-10] v2.CubeHBaseRPC:309 : Visiting hbase table KYLIN_F6NPB6IA0Z: cuboid exact match, from 5 to 5 Start: \x00\x00\x00\x00\x00\x00\x00\x00\x00\x05\x00\x00\x00\x0B\x3C\xCB (\x00\x00\x00\x00\x00\x00\x00\x00\x00\x05\x00\x00\x00\x0B<\xCB) Stop: \x00\x00\x00\x00\x00\x00\x00\x00\x00\x05\xFF\xFF\xFF\x0B\x3C\xCB\x00 (\x00\x00\x00\x00\x00\x00\x00\x00\x00\x05\xFF\xFF\xFF\x0B<\xCB\x00) Fuzzy key counts: 1. Fuzzy keys : \x00\x00\x00\x00\x00\x00\x00\x00\x00\x05\x00\x00\x00\x0B\x3C\xCB \x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x01\x01\x01\x00\x00\x00;
2016-05-17 10:14:55,083 DEBUG [http-bio-7070-exec-10] v2.CubeHBaseEndpointRPC:292 : Submitting rpc to 2 shards starting from shard 1, scan requests count 1
2016-05-17 10:14:55,172 INFO [http-bio-7070-exec-10] v2.CubeHBaseEndpointRPC:123 : Timeout for ExpectedSizeIterator is 60000
2016-05-17 10:14:55,189 DEBUG [http-bio-7070-exec-10] enumerator.OLAPEnumerator:127 : return TupleIterator...
2016-05-17 10:14:55,190 DEBUG [http-bio-7070-exec-10] enumerator.OLAPEnumerator:128 : Storage cache used for this storage query:null
2016-05-17 10:15:05,367 INFO [pool-6-thread-2] v2.CubeHBaseEndpointRPC:320 : <spawned by http-bio-7070-exec-10>Endpoint RPC returned from HTable KYLIN_F6NPB6IA0Z Shard \x4B\x59\x4C\x49\x4E\x5F\x46\x36\x4E\x50\x42\x36\x49\x41\x30\x5A\x2C\x2C\x31\x34\x36\x33\x32\x32\x35\x31\x30\x33\x35\x35\x38\x2E\x32\x32\x37\x39\x34\x66\x30\x37\x39\x34\x37\x63\x39\x34\x36\x30\x34\x64\x33\x66\x66\x36\x33\x34\x31\x62\x36\x34\x65\x34\x30\x33\x2E on host: hfa-pro0043.hadoop.cpcc.iflyyun.cn.Total scanned row: 128455. Total filtered/aggred row: 0. Time elapsed in EP: 9379(ms). Server CPU usage: 0.6528155468594548, server physical mem left: 1.5867633664E10, server swap mem left:1.00663287808E11.Etc message: 1,1183,9378,9379,.
2016-05-17 10:15:06,390 DEBUG [pool-6-thread-2] util.CompressionUtils:65 : Original: 36540710 bytes. Decompressed: 54563500 bytes
2016-05-17 10:15:06,401 INFO [http-bio-7070-exec-10] lookup.SnapshotManager:178 : Loading snapshotTable from /table_snapshot/week_calendar/95bd13bd-4283-4acc-8335-22e94481b1f8.snapshot, with loadData: true
2016-05-17 10:15:06,436 DEBUG [http-bio-7070-exec-10] lookup.SnapshotManager:184 : Loaded snapshot at /table_snapshot/week_calendar/95bd13bd-4283-4acc-8335-22e94481b1f8.snapshot
2016-05-17 10:15:08,057 INFO [pool-6-thread-1] v2.CubeHBaseEndpointRPC:320 : <spawned by http-bio-7070-exec-10>Endpoint RPC returned from HTable KYLIN_F6NPB6IA0Z Shard \x4B\x59\x4C\x49\x4E\x5F\x46\x36\x4E\x50\x42\x36\x49\x41\x30\x5A\x2C\x00\x01\x2C\x31\x34\x36\x33\x32\x32\x35\x31\x30\x33\x35\x35\x38\x2E\x37\x65\x39\x63\x38\x30\x37\x33\x31\x62\x33\x34\x63\x62\x63\x32\x36\x35\x33\x34\x65\x32\x39\x64\x65\x35\x38\x39\x64\x36\x38\x66\x2E on host: hfa-pro0042.hadoop.cpcc.iflyyun.cn.Total scanned row: 127642. Total filtered/aggred row: 0. Time elapsed in EP: 11935(ms). Server CPU usage: 0.9497198437438346, server physical mem left: 1.416450048E9, server swap mem left:1.00663287808E11.Etc message: 1,1278,11935,11935,.
2016-05-17 10:15:09,453 DEBUG [pool-6-thread-1] util.CompressionUtils:65 : Original: 35193219 bytes. Decompressed: 53229962 bytes
发件人: user-return-560-jyzheng=iflytek.com@kylin.apache.org [mailto:user-return-560-jyzheng=iflytek.com@kylin.apache.org] 代表 Li Yang
发送时间: 2016年5月15日 19:32
收件人: user@kylin.apache.org
抄送: dev@kylin.apache.org
主题: Re: kylin query failed
How high is the cardinality? By saying "failed" and then have to restart, do you mean crash? Any clue in kylin.log?
Sorry for many questions, but it's hard to help without the details. The new 1.5.2 release will come with a diagnosis tool that can extract necessary info into a zip, which you can share with community to diagnose.
On Mon, May 9, 2016 at 5:12 PM, jyzheng <jyzheng@iflytek.com <ma...@iflytek.com> > wrote:
When I send sql query to Kylin like this:
select app_name, count(distinct uid) as uv
from sdk_log
left join week_calendar on sdk_log.day_time = week_calendar.week_cal
where sdk_log.day_time = date '2016-05-01'
group by app_name
order by uv desc
limit 200
and failed. So I have to restart Kylin.
The dimension `app_name` is a great high cardinality. If I switch to `app_tag` dimension , it will return the right result. So I can’t get the top measure dimension in Kylin? If can, and what can I do to solve this?
--------------------------------
郑江雨 云平台
Phone: 15155195496 <tel:15155195496>
Re: kylin query failed
Posted by Li Yang <li...@apache.org>.
How high is the cardinality? By saying "failed" and then have to restart,
do you mean crash? Any clue in kylin.log?
Sorry for many questions, but it's hard to help without the details. The
new 1.5.2 release will come with a diagnosis tool that can extract
necessary info into a zip, which you can share with community to diagnose.
On Mon, May 9, 2016 at 5:12 PM, jyzheng <jy...@iflytek.com> wrote:
>
>
> When I send sql query to Kylin like this:
>
>
>
> select app_name, count(distinct uid) as uv
> from sdk_log
> left join week_calendar on sdk_log.day_time = week_calendar.week_cal
> where sdk_log.day_time = date '2016-05-01'
> group by app_name
> order by uv desc
> limit 200
>
>
>
>
>
> and failed. So I have to restart Kylin.
>
>
>
> The dimension `app_name` is a great high cardinality. If I switch to
> `app_tag` dimension , it will return the right result. So I can’t get the
> top measure dimension in Kylin? If can, and what can I do to solve this?
>
> --------------------------------
>
> 郑江雨 云平台
>
> Phone: 15155195496
>
>
>
Re: kylin query failed
Posted by Li Yang <li...@apache.org>.
How high is the cardinality? By saying "failed" and then have to restart,
do you mean crash? Any clue in kylin.log?
Sorry for many questions, but it's hard to help without the details. The
new 1.5.2 release will come with a diagnosis tool that can extract
necessary info into a zip, which you can share with community to diagnose.
On Mon, May 9, 2016 at 5:12 PM, jyzheng <jy...@iflytek.com> wrote:
>
>
> When I send sql query to Kylin like this:
>
>
>
> select app_name, count(distinct uid) as uv
> from sdk_log
> left join week_calendar on sdk_log.day_time = week_calendar.week_cal
> where sdk_log.day_time = date '2016-05-01'
> group by app_name
> order by uv desc
> limit 200
>
>
>
>
>
> and failed. So I have to restart Kylin.
>
>
>
> The dimension `app_name` is a great high cardinality. If I switch to
> `app_tag` dimension , it will return the right result. So I can’t get the
> top measure dimension in Kylin? If can, and what can I do to solve this?
>
> --------------------------------
>
> 郑江雨 云平台
>
> Phone: 15155195496
>
>
>