You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@kylin.apache.org by yu feng <ol...@gmail.com> on 2017/08/22 04:24:43 UTC
Kylin query return empty after upgrade to 2.0
Hi, After I upgrade our env. from 1.5.2.1 to 2.0.0, building things go
well, However, every query to older(generated before upgrading) return
empty(0 result), and newly build segment return result success.
I add some debug log and find in GTFilterScanner, I get 0 input
from inputIterator and return 0 to GTAggregateScanner, Here is my kylin log
:
2017-08-22 11:46:07,308 INFO [kylin-coproc--pool157-t3958]
v2.CubeHBaseEndpointRPC:200 : <sub-thread for Query
e6e13b48-f28c-422c-b941-dfec2e836ea5 GTScanRequest 1b
b7584d>Endpoint RPC returned from HTable NEW_KYLIN_JDVOZ5IBWC Shard
\x4E\x45\x57\x5F\x4B\x59\x4C\x49\x4E\x5F\x4A\x44\x56\x4F\x5A\x35\x49\x42\x57\x43\x2C\x00\x01\x2C\
x31\x35\x30\x33\x31\x38\x31\x39\x34\x38\x33\x33\x33\x2E\x62\x37\x35\x31\x38\x32\x35\x32\x32\x63\x36\x64\x61\x63\x36\x39\x32\x33\x33\x63\x33\x32\x30\x38\x36\x34\x62\x
39\x38\x31\x37\x62\x2E on host: hz-hbase2.photo.163.org.Total scanned row:
0. Total scanned bytes: 0. Total filtered/aggred row: 0. Time elapsed in
EP: 2(ms). Server
CPU usage: 0.13636363636363635, server physical mem left: 4.35449856E8,
server swap mem left:8.589930496E9.Etc message: start latency: 6@0,agg
done@1,compress done@
1,server stats done@2,
debugGitTag:997bac07cd74f9a3ac9d50714e8740ddc2e5c1c7;.Normal Complete:
true.Compressed row size: 8
2017-08-22 11:46:07,561 INFO [kylin-coproc--pool157-t3961]
v2.CubeHBaseEndpointRPC:200 : <sub-thread for Query
e6e13b48-f28c-422c-b941-dfec2e836ea5 GTScanRequest 63
c14bf2>Endpoint RPC returned from HTable KYLIN_0H7MAKPJQL Shard
\x4B\x59\x4C\x49\x4E\x5F\x30\x48\x37\x4D\x41\x4B\x50\x4A\x51\x4C\x2C\x2C\x31\x35\x30\x33\x33\x36\x38\
x31\x36\x34\x37\x32\x32\x2E\x30\x63\x37\x39\x64\x34\x63\x36\x62\x63\x61\x62\x64\x39\x36\x64\x61\x61\x34\x31\x34\x32\x64\x64\x35\x64\x37\x64\x31\x34\x65\x34\x2E
on ho
st: hz-hmaster0.xs.163.org.Total scanned row: 87935. Total scanned bytes:
4752238. Total filtered/aggred row: 87934. Time elapsed in EP: 254(ms).
Server CPU usage: 0
.10531496062992125, server physical mem left: 4.39107584E8, server swap mem
left:2.553663488E9.Etc message: start latency: 2@1,agg done@253,compress
done@253,server
stats done@254,
debugGitTag:997bac07cd74f9a3ac9d50714e8740ddc2e5c1c7;.Normal Complete:
true.Compressed row size: 13
the first is older segment and later one is newly build segment, there scan
ranges are the same.
Here is my hbase log about the empty result :
2017-08-22 11:04:30,378 DEBUG [RpcServer.reader=9,port=60020]
security.HBaseSaslRpcServer: SASL server GSSAPI callback: setting
canonicalized client ID: nrpt/dev@HADOOP.HZ.NETEASE.COM
2017-08-22 11:04:31,714 INFO [Query
25ed5aec-6a9b-4507-bed4-198812cc25f0-226] gridtable.GTScanRequest: pre
aggregating results before returning
2017-08-22 11:04:31,714 INFO [Query
25ed5aec-6a9b-4507-bed4-198812cc25f0-226] gridtable.GTAggregateScanner:
GTAggregateScanner input rows: 0
2017-08-22 11:04:31,714 INFO [Query
25ed5aec-6a9b-4507-bed4-198812cc25f0-226] endpoint.CubeVisitService: Total
scanned 0 rows and 0 bytes
2017-08-22 11:04:31,714 INFO [Query
25ed5aec-6a9b-4507-bed4-198812cc25f0-226] endpoint.CubeVisitService: Size
of final result = 8 (0 before compressing)
2017-08-22 11:04:31,726 INFO [Query
25ed5aec-6a9b-4507-bed4-198812cc25f0-142] gridtable.GTScanRequest: pre
aggregating results before returning
2017-08-22 11:04:31,727 INFO [Query
25ed5aec-6a9b-4507-bed4-198812cc25f0-142] gridtable.GTAggregateScanner:
GTAggregateScanner input rows: 0
2017-08-22 11:04:31,727 INFO [Query
25ed5aec-6a9b-4507-bed4-198812cc25f0-142] endpoint.CubeVisitService: Total
scanned 0 rows and 0 bytes
2017-08-22 11:04:31,727 INFO [Query
25ed5aec-6a9b-4507-bed4-198812cc25f0-142] endpoint.CubeVisitService: Size
of final result = 8 (0 before compressing)
2017-08-22 11:04:31,730 INFO [Query
25ed5aec-6a9b-4507-bed4-198812cc25f0-221] gridtable.GTScanRequest: pre
aggregating results before returning
2017-08-22 11:04:31,731 INFO [Query
25ed5aec-6a9b-4507-bed4-198812cc25f0-221] gridtable.GTAggregateScanner:
GTAggregateScanner input rows: 0
2017-08-22 11:04:31,731 INFO [Query
25ed5aec-6a9b-4507-bed4-198812cc25f0-221] endpoint.CubeVisitService: Total
scanned 0 rows and 0 bytes
2017-08-22 11:04:31,731 INFO [Query
25ed5aec-6a9b-4507-bed4-198812cc25f0-221] endpoint.CubeVisitService: Size
of final result = 8 (0 before compressing)
2017-08-22 11:04:31,740 INFO [Query
25ed5aec-6a9b-4507-bed4-198812cc25f0-194] gridtable.GTScanRequest: pre
aggregating results before returning
2017-08-22 11:04:31,740 INFO [Query
25ed5aec-6a9b-4507-bed4-198812cc25f0-194] gridtable.GTAggregateScanner:
GTAggregateScanner input rows: 0
2017-08-22 11:04:31,740 INFO [Query
25ed5aec-6a9b-4507-bed4-198812cc25f0-194] endpoint.CubeVisitService: Total
scanned 0 rows and 0 bytes
2017-08-22 11:04:31,740 INFO [Query
25ed5aec-6a9b-4507-bed4-198812cc25f0-194] endpoint.CubeVisitService: Size
of final result = 8 (0 before compressing)
I guess maybe some rowkey rule is changed?
Did anyone meet the problem? Thanks a lot.
Re: 答复: Kylin query return empty after upgrade to 2.0
Posted by ShaoFeng Shi <sh...@apache.org>.
Donald, you're correct.
2017-08-23 17:43 GMT+08:00 Donald,Zheng(vip.com) <do...@vipshop.com>:
> Maybe it’s about cuboid scheduler.
> Specified query involved only mandatory dimensions will scan the mandatory
> only cuboid if kylin.cube.aggrgroup.is-mandatory-only-valid=true.
> And these cuboids don’t exist before 1.5.3
>
> 发件人: ShaoFeng Shi [mailto:shaofengshi@apache.org]
> 发送时间: 2017年8月23日 16:49
> 收件人: dev
> 抄送: user
> 主题: Re: Kylin query return empty after upgrade to 2.0
>
> Yu has solved this problem by setting this property back to its default
> value:
>
> kylin.cube.aggrgroup.is-mandatory-only-valid<http://
> kylin.cube.aggrgroup.is>=false
>
>
>
> I don't know the detail; but it seems this property will impact on the
> execution plan.
>
> 2017-08-23 11:27 GMT+08:00 Billy Liu <billyliu@apache.org<mailto:bi
> llyliu@apache.org>>:
> Query by REST API? or JDBC?
>
> 2017-08-22 12:24 GMT+08:00 yu feng <olaptestyu@gmail.com<mailto:o
> laptestyu@gmail.com>>:
>
> > Hi, After I upgrade our env. from 1.5.2.1 to 2.0.0, building things go
> > well, However, every query to older(generated before upgrading) return
> > empty(0 result), and newly build segment return result success.
> >
> > I add some debug log and find in GTFilterScanner, I get 0 input
> > from inputIterator and return 0 to GTAggregateScanner, Here is my kylin
> log
> > :
> >
> > 2017-08-22 11:46:07,308 INFO [kylin-coproc--pool157-t3958]
> > v2.CubeHBaseEndpointRPC:200 : <sub-thread for Query
> e6e13b48-f28c-422c-b941-dfec2e836ea5
> > GTScanRequest 1b
> > b7584d>Endpoint RPC returned from HTable NEW_KYLIN_JDVOZ5IBWC Shard
> > \x4E\x45\x57\x5F\x4B\x59\x4C\x49\x4E\x5F\x4A\x44\x56\x4F\
> > x5A\x35\x49\x42\x57\x43\x2C\x00\x01\x2C\
> > x31\x35\x30\x33\x31\x38\x31\x39\x34\x38\x33\x33\x33\x2E\
> > x62\x37\x35\x31\x38\x32\x35\x32\x32\x63\x36\x64\x61\x63\
> > x36\x39\x32\x33\x33\x63\x33\x32\x30\x38\x36\x34\x62\x
> > 39\x38\x31\x37\x62\x2E on host: hz-hbase2.photo.163.org.Total scanned
> row:
> > 0. Total scanned bytes: 0. Total filtered/aggred row: 0. Time elapsed in
> > EP: 2(ms). Server
> > CPU usage: 0.13636363636363635, server physical mem left: 4.35449856E8,
> > server swap mem left:8.589930496E9.Etc message: start latency: 6@0,agg
> > done@1,compress done@
> > 1,server stats done@2, debugGitTag:997bac07cd74f9a3ac9d50714e8740
> ddc2e5c1c7;.Normal
> > Complete: true.Compressed row size: 8
> >
> > 2017-08-22 11:46:07,561 INFO [kylin-coproc--pool157-t3961]
> > v2.CubeHBaseEndpointRPC:200 : <sub-thread for Query
> e6e13b48-f28c-422c-b941-dfec2e836ea5
> > GTScanRequest 63
> > c14bf2>Endpoint RPC returned from HTable KYLIN_0H7MAKPJQL Shard
> > \x4B\x59\x4C\x49\x4E\x5F\x30\x48\x37\x4D\x41\x4B\x50\x4A\
> > x51\x4C\x2C\x2C\x31\x35\x30\x33\x33\x36\x38\
> > x31\x36\x34\x37\x32\x32\x2E\x30\x63\x37\x39\x64\x34\x63\
> > x36\x62\x63\x61\x62\x64\x39\x36\x64\x61\x61\x34\x31\x34\
> > x32\x64\x64\x35\x64\x37\x64\x31\x34\x65\x34\x2E on ho
> > st: hz-hmaster0.xs.163.org.Total scanned row: 87935. Total scanned bytes:
> > 4752238. Total filtered/aggred row: 87934. Time elapsed in EP: 254(ms).
> > Server CPU usage: 0
> > .10531496062992125, server physical mem left: 4.39107584E8, server swap
> > mem left:2.553663488E9.Etc message: start latency: 2@1,agg done@253
> ,compress
> > done@253,server
> > stats done@254, debugGitTag:997bac07cd74f9a3ac9d50714e8740
> ddc2e5c1c7;.Normal
> > Complete: true.Compressed row size: 13
> >
> > the first is older segment and later one is newly build segment, there
> > scan ranges are the same.
> >
> > Here is my hbase log about the empty result :
> >
> > 2017-08-22 11:04:30,378 DEBUG [RpcServer.reader=9,port=60020]
> > security.HBaseSaslRpcServer: SASL server GSSAPI callback: setting
> > canonicalized client ID: nrpt/dev@HADOOP.HZ.NETEASE.COM<mailto:
> dev@HADOOP.HZ.NETEASE.COM>
> > 2017-08-22 11:04:31,714 INFO [Query 25ed5aec-6a9b-4507-bed4-
> 198812cc25f0-226]
> > gridtable.GTScanRequest: pre aggregating results before returning
> > 2017-08-22 11:04:31,714 INFO [Query 25ed5aec-6a9b-4507-bed4-
> 198812cc25f0-226]
> > gridtable.GTAggregateScanner: GTAggregateScanner input rows: 0
> > 2017-08-22 11:04:31,714 INFO [Query 25ed5aec-6a9b-4507-bed4-
> 198812cc25f0-226]
> > endpoint.CubeVisitService: Total scanned 0 rows and 0 bytes
> > 2017-08-22 11:04:31,714 INFO [Query 25ed5aec-6a9b-4507-bed4-
> 198812cc25f0-226]
> > endpoint.CubeVisitService: Size of final result = 8 (0 before
> compressing)
> > 2017-08-22 11:04:31,726 INFO [Query 25ed5aec-6a9b-4507-bed4-
> 198812cc25f0-142]
> > gridtable.GTScanRequest: pre aggregating results before returning
> > 2017-08-22 11:04:31,727 INFO [Query 25ed5aec-6a9b-4507-bed4-
> 198812cc25f0-142]
> > gridtable.GTAggregateScanner: GTAggregateScanner input rows: 0
> > 2017-08-22 11:04:31,727 INFO [Query 25ed5aec-6a9b-4507-bed4-
> 198812cc25f0-142]
> > endpoint.CubeVisitService: Total scanned 0 rows and 0 bytes
> > 2017-08-22 11:04:31,727 INFO [Query 25ed5aec-6a9b-4507-bed4-
> 198812cc25f0-142]
> > endpoint.CubeVisitService: Size of final result = 8 (0 before
> compressing)
> > 2017-08-22 11:04:31,730 INFO [Query 25ed5aec-6a9b-4507-bed4-
> 198812cc25f0-221]
> > gridtable.GTScanRequest: pre aggregating results before returning
> > 2017-08-22 11:04:31,731 INFO [Query 25ed5aec-6a9b-4507-bed4-
> 198812cc25f0-221]
> > gridtable.GTAggregateScanner: GTAggregateScanner input rows: 0
> > 2017-08-22 11:04:31,731 INFO [Query 25ed5aec-6a9b-4507-bed4-
> 198812cc25f0-221]
> > endpoint.CubeVisitService: Total scanned 0 rows and 0 bytes
> > 2017-08-22 11:04:31,731 INFO [Query 25ed5aec-6a9b-4507-bed4-
> 198812cc25f0-221]
> > endpoint.CubeVisitService: Size of final result = 8 (0 before
> compressing)
> > 2017-08-22 11:04:31,740 INFO [Query 25ed5aec-6a9b-4507-bed4-
> 198812cc25f0-194]
> > gridtable.GTScanRequest: pre aggregating results before returning
> > 2017-08-22 11:04:31,740 INFO [Query 25ed5aec-6a9b-4507-bed4-
> 198812cc25f0-194]
> > gridtable.GTAggregateScanner: GTAggregateScanner input rows: 0
> > 2017-08-22 11:04:31,740 INFO [Query 25ed5aec-6a9b-4507-bed4-
> 198812cc25f0-194]
> > endpoint.CubeVisitService: Total scanned 0 rows and 0 bytes
> > 2017-08-22 11:04:31,740 INFO [Query 25ed5aec-6a9b-4507-bed4-
> 198812cc25f0-194]
> > endpoint.CubeVisitService: Size of final result = 8 (0 before
> compressing)
> >
> > I guess maybe some rowkey rule is changed?
> >
> > Did anyone meet the problem? Thanks a lot.
> >
> >
>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
> 本电子邮件可能为保密文件。如果阁下非电子邮件所指定之收件人，谨请立即通知本人。敬请阁下不要使用、保存、复印、打印、散布本电子邮件及其内容，或将其用于其他任何目的或向任何人披露。谢谢您的合作！
> This communication is intended only for the addressee(s) and may contain
> information that is privileged and confidential. You are hereby notified
> that, if you are not an intended recipient listed above, or an authorized
> employee or agent of an addressee of this communication responsible for
> delivering e-mail messages to an intended recipient, any dissemination,
> distribution or reproduction of this communication (including any
> attachments hereto) is strictly prohibited. If you have received this
> communication in error, please notify us immediately by a reply e-mail
> addressed to the sender and permanently delete the original e-mail
> communication and any attachments from all storage devices without making
> or otherwise retaining a copy.
>
--
Best regards,
Shaofeng Shi 史少锋
答复: Kylin query return empty after upgrade to 2.0
Posted by "Donald,Zheng(vip.com)" <do...@vipshop.com>.
Maybe it’s about cuboid scheduler.
Specified query involved only mandatory dimensions will scan the mandatory only cuboid if kylin.cube.aggrgroup.is-mandatory-only-valid=true.
And these cuboids don’t exist before 1.5.3
发件人: ShaoFeng Shi [mailto:shaofengshi@apache.org]
发送时间: 2017年8月23日 16:49
收件人: dev
抄送: user
主题: Re: Kylin query return empty after upgrade to 2.0
Yu has solved this problem by setting this property back to its default value:
kylin.cube.aggrgroup.is-mandatory-only-valid<http://kylin.cube.aggrgroup.is>=false
I don't know the detail; but it seems this property will impact on the execution plan.
2017-08-23 11:27 GMT+08:00 Billy Liu <bi...@apache.org>>:
Query by REST API? or JDBC?
2017-08-22 12:24 GMT+08:00 yu feng <ol...@gmail.com>>:
> Hi, After I upgrade our env. from 1.5.2.1 to 2.0.0, building things go
> well, However, every query to older(generated before upgrading) return
> empty(0 result), and newly build segment return result success.
>
> I add some debug log and find in GTFilterScanner, I get 0 input
> from inputIterator and return 0 to GTAggregateScanner, Here is my kylin log
> :
>
> 2017-08-22 11:46:07,308 INFO [kylin-coproc--pool157-t3958]
> v2.CubeHBaseEndpointRPC:200 : <sub-thread for Query e6e13b48-f28c-422c-b941-dfec2e836ea5
> GTScanRequest 1b
> b7584d>Endpoint RPC returned from HTable NEW_KYLIN_JDVOZ5IBWC Shard
> \x4E\x45\x57\x5F\x4B\x59\x4C\x49\x4E\x5F\x4A\x44\x56\x4F\
> x5A\x35\x49\x42\x57\x43\x2C\x00\x01\x2C\
> x31\x35\x30\x33\x31\x38\x31\x39\x34\x38\x33\x33\x33\x2E\
> x62\x37\x35\x31\x38\x32\x35\x32\x32\x63\x36\x64\x61\x63\
> x36\x39\x32\x33\x33\x63\x33\x32\x30\x38\x36\x34\x62\x
> 39\x38\x31\x37\x62\x2E on host: hz-hbase2.photo.163.org.Total scanned row:
> 0. Total scanned bytes: 0. Total filtered/aggred row: 0. Time elapsed in
> EP: 2(ms). Server
> CPU usage: 0.13636363636363635, server physical mem left: 4.35449856E8,
> server swap mem left:8.589930496E9.Etc message: start latency: 6@0,agg
> done@1,compress done@
> 1,server stats done@2, debugGitTag:997bac07cd74f9a3ac9d50714e8740ddc2e5c1c7;.Normal
> Complete: true.Compressed row size: 8
>
> 2017-08-22 11:46:07,561 INFO [kylin-coproc--pool157-t3961]
> v2.CubeHBaseEndpointRPC:200 : <sub-thread for Query e6e13b48-f28c-422c-b941-dfec2e836ea5
> GTScanRequest 63
> c14bf2>Endpoint RPC returned from HTable KYLIN_0H7MAKPJQL Shard
> \x4B\x59\x4C\x49\x4E\x5F\x30\x48\x37\x4D\x41\x4B\x50\x4A\
> x51\x4C\x2C\x2C\x31\x35\x30\x33\x33\x36\x38\
> x31\x36\x34\x37\x32\x32\x2E\x30\x63\x37\x39\x64\x34\x63\
> x36\x62\x63\x61\x62\x64\x39\x36\x64\x61\x61\x34\x31\x34\
> x32\x64\x64\x35\x64\x37\x64\x31\x34\x65\x34\x2E on ho
> st: hz-hmaster0.xs.163.org.Total scanned row: 87935. Total scanned bytes:
> 4752238. Total filtered/aggred row: 87934. Time elapsed in EP: 254(ms).
> Server CPU usage: 0
> .10531496062992125, server physical mem left: 4.39107584E8, server swap
> mem left:2.553663488E9.Etc message: start latency: 2@1,agg done@253,compress
> done@253,server
> stats done@254, debugGitTag:997bac07cd74f9a3ac9d50714e8740ddc2e5c1c7;.Normal
> Complete: true.Compressed row size: 13
>
> the first is older segment and later one is newly build segment, there
> scan ranges are the same.
>
> Here is my hbase log about the empty result :
>
> 2017-08-22 11:04:30,378 DEBUG [RpcServer.reader=9,port=60020]
> security.HBaseSaslRpcServer: SASL server GSSAPI callback: setting
> canonicalized client ID: nrpt/dev@HADOOP.HZ.NETEASE.COM<ma...@HADOOP.HZ.NETEASE.COM>
> 2017-08-22 11:04:31,714 INFO [Query 25ed5aec-6a9b-4507-bed4-198812cc25f0-226]
> gridtable.GTScanRequest: pre aggregating results before returning
> 2017-08-22 11:04:31,714 INFO [Query 25ed5aec-6a9b-4507-bed4-198812cc25f0-226]
> gridtable.GTAggregateScanner: GTAggregateScanner input rows: 0
> 2017-08-22 11:04:31,714 INFO [Query 25ed5aec-6a9b-4507-bed4-198812cc25f0-226]
> endpoint.CubeVisitService: Total scanned 0 rows and 0 bytes
> 2017-08-22 11:04:31,714 INFO [Query 25ed5aec-6a9b-4507-bed4-198812cc25f0-226]
> endpoint.CubeVisitService: Size of final result = 8 (0 before compressing)
> 2017-08-22 11:04:31,726 INFO [Query 25ed5aec-6a9b-4507-bed4-198812cc25f0-142]
> gridtable.GTScanRequest: pre aggregating results before returning
> 2017-08-22 11:04:31,727 INFO [Query 25ed5aec-6a9b-4507-bed4-198812cc25f0-142]
> gridtable.GTAggregateScanner: GTAggregateScanner input rows: 0
> 2017-08-22 11:04:31,727 INFO [Query 25ed5aec-6a9b-4507-bed4-198812cc25f0-142]
> endpoint.CubeVisitService: Total scanned 0 rows and 0 bytes
> 2017-08-22 11:04:31,727 INFO [Query 25ed5aec-6a9b-4507-bed4-198812cc25f0-142]
> endpoint.CubeVisitService: Size of final result = 8 (0 before compressing)
> 2017-08-22 11:04:31,730 INFO [Query 25ed5aec-6a9b-4507-bed4-198812cc25f0-221]
> gridtable.GTScanRequest: pre aggregating results before returning
> 2017-08-22 11:04:31,731 INFO [Query 25ed5aec-6a9b-4507-bed4-198812cc25f0-221]
> gridtable.GTAggregateScanner: GTAggregateScanner input rows: 0
> 2017-08-22 11:04:31,731 INFO [Query 25ed5aec-6a9b-4507-bed4-198812cc25f0-221]
> endpoint.CubeVisitService: Total scanned 0 rows and 0 bytes
> 2017-08-22 11:04:31,731 INFO [Query 25ed5aec-6a9b-4507-bed4-198812cc25f0-221]
> endpoint.CubeVisitService: Size of final result = 8 (0 before compressing)
> 2017-08-22 11:04:31,740 INFO [Query 25ed5aec-6a9b-4507-bed4-198812cc25f0-194]
> gridtable.GTScanRequest: pre aggregating results before returning
> 2017-08-22 11:04:31,740 INFO [Query 25ed5aec-6a9b-4507-bed4-198812cc25f0-194]
> gridtable.GTAggregateScanner: GTAggregateScanner input rows: 0
> 2017-08-22 11:04:31,740 INFO [Query 25ed5aec-6a9b-4507-bed4-198812cc25f0-194]
> endpoint.CubeVisitService: Total scanned 0 rows and 0 bytes
> 2017-08-22 11:04:31,740 INFO [Query 25ed5aec-6a9b-4507-bed4-198812cc25f0-194]
> endpoint.CubeVisitService: Size of final result = 8 (0 before compressing)
>
> I guess maybe some rowkey rule is changed?
>
> Did anyone meet the problem? Thanks a lot.
>
>
--
Best regards,
Shaofeng Shi 史少锋
本电子邮件可能为保密文件。如果阁下非电子邮件所指定之收件人，谨请立即通知本人。敬请阁下不要使用、保存、复印、打印、散布本电子邮件及其内容，或将其用于其他任何目的或向任何人披露。谢谢您的合作！ This communication is intended only for the addressee(s) and may contain information that is privileged and confidential. You are hereby notified that, if you are not an intended recipient listed above, or an authorized employee or agent of an addressee of this communication responsible for delivering e-mail messages to an intended recipient, any dissemination, distribution or reproduction of this communication (including any attachments hereto) is strictly prohibited. If you have received this communication in error, please notify us immediately by a reply e-mail addressed to the sender and permanently delete the original e-mail communication and any attachments from all storage devices without making or otherwise retaining a copy.
答复: Kylin query return empty after upgrade to 2.0
Posted by "Donald,Zheng(vip.com)" <do...@vipshop.com>.
Maybe it’s about cuboid scheduler.
Specified query involved only mandatory dimensions will scan the mandatory only cuboid if kylin.cube.aggrgroup.is-mandatory-only-valid=true.
And these cuboids don’t exist before 1.5.3
发件人: ShaoFeng Shi [mailto:shaofengshi@apache.org]
发送时间: 2017年8月23日 16:49
收件人: dev
抄送: user
主题: Re: Kylin query return empty after upgrade to 2.0
Yu has solved this problem by setting this property back to its default value:
kylin.cube.aggrgroup.is-mandatory-only-valid<http://kylin.cube.aggrgroup.is>=false
I don't know the detail; but it seems this property will impact on the execution plan.
2017-08-23 11:27 GMT+08:00 Billy Liu <bi...@apache.org>>:
Query by REST API? or JDBC?
2017-08-22 12:24 GMT+08:00 yu feng <ol...@gmail.com>>:
> Hi, After I upgrade our env. from 1.5.2.1 to 2.0.0, building things go
> well, However, every query to older(generated before upgrading) return
> empty(0 result), and newly build segment return result success.
>
> I add some debug log and find in GTFilterScanner, I get 0 input
> from inputIterator and return 0 to GTAggregateScanner, Here is my kylin log
> :
>
> 2017-08-22 11:46:07,308 INFO [kylin-coproc--pool157-t3958]
> v2.CubeHBaseEndpointRPC:200 : <sub-thread for Query e6e13b48-f28c-422c-b941-dfec2e836ea5
> GTScanRequest 1b
> b7584d>Endpoint RPC returned from HTable NEW_KYLIN_JDVOZ5IBWC Shard
> \x4E\x45\x57\x5F\x4B\x59\x4C\x49\x4E\x5F\x4A\x44\x56\x4F\
> x5A\x35\x49\x42\x57\x43\x2C\x00\x01\x2C\
> x31\x35\x30\x33\x31\x38\x31\x39\x34\x38\x33\x33\x33\x2E\
> x62\x37\x35\x31\x38\x32\x35\x32\x32\x63\x36\x64\x61\x63\
> x36\x39\x32\x33\x33\x63\x33\x32\x30\x38\x36\x34\x62\x
> 39\x38\x31\x37\x62\x2E on host: hz-hbase2.photo.163.org.Total scanned row:
> 0. Total scanned bytes: 0. Total filtered/aggred row: 0. Time elapsed in
> EP: 2(ms). Server
> CPU usage: 0.13636363636363635, server physical mem left: 4.35449856E8,
> server swap mem left:8.589930496E9.Etc message: start latency: 6@0,agg
> done@1,compress done@
> 1,server stats done@2, debugGitTag:997bac07cd74f9a3ac9d50714e8740ddc2e5c1c7;.Normal
> Complete: true.Compressed row size: 8
>
> 2017-08-22 11:46:07,561 INFO [kylin-coproc--pool157-t3961]
> v2.CubeHBaseEndpointRPC:200 : <sub-thread for Query e6e13b48-f28c-422c-b941-dfec2e836ea5
> GTScanRequest 63
> c14bf2>Endpoint RPC returned from HTable KYLIN_0H7MAKPJQL Shard
> \x4B\x59\x4C\x49\x4E\x5F\x30\x48\x37\x4D\x41\x4B\x50\x4A\
> x51\x4C\x2C\x2C\x31\x35\x30\x33\x33\x36\x38\
> x31\x36\x34\x37\x32\x32\x2E\x30\x63\x37\x39\x64\x34\x63\
> x36\x62\x63\x61\x62\x64\x39\x36\x64\x61\x61\x34\x31\x34\
> x32\x64\x64\x35\x64\x37\x64\x31\x34\x65\x34\x2E on ho
> st: hz-hmaster0.xs.163.org.Total scanned row: 87935. Total scanned bytes:
> 4752238. Total filtered/aggred row: 87934. Time elapsed in EP: 254(ms).
> Server CPU usage: 0
> .10531496062992125, server physical mem left: 4.39107584E8, server swap
> mem left:2.553663488E9.Etc message: start latency: 2@1,agg done@253,compress
> done@253,server
> stats done@254, debugGitTag:997bac07cd74f9a3ac9d50714e8740ddc2e5c1c7;.Normal
> Complete: true.Compressed row size: 13
>
> the first is older segment and later one is newly build segment, there
> scan ranges are the same.
>
> Here is my hbase log about the empty result :
>
> 2017-08-22 11:04:30,378 DEBUG [RpcServer.reader=9,port=60020]
> security.HBaseSaslRpcServer: SASL server GSSAPI callback: setting
> canonicalized client ID: nrpt/dev@HADOOP.HZ.NETEASE.COM<ma...@HADOOP.HZ.NETEASE.COM>
> 2017-08-22 11:04:31,714 INFO [Query 25ed5aec-6a9b-4507-bed4-198812cc25f0-226]
> gridtable.GTScanRequest: pre aggregating results before returning
> 2017-08-22 11:04:31,714 INFO [Query 25ed5aec-6a9b-4507-bed4-198812cc25f0-226]
> gridtable.GTAggregateScanner: GTAggregateScanner input rows: 0
> 2017-08-22 11:04:31,714 INFO [Query 25ed5aec-6a9b-4507-bed4-198812cc25f0-226]
> endpoint.CubeVisitService: Total scanned 0 rows and 0 bytes
> 2017-08-22 11:04:31,714 INFO [Query 25ed5aec-6a9b-4507-bed4-198812cc25f0-226]
> endpoint.CubeVisitService: Size of final result = 8 (0 before compressing)
> 2017-08-22 11:04:31,726 INFO [Query 25ed5aec-6a9b-4507-bed4-198812cc25f0-142]
> gridtable.GTScanRequest: pre aggregating results before returning
> 2017-08-22 11:04:31,727 INFO [Query 25ed5aec-6a9b-4507-bed4-198812cc25f0-142]
> gridtable.GTAggregateScanner: GTAggregateScanner input rows: 0
> 2017-08-22 11:04:31,727 INFO [Query 25ed5aec-6a9b-4507-bed4-198812cc25f0-142]
> endpoint.CubeVisitService: Total scanned 0 rows and 0 bytes
> 2017-08-22 11:04:31,727 INFO [Query 25ed5aec-6a9b-4507-bed4-198812cc25f0-142]
> endpoint.CubeVisitService: Size of final result = 8 (0 before compressing)
> 2017-08-22 11:04:31,730 INFO [Query 25ed5aec-6a9b-4507-bed4-198812cc25f0-221]
> gridtable.GTScanRequest: pre aggregating results before returning
> 2017-08-22 11:04:31,731 INFO [Query 25ed5aec-6a9b-4507-bed4-198812cc25f0-221]
> gridtable.GTAggregateScanner: GTAggregateScanner input rows: 0
> 2017-08-22 11:04:31,731 INFO [Query 25ed5aec-6a9b-4507-bed4-198812cc25f0-221]
> endpoint.CubeVisitService: Total scanned 0 rows and 0 bytes
> 2017-08-22 11:04:31,731 INFO [Query 25ed5aec-6a9b-4507-bed4-198812cc25f0-221]
> endpoint.CubeVisitService: Size of final result = 8 (0 before compressing)
> 2017-08-22 11:04:31,740 INFO [Query 25ed5aec-6a9b-4507-bed4-198812cc25f0-194]
> gridtable.GTScanRequest: pre aggregating results before returning
> 2017-08-22 11:04:31,740 INFO [Query 25ed5aec-6a9b-4507-bed4-198812cc25f0-194]
> gridtable.GTAggregateScanner: GTAggregateScanner input rows: 0
> 2017-08-22 11:04:31,740 INFO [Query 25ed5aec-6a9b-4507-bed4-198812cc25f0-194]
> endpoint.CubeVisitService: Total scanned 0 rows and 0 bytes
> 2017-08-22 11:04:31,740 INFO [Query 25ed5aec-6a9b-4507-bed4-198812cc25f0-194]
> endpoint.CubeVisitService: Size of final result = 8 (0 before compressing)
>
> I guess maybe some rowkey rule is changed?
>
> Did anyone meet the problem? Thanks a lot.
>
>
--
Best regards,
Shaofeng Shi 史少锋
本电子邮件可能为保密文件。如果阁下非电子邮件所指定之收件人，谨请立即通知本人。敬请阁下不要使用、保存、复印、打印、散布本电子邮件及其内容，或将其用于其他任何目的或向任何人披露。谢谢您的合作！ This communication is intended only for the addressee(s) and may contain information that is privileged and confidential. You are hereby notified that, if you are not an intended recipient listed above, or an authorized employee or agent of an addressee of this communication responsible for delivering e-mail messages to an intended recipient, any dissemination, distribution or reproduction of this communication (including any attachments hereto) is strictly prohibited. If you have received this communication in error, please notify us immediately by a reply e-mail addressed to the sender and permanently delete the original e-mail communication and any attachments from all storage devices without making or otherwise retaining a copy.
Re: Kylin query return empty after upgrade to 2.0
Posted by ShaoFeng Shi <sh...@apache.org>.
Yu has solved this problem by setting this property back to its default
value:
kylin.cube.aggrgroup.is-mandatory-only-valid
<http://kylin.cube.aggrgroup.is>=false
I don't know the detail; but it seems this property will impact on the
execution plan.
2017-08-23 11:27 GMT+08:00 Billy Liu <bi...@apache.org>:
> Query by REST API? or JDBC?
>
> 2017-08-22 12:24 GMT+08:00 yu feng <ol...@gmail.com>:
>
> > Hi, After I upgrade our env. from 1.5.2.1 to 2.0.0, building things go
> > well, However, every query to older(generated before upgrading) return
> > empty(0 result), and newly build segment return result success.
> >
> > I add some debug log and find in GTFilterScanner, I get 0 input
> > from inputIterator and return 0 to GTAggregateScanner, Here is my kylin
> log
> > :
> >
> > 2017-08-22 11:46:07,308 INFO [kylin-coproc--pool157-t3958]
> > v2.CubeHBaseEndpointRPC:200 : <sub-thread for Query
> e6e13b48-f28c-422c-b941-dfec2e836ea5
> > GTScanRequest 1b
> > b7584d>Endpoint RPC returned from HTable NEW_KYLIN_JDVOZ5IBWC Shard
> > \x4E\x45\x57\x5F\x4B\x59\x4C\x49\x4E\x5F\x4A\x44\x56\x4F\
> > x5A\x35\x49\x42\x57\x43\x2C\x00\x01\x2C\
> > x31\x35\x30\x33\x31\x38\x31\x39\x34\x38\x33\x33\x33\x2E\
> > x62\x37\x35\x31\x38\x32\x35\x32\x32\x63\x36\x64\x61\x63\
> > x36\x39\x32\x33\x33\x63\x33\x32\x30\x38\x36\x34\x62\x
> > 39\x38\x31\x37\x62\x2E on host: hz-hbase2.photo.163.org.Total scanned
> row:
> > 0. Total scanned bytes: 0. Total filtered/aggred row: 0. Time elapsed in
> > EP: 2(ms). Server
> > CPU usage: 0.13636363636363635, server physical mem left: 4.35449856E8,
> > server swap mem left:8.589930496E9.Etc message: start latency: 6@0,agg
> > done@1,compress done@
> > 1,server stats done@2, debugGitTag:997bac07cd74f9a3ac9d50714e8740
> ddc2e5c1c7;.Normal
> > Complete: true.Compressed row size: 8
> >
> > 2017-08-22 11:46:07,561 INFO [kylin-coproc--pool157-t3961]
> > v2.CubeHBaseEndpointRPC:200 : <sub-thread for Query
> e6e13b48-f28c-422c-b941-dfec2e836ea5
> > GTScanRequest 63
> > c14bf2>Endpoint RPC returned from HTable KYLIN_0H7MAKPJQL Shard
> > \x4B\x59\x4C\x49\x4E\x5F\x30\x48\x37\x4D\x41\x4B\x50\x4A\
> > x51\x4C\x2C\x2C\x31\x35\x30\x33\x33\x36\x38\
> > x31\x36\x34\x37\x32\x32\x2E\x30\x63\x37\x39\x64\x34\x63\
> > x36\x62\x63\x61\x62\x64\x39\x36\x64\x61\x61\x34\x31\x34\
> > x32\x64\x64\x35\x64\x37\x64\x31\x34\x65\x34\x2E on ho
> > st: hz-hmaster0.xs.163.org.Total scanned row: 87935. Total scanned bytes:
> > 4752238. Total filtered/aggred row: 87934. Time elapsed in EP: 254(ms).
> > Server CPU usage: 0
> > .10531496062992125, server physical mem left: 4.39107584E8, server swap
> > mem left:2.553663488E9.Etc message: start latency: 2@1,agg done@253
> ,compress
> > done@253,server
> > stats done@254, debugGitTag:997bac07cd74f9a3ac9d50714e8740
> ddc2e5c1c7;.Normal
> > Complete: true.Compressed row size: 13
> >
> > the first is older segment and later one is newly build segment, there
> > scan ranges are the same.
> >
> > Here is my hbase log about the empty result :
> >
> > 2017-08-22 11:04:30,378 DEBUG [RpcServer.reader=9,port=60020]
> > security.HBaseSaslRpcServer: SASL server GSSAPI callback: setting
> > canonicalized client ID: nrpt/dev@HADOOP.HZ.NETEASE.COM
> > 2017-08-22 11:04:31,714 INFO [Query 25ed5aec-6a9b-4507-bed4-
> 198812cc25f0-226]
> > gridtable.GTScanRequest: pre aggregating results before returning
> > 2017-08-22 11:04:31,714 INFO [Query 25ed5aec-6a9b-4507-bed4-
> 198812cc25f0-226]
> > gridtable.GTAggregateScanner: GTAggregateScanner input rows: 0
> > 2017-08-22 11:04:31,714 INFO [Query 25ed5aec-6a9b-4507-bed4-
> 198812cc25f0-226]
> > endpoint.CubeVisitService: Total scanned 0 rows and 0 bytes
> > 2017-08-22 11:04:31,714 INFO [Query 25ed5aec-6a9b-4507-bed4-
> 198812cc25f0-226]
> > endpoint.CubeVisitService: Size of final result = 8 (0 before
> compressing)
> > 2017-08-22 11:04:31,726 INFO [Query 25ed5aec-6a9b-4507-bed4-
> 198812cc25f0-142]
> > gridtable.GTScanRequest: pre aggregating results before returning
> > 2017-08-22 11:04:31,727 INFO [Query 25ed5aec-6a9b-4507-bed4-
> 198812cc25f0-142]
> > gridtable.GTAggregateScanner: GTAggregateScanner input rows: 0
> > 2017-08-22 11:04:31,727 INFO [Query 25ed5aec-6a9b-4507-bed4-
> 198812cc25f0-142]
> > endpoint.CubeVisitService: Total scanned 0 rows and 0 bytes
> > 2017-08-22 11:04:31,727 INFO [Query 25ed5aec-6a9b-4507-bed4-
> 198812cc25f0-142]
> > endpoint.CubeVisitService: Size of final result = 8 (0 before
> compressing)
> > 2017-08-22 11:04:31,730 INFO [Query 25ed5aec-6a9b-4507-bed4-
> 198812cc25f0-221]
> > gridtable.GTScanRequest: pre aggregating results before returning
> > 2017-08-22 11:04:31,731 INFO [Query 25ed5aec-6a9b-4507-bed4-
> 198812cc25f0-221]
> > gridtable.GTAggregateScanner: GTAggregateScanner input rows: 0
> > 2017-08-22 11:04:31,731 INFO [Query 25ed5aec-6a9b-4507-bed4-
> 198812cc25f0-221]
> > endpoint.CubeVisitService: Total scanned 0 rows and 0 bytes
> > 2017-08-22 11:04:31,731 INFO [Query 25ed5aec-6a9b-4507-bed4-
> 198812cc25f0-221]
> > endpoint.CubeVisitService: Size of final result = 8 (0 before
> compressing)
> > 2017-08-22 11:04:31,740 INFO [Query 25ed5aec-6a9b-4507-bed4-
> 198812cc25f0-194]
> > gridtable.GTScanRequest: pre aggregating results before returning
> > 2017-08-22 11:04:31,740 INFO [Query 25ed5aec-6a9b-4507-bed4-
> 198812cc25f0-194]
> > gridtable.GTAggregateScanner: GTAggregateScanner input rows: 0
> > 2017-08-22 11:04:31,740 INFO [Query 25ed5aec-6a9b-4507-bed4-
> 198812cc25f0-194]
> > endpoint.CubeVisitService: Total scanned 0 rows and 0 bytes
> > 2017-08-22 11:04:31,740 INFO [Query 25ed5aec-6a9b-4507-bed4-
> 198812cc25f0-194]
> > endpoint.CubeVisitService: Size of final result = 8 (0 before
> compressing)
> >
> > I guess maybe some rowkey rule is changed?
> >
> > Did anyone meet the problem? Thanks a lot.
> >
> >
>
--
Best regards,
Shaofeng Shi 史少锋
Re: Kylin query return empty after upgrade to 2.0
Posted by ShaoFeng Shi <sh...@apache.org>.
Yu has solved this problem by setting this property back to its default
value:
kylin.cube.aggrgroup.is-mandatory-only-valid
<http://kylin.cube.aggrgroup.is>=false
I don't know the detail; but it seems this property will impact on the
execution plan.
2017-08-23 11:27 GMT+08:00 Billy Liu <bi...@apache.org>:
> Query by REST API? or JDBC?
>
> 2017-08-22 12:24 GMT+08:00 yu feng <ol...@gmail.com>:
>
> > Hi, After I upgrade our env. from 1.5.2.1 to 2.0.0, building things go
> > well, However, every query to older(generated before upgrading) return
> > empty(0 result), and newly build segment return result success.
> >
> > I add some debug log and find in GTFilterScanner, I get 0 input
> > from inputIterator and return 0 to GTAggregateScanner, Here is my kylin
> log
> > :
> >
> > 2017-08-22 11:46:07,308 INFO [kylin-coproc--pool157-t3958]
> > v2.CubeHBaseEndpointRPC:200 : <sub-thread for Query
> e6e13b48-f28c-422c-b941-dfec2e836ea5
> > GTScanRequest 1b
> > b7584d>Endpoint RPC returned from HTable NEW_KYLIN_JDVOZ5IBWC Shard
> > \x4E\x45\x57\x5F\x4B\x59\x4C\x49\x4E\x5F\x4A\x44\x56\x4F\
> > x5A\x35\x49\x42\x57\x43\x2C\x00\x01\x2C\
> > x31\x35\x30\x33\x31\x38\x31\x39\x34\x38\x33\x33\x33\x2E\
> > x62\x37\x35\x31\x38\x32\x35\x32\x32\x63\x36\x64\x61\x63\
> > x36\x39\x32\x33\x33\x63\x33\x32\x30\x38\x36\x34\x62\x
> > 39\x38\x31\x37\x62\x2E on host: hz-hbase2.photo.163.org.Total scanned
> row:
> > 0. Total scanned bytes: 0. Total filtered/aggred row: 0. Time elapsed in
> > EP: 2(ms). Server
> > CPU usage: 0.13636363636363635, server physical mem left: 4.35449856E8,
> > server swap mem left:8.589930496E9.Etc message: start latency: 6@0,agg
> > done@1,compress done@
> > 1,server stats done@2, debugGitTag:997bac07cd74f9a3ac9d50714e8740
> ddc2e5c1c7;.Normal
> > Complete: true.Compressed row size: 8
> >
> > 2017-08-22 11:46:07,561 INFO [kylin-coproc--pool157-t3961]
> > v2.CubeHBaseEndpointRPC:200 : <sub-thread for Query
> e6e13b48-f28c-422c-b941-dfec2e836ea5
> > GTScanRequest 63
> > c14bf2>Endpoint RPC returned from HTable KYLIN_0H7MAKPJQL Shard
> > \x4B\x59\x4C\x49\x4E\x5F\x30\x48\x37\x4D\x41\x4B\x50\x4A\
> > x51\x4C\x2C\x2C\x31\x35\x30\x33\x33\x36\x38\
> > x31\x36\x34\x37\x32\x32\x2E\x30\x63\x37\x39\x64\x34\x63\
> > x36\x62\x63\x61\x62\x64\x39\x36\x64\x61\x61\x34\x31\x34\
> > x32\x64\x64\x35\x64\x37\x64\x31\x34\x65\x34\x2E on ho
> > st: hz-hmaster0.xs.163.org.Total scanned row: 87935. Total scanned bytes:
> > 4752238. Total filtered/aggred row: 87934. Time elapsed in EP: 254(ms).
> > Server CPU usage: 0
> > .10531496062992125, server physical mem left: 4.39107584E8, server swap
> > mem left:2.553663488E9.Etc message: start latency: 2@1,agg done@253
> ,compress
> > done@253,server
> > stats done@254, debugGitTag:997bac07cd74f9a3ac9d50714e8740
> ddc2e5c1c7;.Normal
> > Complete: true.Compressed row size: 13
> >
> > the first is older segment and later one is newly build segment, there
> > scan ranges are the same.
> >
> > Here is my hbase log about the empty result :
> >
> > 2017-08-22 11:04:30,378 DEBUG [RpcServer.reader=9,port=60020]
> > security.HBaseSaslRpcServer: SASL server GSSAPI callback: setting
> > canonicalized client ID: nrpt/dev@HADOOP.HZ.NETEASE.COM
> > 2017-08-22 11:04:31,714 INFO [Query 25ed5aec-6a9b-4507-bed4-
> 198812cc25f0-226]
> > gridtable.GTScanRequest: pre aggregating results before returning
> > 2017-08-22 11:04:31,714 INFO [Query 25ed5aec-6a9b-4507-bed4-
> 198812cc25f0-226]
> > gridtable.GTAggregateScanner: GTAggregateScanner input rows: 0
> > 2017-08-22 11:04:31,714 INFO [Query 25ed5aec-6a9b-4507-bed4-
> 198812cc25f0-226]
> > endpoint.CubeVisitService: Total scanned 0 rows and 0 bytes
> > 2017-08-22 11:04:31,714 INFO [Query 25ed5aec-6a9b-4507-bed4-
> 198812cc25f0-226]
> > endpoint.CubeVisitService: Size of final result = 8 (0 before
> compressing)
> > 2017-08-22 11:04:31,726 INFO [Query 25ed5aec-6a9b-4507-bed4-
> 198812cc25f0-142]
> > gridtable.GTScanRequest: pre aggregating results before returning
> > 2017-08-22 11:04:31,727 INFO [Query 25ed5aec-6a9b-4507-bed4-
> 198812cc25f0-142]
> > gridtable.GTAggregateScanner: GTAggregateScanner input rows: 0
> > 2017-08-22 11:04:31,727 INFO [Query 25ed5aec-6a9b-4507-bed4-
> 198812cc25f0-142]
> > endpoint.CubeVisitService: Total scanned 0 rows and 0 bytes
> > 2017-08-22 11:04:31,727 INFO [Query 25ed5aec-6a9b-4507-bed4-
> 198812cc25f0-142]
> > endpoint.CubeVisitService: Size of final result = 8 (0 before
> compressing)
> > 2017-08-22 11:04:31,730 INFO [Query 25ed5aec-6a9b-4507-bed4-
> 198812cc25f0-221]
> > gridtable.GTScanRequest: pre aggregating results before returning
> > 2017-08-22 11:04:31,731 INFO [Query 25ed5aec-6a9b-4507-bed4-
> 198812cc25f0-221]
> > gridtable.GTAggregateScanner: GTAggregateScanner input rows: 0
> > 2017-08-22 11:04:31,731 INFO [Query 25ed5aec-6a9b-4507-bed4-
> 198812cc25f0-221]
> > endpoint.CubeVisitService: Total scanned 0 rows and 0 bytes
> > 2017-08-22 11:04:31,731 INFO [Query 25ed5aec-6a9b-4507-bed4-
> 198812cc25f0-221]
> > endpoint.CubeVisitService: Size of final result = 8 (0 before
> compressing)
> > 2017-08-22 11:04:31,740 INFO [Query 25ed5aec-6a9b-4507-bed4-
> 198812cc25f0-194]
> > gridtable.GTScanRequest: pre aggregating results before returning
> > 2017-08-22 11:04:31,740 INFO [Query 25ed5aec-6a9b-4507-bed4-
> 198812cc25f0-194]
> > gridtable.GTAggregateScanner: GTAggregateScanner input rows: 0
> > 2017-08-22 11:04:31,740 INFO [Query 25ed5aec-6a9b-4507-bed4-
> 198812cc25f0-194]
> > endpoint.CubeVisitService: Total scanned 0 rows and 0 bytes
> > 2017-08-22 11:04:31,740 INFO [Query 25ed5aec-6a9b-4507-bed4-
> 198812cc25f0-194]
> > endpoint.CubeVisitService: Size of final result = 8 (0 before
> compressing)
> >
> > I guess maybe some rowkey rule is changed?
> >
> > Did anyone meet the problem? Thanks a lot.
> >
> >
>
--
Best regards,
Shaofeng Shi 史少锋
Re: Kylin query return empty after upgrade to 2.0
Posted by Billy Liu <bi...@apache.org>.
Query by REST API? or JDBC?
2017-08-22 12:24 GMT+08:00 yu feng <ol...@gmail.com>:
> Hi, After I upgrade our env. from 1.5.2.1 to 2.0.0, building things go
> well, However, every query to older(generated before upgrading) return
> empty(0 result), and newly build segment return result success.
>
> I add some debug log and find in GTFilterScanner, I get 0 input
> from inputIterator and return 0 to GTAggregateScanner, Here is my kylin log
> :
>
> 2017-08-22 11:46:07,308 INFO [kylin-coproc--pool157-t3958]
> v2.CubeHBaseEndpointRPC:200 : <sub-thread for Query e6e13b48-f28c-422c-b941-dfec2e836ea5
> GTScanRequest 1b
> b7584d>Endpoint RPC returned from HTable NEW_KYLIN_JDVOZ5IBWC Shard
> \x4E\x45\x57\x5F\x4B\x59\x4C\x49\x4E\x5F\x4A\x44\x56\x4F\
> x5A\x35\x49\x42\x57\x43\x2C\x00\x01\x2C\
> x31\x35\x30\x33\x31\x38\x31\x39\x34\x38\x33\x33\x33\x2E\
> x62\x37\x35\x31\x38\x32\x35\x32\x32\x63\x36\x64\x61\x63\
> x36\x39\x32\x33\x33\x63\x33\x32\x30\x38\x36\x34\x62\x
> 39\x38\x31\x37\x62\x2E on host: hz-hbase2.photo.163.org.Total scanned row:
> 0. Total scanned bytes: 0. Total filtered/aggred row: 0. Time elapsed in
> EP: 2(ms). Server
> CPU usage: 0.13636363636363635, server physical mem left: 4.35449856E8,
> server swap mem left:8.589930496E9.Etc message: start latency: 6@0,agg
> done@1,compress done@
> 1,server stats done@2, debugGitTag:997bac07cd74f9a3ac9d50714e8740ddc2e5c1c7;.Normal
> Complete: true.Compressed row size: 8
>
> 2017-08-22 11:46:07,561 INFO [kylin-coproc--pool157-t3961]
> v2.CubeHBaseEndpointRPC:200 : <sub-thread for Query e6e13b48-f28c-422c-b941-dfec2e836ea5
> GTScanRequest 63
> c14bf2>Endpoint RPC returned from HTable KYLIN_0H7MAKPJQL Shard
> \x4B\x59\x4C\x49\x4E\x5F\x30\x48\x37\x4D\x41\x4B\x50\x4A\
> x51\x4C\x2C\x2C\x31\x35\x30\x33\x33\x36\x38\
> x31\x36\x34\x37\x32\x32\x2E\x30\x63\x37\x39\x64\x34\x63\
> x36\x62\x63\x61\x62\x64\x39\x36\x64\x61\x61\x34\x31\x34\
> x32\x64\x64\x35\x64\x37\x64\x31\x34\x65\x34\x2E on ho
> st: hz-hmaster0.xs.163.org.Total scanned row: 87935. Total scanned bytes:
> 4752238. Total filtered/aggred row: 87934. Time elapsed in EP: 254(ms).
> Server CPU usage: 0
> .10531496062992125, server physical mem left: 4.39107584E8, server swap
> mem left:2.553663488E9.Etc message: start latency: 2@1,agg done@253,compress
> done@253,server
> stats done@254, debugGitTag:997bac07cd74f9a3ac9d50714e8740ddc2e5c1c7;.Normal
> Complete: true.Compressed row size: 13
>
> the first is older segment and later one is newly build segment, there
> scan ranges are the same.
>
> Here is my hbase log about the empty result :
>
> 2017-08-22 11:04:30,378 DEBUG [RpcServer.reader=9,port=60020]
> security.HBaseSaslRpcServer: SASL server GSSAPI callback: setting
> canonicalized client ID: nrpt/dev@HADOOP.HZ.NETEASE.COM
> 2017-08-22 11:04:31,714 INFO [Query 25ed5aec-6a9b-4507-bed4-198812cc25f0-226]
> gridtable.GTScanRequest: pre aggregating results before returning
> 2017-08-22 11:04:31,714 INFO [Query 25ed5aec-6a9b-4507-bed4-198812cc25f0-226]
> gridtable.GTAggregateScanner: GTAggregateScanner input rows: 0
> 2017-08-22 11:04:31,714 INFO [Query 25ed5aec-6a9b-4507-bed4-198812cc25f0-226]
> endpoint.CubeVisitService: Total scanned 0 rows and 0 bytes
> 2017-08-22 11:04:31,714 INFO [Query 25ed5aec-6a9b-4507-bed4-198812cc25f0-226]
> endpoint.CubeVisitService: Size of final result = 8 (0 before compressing)
> 2017-08-22 11:04:31,726 INFO [Query 25ed5aec-6a9b-4507-bed4-198812cc25f0-142]
> gridtable.GTScanRequest: pre aggregating results before returning
> 2017-08-22 11:04:31,727 INFO [Query 25ed5aec-6a9b-4507-bed4-198812cc25f0-142]
> gridtable.GTAggregateScanner: GTAggregateScanner input rows: 0
> 2017-08-22 11:04:31,727 INFO [Query 25ed5aec-6a9b-4507-bed4-198812cc25f0-142]
> endpoint.CubeVisitService: Total scanned 0 rows and 0 bytes
> 2017-08-22 11:04:31,727 INFO [Query 25ed5aec-6a9b-4507-bed4-198812cc25f0-142]
> endpoint.CubeVisitService: Size of final result = 8 (0 before compressing)
> 2017-08-22 11:04:31,730 INFO [Query 25ed5aec-6a9b-4507-bed4-198812cc25f0-221]
> gridtable.GTScanRequest: pre aggregating results before returning
> 2017-08-22 11:04:31,731 INFO [Query 25ed5aec-6a9b-4507-bed4-198812cc25f0-221]
> gridtable.GTAggregateScanner: GTAggregateScanner input rows: 0
> 2017-08-22 11:04:31,731 INFO [Query 25ed5aec-6a9b-4507-bed4-198812cc25f0-221]
> endpoint.CubeVisitService: Total scanned 0 rows and 0 bytes
> 2017-08-22 11:04:31,731 INFO [Query 25ed5aec-6a9b-4507-bed4-198812cc25f0-221]
> endpoint.CubeVisitService: Size of final result = 8 (0 before compressing)
> 2017-08-22 11:04:31,740 INFO [Query 25ed5aec-6a9b-4507-bed4-198812cc25f0-194]
> gridtable.GTScanRequest: pre aggregating results before returning
> 2017-08-22 11:04:31,740 INFO [Query 25ed5aec-6a9b-4507-bed4-198812cc25f0-194]
> gridtable.GTAggregateScanner: GTAggregateScanner input rows: 0
> 2017-08-22 11:04:31,740 INFO [Query 25ed5aec-6a9b-4507-bed4-198812cc25f0-194]
> endpoint.CubeVisitService: Total scanned 0 rows and 0 bytes
> 2017-08-22 11:04:31,740 INFO [Query 25ed5aec-6a9b-4507-bed4-198812cc25f0-194]
> endpoint.CubeVisitService: Size of final result = 8 (0 before compressing)
>
> I guess maybe some rowkey rule is changed?
>
> Did anyone meet the problem? Thanks a lot.
>
>
Re: Kylin query return empty after upgrade to 2.0
Posted by Billy Liu <bi...@apache.org>.
Query by REST API? or JDBC?
2017-08-22 12:24 GMT+08:00 yu feng <ol...@gmail.com>:
> Hi, After I upgrade our env. from 1.5.2.1 to 2.0.0, building things go
> well, However, every query to older(generated before upgrading) return
> empty(0 result), and newly build segment return result success.
>
> I add some debug log and find in GTFilterScanner, I get 0 input
> from inputIterator and return 0 to GTAggregateScanner, Here is my kylin log
> :
>
> 2017-08-22 11:46:07,308 INFO [kylin-coproc--pool157-t3958]
> v2.CubeHBaseEndpointRPC:200 : <sub-thread for Query e6e13b48-f28c-422c-b941-dfec2e836ea5
> GTScanRequest 1b
> b7584d>Endpoint RPC returned from HTable NEW_KYLIN_JDVOZ5IBWC Shard
> \x4E\x45\x57\x5F\x4B\x59\x4C\x49\x4E\x5F\x4A\x44\x56\x4F\
> x5A\x35\x49\x42\x57\x43\x2C\x00\x01\x2C\
> x31\x35\x30\x33\x31\x38\x31\x39\x34\x38\x33\x33\x33\x2E\
> x62\x37\x35\x31\x38\x32\x35\x32\x32\x63\x36\x64\x61\x63\
> x36\x39\x32\x33\x33\x63\x33\x32\x30\x38\x36\x34\x62\x
> 39\x38\x31\x37\x62\x2E on host: hz-hbase2.photo.163.org.Total scanned row:
> 0. Total scanned bytes: 0. Total filtered/aggred row: 0. Time elapsed in
> EP: 2(ms). Server
> CPU usage: 0.13636363636363635, server physical mem left: 4.35449856E8,
> server swap mem left:8.589930496E9.Etc message: start latency: 6@0,agg
> done@1,compress done@
> 1,server stats done@2, debugGitTag:997bac07cd74f9a3ac9d50714e8740ddc2e5c1c7;.Normal
> Complete: true.Compressed row size: 8
>
> 2017-08-22 11:46:07,561 INFO [kylin-coproc--pool157-t3961]
> v2.CubeHBaseEndpointRPC:200 : <sub-thread for Query e6e13b48-f28c-422c-b941-dfec2e836ea5
> GTScanRequest 63
> c14bf2>Endpoint RPC returned from HTable KYLIN_0H7MAKPJQL Shard
> \x4B\x59\x4C\x49\x4E\x5F\x30\x48\x37\x4D\x41\x4B\x50\x4A\
> x51\x4C\x2C\x2C\x31\x35\x30\x33\x33\x36\x38\
> x31\x36\x34\x37\x32\x32\x2E\x30\x63\x37\x39\x64\x34\x63\
> x36\x62\x63\x61\x62\x64\x39\x36\x64\x61\x61\x34\x31\x34\
> x32\x64\x64\x35\x64\x37\x64\x31\x34\x65\x34\x2E on ho
> st: hz-hmaster0.xs.163.org.Total scanned row: 87935. Total scanned bytes:
> 4752238. Total filtered/aggred row: 87934. Time elapsed in EP: 254(ms).
> Server CPU usage: 0
> .10531496062992125, server physical mem left: 4.39107584E8, server swap
> mem left:2.553663488E9.Etc message: start latency: 2@1,agg done@253,compress
> done@253,server
> stats done@254, debugGitTag:997bac07cd74f9a3ac9d50714e8740ddc2e5c1c7;.Normal
> Complete: true.Compressed row size: 13
>
> the first is older segment and later one is newly build segment, there
> scan ranges are the same.
>
> Here is my hbase log about the empty result :
>
> 2017-08-22 11:04:30,378 DEBUG [RpcServer.reader=9,port=60020]
> security.HBaseSaslRpcServer: SASL server GSSAPI callback: setting
> canonicalized client ID: nrpt/dev@HADOOP.HZ.NETEASE.COM
> 2017-08-22 11:04:31,714 INFO [Query 25ed5aec-6a9b-4507-bed4-198812cc25f0-226]
> gridtable.GTScanRequest: pre aggregating results before returning
> 2017-08-22 11:04:31,714 INFO [Query 25ed5aec-6a9b-4507-bed4-198812cc25f0-226]
> gridtable.GTAggregateScanner: GTAggregateScanner input rows: 0
> 2017-08-22 11:04:31,714 INFO [Query 25ed5aec-6a9b-4507-bed4-198812cc25f0-226]
> endpoint.CubeVisitService: Total scanned 0 rows and 0 bytes
> 2017-08-22 11:04:31,714 INFO [Query 25ed5aec-6a9b-4507-bed4-198812cc25f0-226]
> endpoint.CubeVisitService: Size of final result = 8 (0 before compressing)
> 2017-08-22 11:04:31,726 INFO [Query 25ed5aec-6a9b-4507-bed4-198812cc25f0-142]
> gridtable.GTScanRequest: pre aggregating results before returning
> 2017-08-22 11:04:31,727 INFO [Query 25ed5aec-6a9b-4507-bed4-198812cc25f0-142]
> gridtable.GTAggregateScanner: GTAggregateScanner input rows: 0
> 2017-08-22 11:04:31,727 INFO [Query 25ed5aec-6a9b-4507-bed4-198812cc25f0-142]
> endpoint.CubeVisitService: Total scanned 0 rows and 0 bytes
> 2017-08-22 11:04:31,727 INFO [Query 25ed5aec-6a9b-4507-bed4-198812cc25f0-142]
> endpoint.CubeVisitService: Size of final result = 8 (0 before compressing)
> 2017-08-22 11:04:31,730 INFO [Query 25ed5aec-6a9b-4507-bed4-198812cc25f0-221]
> gridtable.GTScanRequest: pre aggregating results before returning
> 2017-08-22 11:04:31,731 INFO [Query 25ed5aec-6a9b-4507-bed4-198812cc25f0-221]
> gridtable.GTAggregateScanner: GTAggregateScanner input rows: 0
> 2017-08-22 11:04:31,731 INFO [Query 25ed5aec-6a9b-4507-bed4-198812cc25f0-221]
> endpoint.CubeVisitService: Total scanned 0 rows and 0 bytes
> 2017-08-22 11:04:31,731 INFO [Query 25ed5aec-6a9b-4507-bed4-198812cc25f0-221]
> endpoint.CubeVisitService: Size of final result = 8 (0 before compressing)
> 2017-08-22 11:04:31,740 INFO [Query 25ed5aec-6a9b-4507-bed4-198812cc25f0-194]
> gridtable.GTScanRequest: pre aggregating results before returning
> 2017-08-22 11:04:31,740 INFO [Query 25ed5aec-6a9b-4507-bed4-198812cc25f0-194]
> gridtable.GTAggregateScanner: GTAggregateScanner input rows: 0
> 2017-08-22 11:04:31,740 INFO [Query 25ed5aec-6a9b-4507-bed4-198812cc25f0-194]
> endpoint.CubeVisitService: Total scanned 0 rows and 0 bytes
> 2017-08-22 11:04:31,740 INFO [Query 25ed5aec-6a9b-4507-bed4-198812cc25f0-194]
> endpoint.CubeVisitService: Size of final result = 8 (0 before compressing)
>
> I guess maybe some rowkey rule is changed?
>
> Did anyone meet the problem? Thanks a lot.
>
>