You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kylin.apache.org by nichunen <ni...@mininglamp.com> on 2015/11/11 10:32:04 UTC
Problem during stress test for kylin quey
Hi,
We did a stress test for our kylin server with 100 concurrent queies. It worked fine at first. But after 1 day, we can't query kylin any more, and there is log like "query has been running 80396 seconds", many "bad queries" were hung there. Hbase nodes were still alive, and the cubes and jobs could still be listed on the pages. To make sure whether it was the problem of hbase, I restarted hbase, and did a new query, no log from region server shown hbase received the query, for as we know, a successful query will create log like "Klin Coprocessor start; Klin Coprocessor aggregation done". And from the kylin.log, there were still queries hung.
Do you know what caused the problem? In our opnion, it may be because:
1. We use kylin 1.1 on hbase 1.0.1.1(I modified the hbase version in pom.xml to create the package);
2. The tomcat max threads setting, we didn't modify any setting in tomcat;
3. Kylin's problem.
BTW, we read the code of BadQueryDetector, and it seems a query thread will be killed only when low available memory and 5 minutes lasted. We doubt may be this is not very reasonable.
Best Regards,
George/倪春恩
Software Engineer/软件工程师
Mobile:+86-13501723787| Fax:+8610-56842040
北京明略软件系统有限公司（www.mininglamp.com）
北京市昌平区东小口镇中东路398号中煤建设集团大厦1号楼4层
F4,1#,Zhongmei Construction Group Plaza,398# Zhongdong Road,Changping District,Beijing,102218
----------------------------------------------------------------------------------------------------------------------------
Re: Problem during stress test for kylin quey
Posted by "Adunuthula, Seshu" <sa...@ebay.com>.
You can scale out the Kylin deployment, Am curious to know the details of
your deployment,
- Number of Kylin Server nodes
- Number of Hbase nodes.
Increasing these should increase your concurrency levels…
Regards
Seshu Adunuthula
On 11/12/15, 1:31 AM, "Li Yang" <li...@apache.org> wrote:
>How frequent are the 100 concurrent queries? Also a `jstack` dump of the
>hanging process might help.
>
>We haven't done stress testing for a while. However eBay production is
>quite stable regarding Kylin itself.
>
>The only similar issue we had before is a query super busy at CPU
>resource.
>Such slow query cannot be interrupted, thus cannot be killed by
>BadQueryDetector. Again `jstack` command should reveal such issue.
>
>On Thu, Nov 12, 2015 at 11:12 AM, ShaoFeng Shi <sh...@apache.org>
>wrote:
>
>> Hi Chun En, did you analysis those “bad” SQLs, to see whether they well
>> matched with the cube design? Kylin doesn't guarantee every query can be
>> returned in a very short time, but 80396 seconds need administrator's
>> attention. If the query is good, hbase is good, memory is enough, CPU
>>is at
>> normal level, you need investigate what's the real bottleneck;
>>
>> Previous in eBay deployment we encountered an extreme case (37
>>dimensions,
>> separated into a couple of aggregation groups), when the query cross
>> aggregation groups, the time is very long; Later we identified the
>> bottleneck and made an enhancement in Kylin v1.1; after that we didn't
>> observe such issue. As you already uses Kylin 1.1, I don't think it is
>>this
>> case. You may need do more investigation or provide more detailed
>> information here to analysis.
>>
>>
>>
>> 2015-11-11 17:32 GMT+08:00 nichunen <ni...@mininglamp.com>:
>>
>> > Hi,
>> >
>> > We did a stress test for our kylin server with 100 concurrent queies.
>>It
>> > worked fine at first. But after 1 day, we can't query kylin any more,
>>and
>> > there is log like "query has been running 80396 seconds", many "bad
>> > queries" were hung there. Hbase nodes were still alive, and the cubes
>>and
>> > jobs could still be listed on the pages. To make sure whether it was
>>the
>> > problem of hbase, I restarted hbase, and did a new query, no log from
>> > region server shown hbase received the query, for as we know, a
>> successful
>> > query will create log like "Klin Coprocessor start; Klin Coprocessor
>> > aggregation done". And from the kylin.log, there were still queries
>>hung.
>> >
>> >
>> > Do you know what caused the problem? In our opnion, it may be because:
>> > 1. We use kylin 1.1 on hbase 1.0.1.1(I modified the hbase version in
>> > pom.xml to create the package);
>> > 2. The tomcat max threads setting, we didn't modify any setting in
>> tomcat;
>> > 3. Kylin's problem.
>> >
>> > BTW, we read the code of BadQueryDetector, and it seems a query thread
>> > will be killed only when low available memory and 5 minutes lasted. We
>> > doubt may be this is not very reasonable.
>> >
>> > Best Regards,
>> >
>> >
>> >
>> > George/倪春恩
>> >
>> > Software Engineer/软件工程师
>> >
>> > Mobile:+86-13501723787| Fax:+8610-56842040
>> >
>> > 北京明略软件系统有限公司（www <http://www.semidata.com/>.mininglamp.com）
>> >
>> > 北京市昌平区东小口镇中东路398号中煤建设集团大厦1号楼4层
>> >
>> > F4,1#,Zhongmei Construction Group Plaza,398# Zhongdong Road,Changping
>> > District,Beijing,102218
>> >
>> >
>> >
>>
>>-------------------------------------------------------------------------
>>---------------------------------------------------
>> >
>> > [image: cid:image003.jpg@01D076E2.04C796E0]
>> >
>>
>>
>>
>> --
>> Best regards,
>>
>> Shaofeng Shi
>>
Re: Problem during stress test for kylin quey
Posted by Li Yang <li...@apache.org>.
How frequent are the 100 concurrent queries? Also a `jstack` dump of the
hanging process might help.
We haven't done stress testing for a while. However eBay production is
quite stable regarding Kylin itself.
The only similar issue we had before is a query super busy at CPU resource.
Such slow query cannot be interrupted, thus cannot be killed by
BadQueryDetector. Again `jstack` command should reveal such issue.
On Thu, Nov 12, 2015 at 11:12 AM, ShaoFeng Shi <sh...@apache.org>
wrote:
> Hi Chun En, did you analysis those “bad” SQLs, to see whether they well
> matched with the cube design? Kylin doesn't guarantee every query can be
> returned in a very short time, but 80396 seconds need administrator's
> attention. If the query is good, hbase is good, memory is enough, CPU is at
> normal level, you need investigate what's the real bottleneck;
>
> Previous in eBay deployment we encountered an extreme case (37 dimensions,
> separated into a couple of aggregation groups), when the query cross
> aggregation groups, the time is very long; Later we identified the
> bottleneck and made an enhancement in Kylin v1.1; after that we didn't
> observe such issue. As you already uses Kylin 1.1, I don't think it is this
> case. You may need do more investigation or provide more detailed
> information here to analysis.
>
>
>
> 2015-11-11 17:32 GMT+08:00 nichunen <ni...@mininglamp.com>:
>
> > Hi,
> >
> > We did a stress test for our kylin server with 100 concurrent queies. It
> > worked fine at first. But after 1 day, we can't query kylin any more, and
> > there is log like "query has been running 80396 seconds", many "bad
> > queries" were hung there. Hbase nodes were still alive, and the cubes and
> > jobs could still be listed on the pages. To make sure whether it was the
> > problem of hbase, I restarted hbase, and did a new query, no log from
> > region server shown hbase received the query, for as we know, a
> successful
> > query will create log like "Klin Coprocessor start; Klin Coprocessor
> > aggregation done". And from the kylin.log, there were still queries hung.
> >
> >
> > Do you know what caused the problem? In our opnion, it may be because:
> > 1. We use kylin 1.1 on hbase 1.0.1.1(I modified the hbase version in
> > pom.xml to create the package);
> > 2. The tomcat max threads setting, we didn't modify any setting in
> tomcat;
> > 3. Kylin's problem.
> >
> > BTW, we read the code of BadQueryDetector, and it seems a query thread
> > will be killed only when low available memory and 5 minutes lasted. We
> > doubt may be this is not very reasonable.
> >
> > Best Regards,
> >
> >
> >
> > George/倪春恩
> >
> > Software Engineer/软件工程师
> >
> > Mobile:+86-13501723787| Fax:+8610-56842040
> >
> > 北京明略软件系统有限公司（www <http://www.semidata.com/>.mininglamp.com）
> >
> > 北京市昌平区东小口镇中东路398号中煤建设集团大厦1号楼4层
> >
> > F4,1#,Zhongmei Construction Group Plaza,398# Zhongdong Road,Changping
> > District,Beijing,102218
> >
> >
> >
> ----------------------------------------------------------------------------------------------------------------------------
> >
> > [image: cid:image003.jpg@01D076E2.04C796E0]
> >
>
>
>
> --
> Best regards,
>
> Shaofeng Shi
>
Re: Problem during stress test for kylin quey
Posted by ShaoFeng Shi <sh...@apache.org>.
Hi Chun En, did you analysis those “bad” SQLs, to see whether they well
matched with the cube design? Kylin doesn't guarantee every query can be
returned in a very short time, but 80396 seconds need administrator's
attention. If the query is good, hbase is good, memory is enough, CPU is at
normal level, you need investigate what's the real bottleneck;
Previous in eBay deployment we encountered an extreme case (37 dimensions,
separated into a couple of aggregation groups), when the query cross
aggregation groups, the time is very long; Later we identified the
bottleneck and made an enhancement in Kylin v1.1; after that we didn't
observe such issue. As you already uses Kylin 1.1, I don't think it is this
case. You may need do more investigation or provide more detailed
information here to analysis.
2015-11-11 17:32 GMT+08:00 nichunen <ni...@mininglamp.com>:
> Hi,
>
> We did a stress test for our kylin server with 100 concurrent queies. It
> worked fine at first. But after 1 day, we can't query kylin any more, and
> there is log like "query has been running 80396 seconds", many "bad
> queries" were hung there. Hbase nodes were still alive, and the cubes and
> jobs could still be listed on the pages. To make sure whether it was the
> problem of hbase, I restarted hbase, and did a new query, no log from
> region server shown hbase received the query, for as we know, a successful
> query will create log like "Klin Coprocessor start; Klin Coprocessor
> aggregation done". And from the kylin.log, there were still queries hung.
>
>
> Do you know what caused the problem? In our opnion, it may be because:
> 1. We use kylin 1.1 on hbase 1.0.1.1(I modified the hbase version in
> pom.xml to create the package);
> 2. The tomcat max threads setting, we didn't modify any setting in tomcat;
> 3. Kylin's problem.
>
> BTW, we read the code of BadQueryDetector, and it seems a query thread
> will be killed only when low available memory and 5 minutes lasted. We
> doubt may be this is not very reasonable.
>
> Best Regards,
>
>
>
> George/倪春恩
>
> Software Engineer/软件工程师
>
> Mobile:+86-13501723787| Fax:+8610-56842040
>
> 北京明略软件系统有限公司（www <http://www.semidata.com/>.mininglamp.com）
>
> 北京市昌平区东小口镇中东路398号中煤建设集团大厦1号楼4层
>
> F4,1#,Zhongmei Construction Group Plaza,398# Zhongdong Road,Changping
> District,Beijing,102218
>
>
> ----------------------------------------------------------------------------------------------------------------------------
>
> [image: cid:image003.jpg@01D076E2.04C796E0]
>
--
Best regards,
Shaofeng Shi