You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@kylin.apache.org by ShaoFeng Shi <sh...@apache.org> on 2019/09/02 02:52:10 UTC
Re: query concurrency
Hello,
Usually, if the cube has been well designed and tunned, one Kylin server
can support 50 to 150 QPS. From the Cisco team's experience, with enabling
some advanced features like query cache and prepared statement, one Kylin
server can support up to 500 QPS:
https://kylin.apache.org/blog/2019/01/17/cisco-throughput-5x/
If your QPS is not ideal, please try to tune a single query's performance
and optimize the cube design.
Best regards,
Shaofeng Shi 史少锋
Apache Kylin PMC
Email: shaofengshi@apache.org
Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
Join Kylin user mail group: user-subscribe@kylin.apache.org
Join Kylin dev mail group: dev-subscribe@kylin.apache.org
shicheng31604@gmail.com <sh...@gmail.com> 于2019年8月26日周一 下午4:32写道：
>
> Hi:
> According to kylin's introduction, an ordinary server can accept tens
> to hundreds of QPS. However, I found in the actual production that the QPS
> of a query node can only reach single digits. It is reasonable to say that
> the concurrency of a SpringBoot project is relatively high, and the
> resolution speed of Calcite does not stop there.HBase's reading speed is no
> problem. But the actual effect is so abnormal, what could be the problem?
>
> ------------------------------
> shicheng31604@gmail.com
>
>
Re: Re: query concurrency
Posted by "shicheng31604@gmail.com" <sh...@gmail.com>.
Our deep storage (hbase ) size reaches 40T , already divided by the factor of replica. And HBase has about 40 thousand regions distributed on 10 hbase nodes.
SQL statement is not so complex, maybe ,here is an example:
SELECT SUM(a.biz_amount) val, a.delivery_type AS type, SUM(a.trade_count) AS counts FROM bd_rept.v_fact_mind_trade_biz_view a WHERE a.brand_identy = 11349 AND a.shop_identy IN (12346) AND a.trade_biz_date >= cast('2019-08-01' AS date) AND a.trade_biz_date <= cast('2019-08-31' AS date) AND a.server_create_time_hour >= cast( '2019-08-01 10:00:00' AS TIMESTAMP ) AND a.server_create_time_hour <= cast( '2019-08-31 23:59:59' AS TIMESTAMP ) AND a.trade_status IN (4, 11) AND a.trade_pay_status IN (1, 3) AND a.trade_type IN (1, 3, 4) AND a.business_type IN (111, 221, 4324331, 51, 6, 71, 123423425, 16, 211111) AND a.delivery_type IN (234,21,345,135,310) GROUP BY a.delivery_type
I can't get cube design details now .But according current known conditions, can we figure out where the problem is?
By the way, we found if one query node got to many requests then it would throw an exception as follows:
shicheng31604@gmail.com
From: ShaoFeng Shi
Date: 2019-09-02 13:34
To: user
Subject: Re: Re: query concurrency
Yes, it depends on the specific scenario. If you can provide the testing scenario, SQL queries, and detailed cube design, then we can discuss it.
Best regards,
Shaofeng Shi 史少锋
Apache Kylin PMC
Email: shaofengshi@apache.org
Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
Join Kylin user mail group: user-subscribe@kylin.apache.org
Join Kylin dev mail group: dev-subscribe@kylin.apache.org
shicheng31604@gmail.com <sh...@gmail.com> 于2019年9月2日周一 上午11:10写道：
But if each SQL statement is different, how much can QPS reach?
shicheng31604@gmail.com
From: ShaoFeng Shi
Date: 2019-09-02 10:52
To: user
Subject: Re: query concurrency
Hello,
Usually, if the cube has been well designed and tunned, one Kylin server can support 50 to 150 QPS. From the Cisco team's experience, with enabling some advanced features like query cache and prepared statement, one Kylin server can support up to 500 QPS:
https://kylin.apache.org/blog/2019/01/17/cisco-throughput-5x/
If your QPS is not ideal, please try to tune a single query's performance and optimize the cube design.
Best regards,
Shaofeng Shi 史少锋
Apache Kylin PMC
Email: shaofengshi@apache.org
Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
Join Kylin user mail group: user-subscribe@kylin.apache.org
Join Kylin dev mail group: dev-subscribe@kylin.apache.org
shicheng31604@gmail.com <sh...@gmail.com> 于2019年8月26日周一 下午4:32写道：
Hi:
According to kylin's introduction, an ordinary server can accept tens to hundreds of QPS. However, I found in the actual production that the QPS of a query node can only reach single digits. It is reasonable to say that the concurrency of a SpringBoot project is relatively high, and the resolution speed of Calcite does not stop there.HBase's reading speed is no problem. But the actual effect is so abnormal, what could be the problem?
shicheng31604@gmail.com
Re: Re: query concurrency
Posted by ShaoFeng Shi <sh...@apache.org>.
Yes, it depends on the specific scenario. If you can provide the testing
scenario, SQL queries, and detailed cube design, then we can discuss it.
Best regards,
Shaofeng Shi 史少锋
Apache Kylin PMC
Email: shaofengshi@apache.org
Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
Join Kylin user mail group: user-subscribe@kylin.apache.org
Join Kylin dev mail group: dev-subscribe@kylin.apache.org
shicheng31604@gmail.com <sh...@gmail.com> 于2019年9月2日周一 上午11:10写道：
> But if each SQL statement is different, how much can QPS reach?
> ------------------------------
> shicheng31604@gmail.com
>
>
> *From:* ShaoFeng Shi <sh...@apache.org>
> *Date:* 2019-09-02 10:52
> *To:* user <us...@kylin.apache.org>
> *Subject:* Re: query concurrency
> Hello,
>
> Usually, if the cube has been well designed and tunned, one Kylin server
> can support 50 to 150 QPS. From the Cisco team's experience, with enabling
> some advanced features like query cache and prepared statement, one Kylin
> server can support up to 500 QPS:
>
> https://kylin.apache.org/blog/2019/01/17/cisco-throughput-5x/
>
> If your QPS is not ideal, please try to tune a single query's performance
> and optimize the cube design.
>
> Best regards,
>
> Shaofeng Shi 史少锋
> Apache Kylin PMC
> Email: shaofengshi@apache.org
>
> Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
> Join Kylin user mail group: user-subscribe@kylin.apache.org
> Join Kylin dev mail group: dev-subscribe@kylin.apache.org
>
>
>
>
> shicheng31604@gmail.com <sh...@gmail.com> 于2019年8月26日周一 下午4:32写道：
>
>>
>> Hi:
>> According to kylin's introduction, an ordinary server can accept
>> tens to hundreds of QPS. However, I found in the actual production that the
>> QPS of a query node can only reach single digits. It is reasonable to say
>> that the concurrency of a SpringBoot project is relatively high, and the
>> resolution speed of Calcite does not stop there.HBase's reading speed is no
>> problem. But the actual effect is so abnormal, what could be the problem?
>>
>> ------------------------------
>> shicheng31604@gmail.com
>>
>>
Re: Re: query concurrency
Posted by "shicheng31604@gmail.com" <sh...@gmail.com>.
But if each SQL statement is different, how much can QPS reach?
shicheng31604@gmail.com
From: ShaoFeng Shi
Date: 2019-09-02 10:52
To: user
Subject: Re: query concurrency
Hello,
Usually, if the cube has been well designed and tunned, one Kylin server can support 50 to 150 QPS. From the Cisco team's experience, with enabling some advanced features like query cache and prepared statement, one Kylin server can support up to 500 QPS:
https://kylin.apache.org/blog/2019/01/17/cisco-throughput-5x/
If your QPS is not ideal, please try to tune a single query's performance and optimize the cube design.
Best regards,
Shaofeng Shi 史少锋
Apache Kylin PMC
Email: shaofengshi@apache.org
Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
Join Kylin user mail group: user-subscribe@kylin.apache.org
Join Kylin dev mail group: dev-subscribe@kylin.apache.org
shicheng31604@gmail.com <sh...@gmail.com> 于2019年8月26日周一 下午4:32写道：
Hi:
According to kylin's introduction, an ordinary server can accept tens to hundreds of QPS. However, I found in the actual production that the QPS of a query node can only reach single digits. It is reasonable to say that the concurrency of a SpringBoot project is relatively high, and the resolution speed of Calcite does not stop there.HBase's reading speed is no problem. But the actual effect is so abnormal, what could be the problem?
shicheng31604@gmail.com