You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kylin.apache.org by "yuan (Jira)" <ji...@apache.org> on 2023/01/12 12:52:00 UTC
[jira] [Created] (KYLIN-5401) Optimize code logic for pushdown queries
yuan created KYLIN-5401:
---------------------------
Summary: Optimize code logic for pushdown queries
Key: KYLIN-5401
URL: https://issues.apache.org/jira/browse/KYLIN-5401
Project: Kylin
Issue Type: Improvement
Components: Query Engine, Spark Engine
Affects Versions: v4.0.3, v4.0.2, v4.0.1
Reporter: yuan
It is found that the push-down query in kylin4.0.x is very slow for the simple query eg: {{{}select * from table limit 10{}}}. It should respond in seconds, but it often takes a few minutes, and the larger the query data set, the more time-consuming it takes Long, which is very abnormal.BI tools often execute some simple queries to display detailed data. Abnormal query duration often causes BI tools to time out and return error messages, which is very unfriendly to user experience.
Through investigation, it is found that there is a *shuffle* process in the query plan of this very simple detailed query, which is outrageous.
The main logic of Kylin executing push-down query is concentrated in {{{}org.apache.kylin.query.pushdown.SparkSqlClient{}}}.Unnecessary Spark DataFrame type transform in {{org.apache.kylin.query.pushdown.SparkSqlClient#DFToList}} is the main cause of this problem.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)