You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@kylin.apache.org by "yuan (Jira)" <ji...@apache.org> on 2023/01/12 12:52:00 UTC

[jira] [Created] (KYLIN-5401) Optimize code logic for pushdown queries

yuan created KYLIN-5401:
---------------------------

             Summary: Optimize code logic for pushdown queries
                 Key: KYLIN-5401
                 URL: https://issues.apache.org/jira/browse/KYLIN-5401
             Project: Kylin
          Issue Type: Improvement
          Components: Query Engine, Spark Engine
    Affects Versions: v4.0.3, v4.0.2, v4.0.1
            Reporter: yuan


It is found that the push-down query in kylin4.0.x is very slow for the simple query eg: {{{}select * from table limit 10{}}}. It should respond in seconds, but it often takes a few minutes, and the larger the query data set, the more time-consuming it takes Long, which is very abnormal.BI tools often execute some simple queries to display detailed data. Abnormal query duration often causes BI tools to time out and return error messages, which is very unfriendly to user experience.
Through investigation, it is found that there is a *shuffle* process in the query plan of this very simple detailed query, which is outrageous.
The main logic of Kylin executing push-down query is concentrated in {{{}org.apache.kylin.query.pushdown.SparkSqlClient{}}}.Unnecessary Spark DataFrame type transform in {{org.apache.kylin.query.pushdown.SparkSqlClient#DFToList}} is the main cause of this problem.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)