You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kylin.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2023/04/23 06:01:00 UTC
[jira] [Commented] (KYLIN-5454) Downloading the async query result may cause OOM error
[ https://issues.apache.org/jira/browse/KYLIN-5454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17715382#comment-17715382 ]
ASF subversion and git services commented on KYLIN-5454:
--------------------------------------------------------
Commit 141eb4a7c306712ee5525789fdcb8cd8dbc68cc1 in kylin's branch refs/heads/kylin5 from Ruixuan Zhang
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=141eb4a7c3 ]
KYLIN-5454 add UT for Async Query Result
> Downloading the async query result may cause OOM error
> ------------------------------------------------------
>
> Key: KYLIN-5454
> URL: https://issues.apache.org/jira/browse/KYLIN-5454
> Project: Kylin
> Issue Type: Bug
> Affects Versions: 5.0-alpha
> Reporter: Ruixuan Zhang
> Assignee: Ruixuan Zhang
> Priority: Major
> Fix For: 5.0-alpha
>
> Attachments: Dev Design - Async Query may cause OOM.pdf
>
>
> When using {{http://host:port/kylin/api/async_query/\{query_id}/result_download?}} to get asynchronous query result, out-of-memory error might occur. The reason is the formatting process while downloading the result needs read & write, and there is batch processing rather than streaming processing. Therefore, a refactor is needed for async query. The main idea of revision is to move the formatting from downloading result from HDFS to processing & saving query result on HDFS, and use streaming read & write instead.
> The details of design are in the attached file.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)