[jira] [Comment Edited] (KYLIN-1048) CPU and memory killer in Cuboid.findById()

["Shaofeng SHI (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KYLIN-1048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14934465#comment-14934465 ] 

Shaofeng SHI edited comment on KYLIN-1048 at 9/29/15 6:39 AM:
--------------------------------------------------------------

The problem is in Cuboid.translateToValidCuboid(), which is to find an existing parent cuboid from the given dimension/filter list; The old algorithm is Breadth-First-Search, algorithm complexity is O(2^N); Rewrite this method to directly conduct the parent cuboid, algorithm complexity reduced to O(1)；After deploy this patch, the CPU/memory is at normal level with the same query, and the result can be returned very shortly.


was (Author: shaofengshi):
The problem is in Cuboid.translateToValidCuboid(), which is to find an existing parent cuboid from the given dimension/filter list; The old algorithm is Breadth-First-Search, algorithm complexity is O(N^2); Rewrite this method to directly conduct the parent cuboid, algorithm complexity reduced to O(1)；After deploy this patch, the CPU/memory is at normal level with the same query, and the result can be returned very shortly.

> CPU and memory killer in Cuboid.findById()
> ------------------------------------------
>
>                 Key: KYLIN-1048
>                 URL: https://issues.apache.org/jira/browse/KYLIN-1048
>             Project: Kylin
>          Issue Type: Improvement
>          Components: Metadata
>    Affects Versions: v1.0, v0.7.2, v0.7.1
>            Reporter: Shaofeng SHI
>            Assignee: Shaofeng SHI
>             Fix For: v2.0, v1.1
>
>
> In an cube which has 37 dimensions with a couple of aggregation groups (and each group has hierarchy), when a SQL which has columns cross aggregation groups, it couldn't be returned in time, the CPU will very high and memory usage increases also, finally cause Kylin server crashed with OutOfMemory error.
> After doing some analysis and profiling, identified that when the cube is a sparse partial cube, the performance in Cuboid.findById() is bad,  which will take much CPU and memory.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)