The algorithm used is "layer", but why in-mem step while building the cube?

[mvneethu <ne...@tigeranalytics.com>]

Hi,

I have a confusion in the algorithms kylin has for different data. There
are:
1. By Layer
2. Fast Cubing (MR)
3. Fast Cubing with spark

From what I have researched, fast cubing algorithm is the "in-mem"
algorithm.
Now, in my production, I am using "layer" cubing algorithm, but I see a step
called "Build Cube In-Mem". 
1. What does this step do?
2. So by-layer also uses in-mem at this step?

Please clarify my doubt.

--
Sent from: http://apache-kylin.74782.x6.nabble.com/

Re: The algorithm used is "layer", but why in-mem step while building the cube?

[mvneethu <ne...@tigeranalytics.com>]

Thanks for the clarification.

Thanks,
Neethu

--
Sent from: http://apache-kylin.74782.x6.nabble.com/

Re: The algorithm used is "layer", but why in-mem step while building the cube?

[ShaoFeng Shi <sh...@apache.org>]

Hi Neethu,

You're correct that the fast cubing algorithm is also called "in-mem";

The algorithm in Spark is "by-layer", not fast cubing.

On submitting a cubing job, Kylin pre-allocates steps for both "by-layer"
and "in-mem"; But it only picks one to execute, the other one will be
skipped.

It might be a little confusing, but it is easy to implement because by
default the algorithm is "auto", Kylin need to select one after fetching
the data from Hive.


2018-08-27 20:10 GMT+08:00 mvneethu <ne...@tigeranalytics.com>:

> Hi,
>
> I have a confusion in the algorithms kylin has for different data. There
> are:
> 1. By Layer
> 2. Fast Cubing (MR)
> 3. Fast Cubing with spark
>
> From what I have researched, fast cubing algorithm is the "in-mem"
> algorithm.
> Now, in my production, I am using "layer" cubing algorithm, but I see a
> step
> called "Build Cube In-Mem".
> 1. What does this step do?
> 2. So by-layer also uses in-mem at this step?
>
> Please clarify my doubt.
>
> --
> Sent from: http://apache-kylin.74782.x6.nabble.com/
>



-- 
Best regards,

Shaofeng Shi 史少锋