Segments magnagement and auto merging

[Roberto Tardío <ro...@stratebi.com>]

Hi,

I have to generate a big cube, about 400 M rows of historical data (and 
many dimensions in small-mid size cluster). To avoid a very big cube 
building process, I  divided this process into month periods (about 
30-40 M rows per month). When this process finish, an hourly load 
process will begin. Then we will have several historical monthly 
segments and then, new incremental hourly segments. About this scenario, 
arise me the following questions:

  * Do you recommend merge all the historical segments?
      o Sometimes we will need to rebuilt some month from the last six
        months. Due to the cube size, we thougth will be faster to
        rebuilt just a month segment.
  * I' going to define the following auto merge times after we get all
    historical data, for hourly incremental load.
      o 1 day
      o 7 days
      o 28 days
      o I understand well, this means that
          + Every day, all hourly segments will be merged.
          + Every 7 days, all daily segments will be merged.
          + Every 28 days, all 7 days segments will be merged.
      o This config arises my two questions:
          + 28 days segments will be automatically merged any time?
          + our historical big segments will be automatically merged?
  * I thougth that maybe I need to develop an script that merge segments
    as I need (using kylin rest API), instead of using Kylin cube auto
    merge option.

Thanks in advance,

Roberto

-- 

*Roberto Tardío Olmos*

/Senior Big Data & Business Intelligence Consultant/
Avenida de Brasil, 17, Planta 16.28020 Madrid
Fijo: 91.788.34.10