You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@kylin.apache.org by maoxiaomao <la...@163.com> on 2019/08/16 07:58:23 UTC
What determine the file num When MERGE Step. Convert Cuboid Data to
HFile
Hi,
I excuted a MERGE job, in Step. Convert Cuboid Data to HFile ,there is about 30,000,000 files in path /kylin_metadata/kylin-7645ff8a-8fa4-36f7-c052-a0b18b1526b7/{cubeName}/hfile/_temporary/1/_temporary/attempt_1558444096078_504979_r_0000**_1/F1 , file name looks like in increasing sort (pic 1.).
I wonder this file num determined by what factors？Hoping for reply :)
pic 1.
Re:What determine the file num When MERGE Step. Convert Cuboid Data
to HFile
Posted by maoxiaomao <la...@163.com>.
Inaddion I'm useing:
kylin2.6.1
hbase 2.0.0-cdh6.0.0
add for this step there is 1500Mappers，100reduers.
50 region for the hbase Table
There is some log when reducer writing file : (It wrote about 900M for the first file，but only less then 100Bfor anoter files)
2019-08-16 14:42:46,805 INFO [main] org.apache.kylin.engine.mr.KylinReducer: Accepting Reducer Key with ordinal: 30000001
2019-08-16 14:42:46,805 INFO [main] org.apache.kylin.engine.mr.KylinReducer: Do reduce, available memory: 8604m
2019-08-16 14:42:47,529 INFO [main] org.apache.kylin.engine.mr.KylinReducer: Accepting Reducer Key with ordinal: 30100001
2019-08-16 14:42:47,529 INFO [main] org.apache.kylin.engine.mr.KylinReducer: Do reduce, available memory: 8465m
2019-08-16 14:42:48,821 INFO [main] org.apache.kylin.engine.mr.KylinReducer: Accepting Reducer Key with ordinal: 30200001
2019-08-16 14:42:48,821 INFO [main] org.apache.kylin.engine.mr.KylinReducer: Do reduce, available memory: 8256m
2019-08-16 14:42:49,875 INFO [main] org.apache.kylin.engine.mr.KylinReducer: Accepting Reducer Key with ordinal: 30300001
2019-08-16 14:42:49,875 INFO [main] org.apache.kylin.engine.mr.KylinReducer: Do reduce, available memory: 8118m
2019-08-16 14:42:50,403 WARN [DataStreamer for file /data/kylin/kylin_metadata/kylin-6bc4c609-25d6-8628-0123-7790642c8644/EXPT_BASE_MINI_V1_HOURLY_APP_BACKEND_ONE_GROUP_SIX_MAX/hfile/_temporary/1/_temporary/attempt_1558444096078_519332_r_000016_0/F2/72469283b65f478c835bbf774180ec0a] org.apache.hadoop.hdfs.DataStreamer: These favored nodes were specified but not chosen: [c3-c03-136-30-16.yidian.com:16020] Specified favored nodes: [c3-c03-136-30-16.yidian.com:16020]
2019-08-16 14:42:50,699 INFO [main] org.apache.kylin.engine.mr.KylinReducer: Accepting Reducer Key with ordinal: 30400001
2019-08-16 14:42:50,699 INFO [main] org.apache.kylin.engine.mr.KylinReducer: Do reduce, available memory: 7930m
2019-08-16 14:42:51,798 INFO [main] org.apache.kylin.engine.mr.KylinReducer: Accepting Reducer Key with ordinal: 30500001
2019-08-16 14:42:51,798 INFO [main] org.apache.kylin.engine.mr.KylinReducer: Do reduce, available memory: 7699m
2019-08-16 14:42:51,988 INFO [main] org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2: Writer=hdfs://yellow/data/kylin/kylin_metadata/kylin-6bc4c609-25d6-8628-0123-7790642c8644/EXPT_BASE_MINI_V1_HOURLY_APP_BACKEND_ONE_GROUP_SIX_MAX/hfile/_temporary/1/_temporary/attempt_1558444096078_519332_r_000016_0/F1/06f3b0e9f0644ee782b7cf4469f44a70, wrote=893827310
7790642c8644/EXPT_BASE_MINI_V1_HOURLY_APP_BACKEND_ONE_GROUP_SIX_MAX/hfile/_temporary/1/_temporary/attempt_1558444096078_519332_r_000016_0/F1/1454ea148f1547499209a266ad25387f, wrote=61
7790642c8644/EXPT_BASE_MINI_V1_HOURLY_APP_BACKEND_ONE_GROUP_SIX_MAX/hfile/_temporary/1/_temporary/attempt_1558444096078_519332_r_000016_0/F1/9d35446634154b4ca4be56f361b57c8b, wrote=55
At 2019-08-16 15:58:23, "maoxiaomao" <la...@163.com> wrote:
Hi,
I excuted a MERGE job, in Step. Convert Cuboid Data to HFile ,there is about 30,000,000 files in path /kylin_metadata/kylin-7645ff8a-8fa4-36f7-c052-a0b18b1526b7/{cubeName}/hfile/_temporary/1/_temporary/attempt_1558444096078_504979_r_0000**_1/F1 , file name looks like in increasing sort (pic 1.).
I wonder this file num determined by what factors？Hoping for reply :)
pic 1.