You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kylin.apache.org by 杨海乐 <ya...@letv.com> on 2016/01/20 10:06:23 UTC
From the Build Base Cuboid Data step to Build N-Dimension steps,
Too much time is taken.
Hello all,
In Build Base Cuboid Data step and Build N-Dimension steps,I find that
every step takes too time to handle records like the log. Should I configure
my hadoop's setting.
Thanks every much.
2016-01-20 16:22:06,092 INFO [main]
org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
hadoop-metrics2.properties
2016-01-20 16:22:06,143 INFO [main]
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period
at 10 second(s).
2016-01-20 16:22:06,143 INFO [main]
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system
started
2016-01-20 16:22:06,150 INFO [main] org.apache.hadoop.mapred.YarnChild:
Executing with tokens:
2016-01-20 16:22:06,150 INFO [main] org.apache.hadoop.mapred.YarnChild:
Kind: mapreduce.job, Service: job_1452787370536_0086, Ident:
(org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@dc42543)
2016-01-20 16:22:06,223 INFO [main] org.apache.hadoop.mapred.YarnChild:
Sleeping for 0ms before retrying again. Got null now.
2016-01-20 16:22:06,412 INFO [main] org.apache.hadoop.mapred.YarnChild:
mapreduce.cluster.local.dir for child:
nm-local-dir/usercache/kylin/appcache/application_1452787370536_0086
2016-01-20 16:22:06,716 INFO [main]
org.apache.hadoop.conf.Configuration.deprecation: session.id is deprecated.
Instead, use dfs.metrics.session-id
2016-01-20 16:22:07,045 INFO [main] org.apache.hadoop.mapred.Task: Using
ResourceCalculatorProcessTree : [ ]
2016-01-20 16:22:07,237 INFO [main] org.apache.hadoop.mapred.MapTask:
Processing split:
hdfs://**/kylin_metadata/kylin-8acccbc4-84b7-4774-b668-f64228c09672/kylin_intermediate_avro_global_device_20151219000000_20151221000000_8acccbc4_84b7_4774_b668_f64228c09672/000000_0:0+16387256
2016-01-20 16:22:07,283 INFO [main] org.apache.hadoop.mapred.MapTask:
(EQUATOR) 0 kvi 26214396(104857584)
2016-01-20 16:22:07,284 INFO [main] org.apache.hadoop.mapred.MapTask:
mapreduce.task.io.sort.mb: 100
2016-01-20 16:22:07,284 INFO [main] org.apache.hadoop.mapred.MapTask: soft
limit at 83886080
2016-01-20 16:22:07,284 INFO [main] org.apache.hadoop.mapred.MapTask:
bufstart = 0; bufvoid = 104857600
2016-01-20 16:22:07,284 INFO [main] org.apache.hadoop.mapred.MapTask:
kvstart = 26214396; length = 6553600
2016-01-20 16:22:07,289 INFO [main]
com.hadoop.compression.lzo.GPLNativeCodeLoader: Loaded native gpl library
from the embedded binaries
2016-01-20 16:22:07,291 INFO [main] com.hadoop.compression.lzo.LzoCodec:
Successfully loaded & initialized native-lzo library [hadoop-lzo rev
826e7d8d3e839964dd9ed2d5f83296254b2c71d3]
2016-01-20 16:22:07,295 INFO [main] org.apache.hadoop.mapred.MapTask: Map
output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2016-01-20 16:22:07,316 INFO [main]
org.apache.hadoop.conf.Configuration.deprecation: hadoop.native.lib is
deprecated. Instead, use io.native.lib.available
2016-01-20 16:22:07,317 INFO [main] org.apache.hadoop.io.compress.CodecPool:
Got brand-new decompressor [.lzo_deflate]
2016-01-20 16:22:07,322 INFO [main] org.apache.hadoop.io.compress.CodecPool:
Got brand-new decompressor [.lzo_deflate]
2016-01-20 16:22:07,322 INFO [main] org.apache.hadoop.io.compress.CodecPool:
Got brand-new decompressor [.lzo_deflate]
2016-01-20 16:22:07,322 INFO [main] org.apache.hadoop.io.compress.CodecPool:
Got brand-new decompressor [.lzo_deflate]
2016-01-20 16:22:07,328 INFO [main]
org.apache.kylin.job.hadoop.AbstractHadoopJob: The absolute path for meta
dir is
/**/tmp/nm-local-dir/usercache/kylin/appcache/application_1452787370536_0086/container_1452787370536_0086_01_000003/meta
2016-01-20 16:22:07,330 INFO [main] org.apache.kylin.common.KylinConfig: Use
KYLIN_CONF=/**/tmp/nm-local-dir/usercache/kylin/appcache/application_1452787370536_0086/container_1452787370536_0086_01_000003/meta
2016-01-20 16:22:07,344 INFO [main] org.apache.kylin.cube.CubeManager:
Initializing CubeManager with config
/**/tmp/nm-local-dir/usercache/kylin/appcache/application_1452787370536_0086/container_1452787370536_0086_01_000003/meta
2016-01-20 16:22:07,346 INFO [main]
org.apache.kylin.common.persistence.ResourceStore: Using metadata url
/**/tmp/nm-local-dir/usercache/kylin/appcache/application_1452787370536_0086/container_1452787370536_0086_01_000003/meta
for resource store
2016-01-20 16:22:07,558 INFO [main] org.apache.kylin.cube.CubeDescManager:
Initializing CubeDescManager with config
/**/tmp/nm-local-dir/usercache/kylin/appcache/application_1452787370536_0086/container_1452787370536_0086_01_000003/meta
2016-01-20 16:22:07,559 INFO [main] org.apache.kylin.cube.CubeDescManager:
Reloading Cube Metadata from folder
/**/tmp/nm-local-dir/usercache/kylin/appcache/application_1452787370536_0086/container_1452787370536_0086_01_000003/meta/cube_desc
2016-01-20 16:23:14,463 INFO [main]
org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled 100000 records!
2016-01-20 16:24:20,530 INFO [main]
org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled 200000 records!
2016-01-20 16:25:26,496 INFO [main]
org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled 300000 records!
2016-01-20 16:26:32,492 INFO [main]
org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled 400000 records!
2016-01-20 16:27:38,671 INFO [main]
org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled 500000 records!
2016-01-20 16:28:45,026 INFO [main]
org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled 600000 records!
2016-01-20 16:29:51,697 INFO [main]
org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled 700000 records!
2016-01-20 16:30:58,548 INFO [main]
org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled 800000 records!
2016-01-20 16:32:05,959 INFO [main]
org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled 900000 records!
2016-01-20 16:33:13,550 INFO [main]
org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled 1000000 records!
2016-01-20 16:34:20,893 INFO [main]
org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled 1100000 records!
2016-01-20 16:35:28,145 INFO [main]
org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled 1200000 records!
2016-01-20 16:36:11,918 INFO [main] org.apache.hadoop.mapred.MapTask:
Spilling map output
2016-01-20 16:36:11,918 INFO [main] org.apache.hadoop.mapred.MapTask:
bufstart = 0; bufend = 63651120; bufvoid = 104857600
2016-01-20 16:36:11,918 INFO [main] org.apache.hadoop.mapred.MapTask:
kvstart = 26214396(104857584); kvend = 21155656(84622624); length =
5058741/6553600
2016-01-20 16:36:11,918 INFO [main] org.apache.hadoop.mapred.MapTask:
(EQUATOR) 68735104 kvi 17183772(68735088)
2016-01-20 16:36:12,669 INFO [SpillThread]
org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor
[.lzo_deflate]
2016-01-20 16:36:12,706 INFO [SpillThread]
org.apache.kylin.job.hadoop.AbstractHadoopJob: The absolute path for meta
dir is
/**/tmp/nm-local-dir/usercache/kylin/appcache/application_1452787370536_0086/container_1452787370536_0086_01_000003/meta
2016-01-20 16:36:37,977 INFO [main]
org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled 1300000 records!
2016-01-20 16:37:37,524 INFO [SpillThread]
org.apache.kylin.job.hadoop.cube.CuboidReducer: Handled 100000 records!
2016-01-20 16:37:48,445 INFO [main]
org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled 1400000 records!
2016-01-20 16:39:00,123 INFO [SpillThread]
org.apache.kylin.job.hadoop.cube.CuboidReducer: Handled 200000 records!
2016-01-20 16:39:00,589 INFO [main]
org.apache.kylin.job.hadoop.cube.BaseCuboidMapper: Handled 1500000 records!
2016-01-20 16:39:46,678 INFO [main] org.apache.hadoop.mapred.MapTask:
Starting flush of map output
2016-01-20 16:40:21,932 INFO [SpillThread]
org.apache.kylin.job.hadoop.cube.CuboidReducer: Handled 300000 records!
2016-01-20 16:41:42,143 INFO [SpillThread]
org.apache.kylin.job.hadoop.cube.CuboidReducer: Handled 400000 records!
2016-01-20 16:43:02,409 INFO [SpillThread]
org.apache.kylin.job.hadoop.cube.CuboidReducer: Handled 500000 records!
2016-01-20 16:44:21,803 INFO [SpillThread]
org.apache.kylin.job.hadoop.cube.CuboidReducer: Handled 600000 records!
2016-01-20 16:45:41,249 INFO [SpillThread]
org.apache.kylin.job.hadoop.cube.CuboidReducer: Handled 700000 records!
2016-01-20 16:47:00,661 INFO [SpillThread]
org.apache.kylin.job.hadoop.cube.CuboidReducer: Handled 800000 records!
2016-01-20 16:48:20,193 INFO [SpillThread]
org.apache.kylin.job.hadoop.cube.CuboidReducer: Handled 900000 records!
2016-01-20 16:49:39,569 INFO [SpillThread]
org.apache.kylin.job.hadoop.cube.CuboidReducer: Handled 1000000 records!
2016-01-20 16:50:59,397 INFO [SpillThread]
org.apache.kylin.job.hadoop.cube.CuboidReducer: Handled 1100000 records!
2016-01-20 16:52:18,927 INFO [SpillThread]
org.apache.kylin.job.hadoop.cube.CuboidReducer: Handled 1200000 records!
2016-01-20 16:53:10,339 INFO [SpillThread] org.apache.hadoop.mapred.MapTask:
Finished spill 0
2016-01-20 16:53:10,339 INFO [main] org.apache.hadoop.mapred.MapTask:
(RESET) equator 68735104 kv 17183772(68735088) kvi 15983048(63932192)
2016-01-20 16:53:10,339 INFO [main] org.apache.hadoop.mapred.MapTask:
Spilling map output
2016-01-20 16:53:10,340 INFO [main] org.apache.hadoop.mapred.MapTask:
bufstart = 68735104; bufend = 83814293; bufvoid = 104857600
2016-01-20 16:53:10,340 INFO [main] org.apache.hadoop.mapred.MapTask:
kvstart = 17183772(68735088); kvend = 15983052(63932208); length =
1200721/6553600
2016-01-20 16:53:10,530 INFO [main]
org.apache.kylin.job.hadoop.AbstractHadoopJob: The absolute path for meta
dir is
/**/tmp/nm-local-dir/usercache/kylin/appcache/application_1452787370536_0086/container_1452787370536_0086_01_000003/meta
2016-01-20 16:54:30,303 INFO [main]
org.apache.kylin.job.hadoop.cube.CuboidReducer: Handled 100000 records!
2016-01-20 16:55:49,729 INFO [main]
org.apache.kylin.job.hadoop.cube.CuboidReducer: Handled 200000 records!
2016-01-20 16:57:09,126 INFO [main]
org.apache.kylin.job.hadoop.cube.CuboidReducer: Handled 300000 records!
2016-01-20 16:57:09,272 INFO [main] org.apache.hadoop.mapred.MapTask:
Finished spill 1
2016-01-20 16:57:09,277 INFO [main] org.apache.hadoop.mapred.Merger: Merging
2 sorted segments
2016-01-20 16:57:09,282 INFO [main] org.apache.hadoop.mapred.Merger: Down to
the last merge-pass, with 2 segments left of total size: 26184336 bytes
2016-01-20 16:57:10,555 INFO [main] org.apache.hadoop.mapred.Task:
Task:attempt_1452787370536_0086_m_000000_0 is done. And is in the process of
committing
2016-01-20 16:57:10,600 INFO [main] org.apache.hadoop.mapred.Task: Task
'attempt_1452787370536_0086_m_000000_0' done.
--
View this message in context: http://apache-kylin.74782.x6.nabble.com/From-the-Build-Base-Cuboid-Data-step-to-Build-N-Dimension-steps-Too-much-time-is-taken-tp3351.html
Sent from the Apache Kylin mailing list archive at Nabble.com.
Re: From the Build Base Cuboid Data step to Build N-Dimension
steps, Too much time is taken.
Posted by 杨海乐 <ya...@letv.com>.
At first,I also think so.Howerver,after resetting the precision to 9.75%,I
find the processing time become so fast.
--
View this message in context: http://apache-kylin.74782.x6.nabble.com/From-the-Build-Base-Cuboid-Data-step-to-Build-N-Dimension-steps-Too-much-time-is-taken-tp3351p3655.html
Sent from the Apache Kylin mailing list archive at Nabble.com.
Re: From the Build Base Cuboid Data step to Build N-Dimension steps,
Too much time is taken.
Posted by Li Yang <li...@apache.org>.
Reduce "kylin.job.mapreduce.default.reduce.input.mb" will give you more
reducers and can speed up the MR if the bottleneck is in reducer and there
are extra reducer slots in your cluster.
However there are many other reasons why a MR is slow. E.g. data skew,
where a certain mapper or reducer gets a extremely big chunk of data and
slow down the whole job.
Based on experience, it's not common that a count distinct being the main
reason of a slow job.
On Thu, Jan 21, 2016 at 5:36 PM, 杨海乐 <ya...@letv.com> wrote:
> I find that The reason is the precision of count distinct measure. The
> precision is 1.2% . So the steps is too slow even though the data is
> little(million).Can I sloue the problem by reducing the value of
> kylin.job.mapreduce.default.reduce.input.mb？
>
> --
> View this message in context:
> http://apache-kylin.74782.x6.nabble.com/From-the-Build-Base-Cuboid-Data-step-to-Build-N-Dimension-steps-Too-much-time-is-taken-tp3351p3368.html
> Sent from the Apache Kylin mailing list archive at Nabble.com.
>
Re: From the Build Base Cuboid Data step to Build N-Dimension
steps, Too much time is taken.
Posted by 杨海乐 <ya...@letv.com>.
I find that The reason is the precision of count distinct measure. The
precision is 1.2% . So the steps is too slow even though the data is
little(million).Can I sloue the problem by reducing the value of
kylin.job.mapreduce.default.reduce.input.mb？
--
View this message in context: http://apache-kylin.74782.x6.nabble.com/From-the-Build-Base-Cuboid-Data-step-to-Build-N-Dimension-steps-Too-much-time-is-taken-tp3351p3368.html
Sent from the Apache Kylin mailing list archive at Nabble.com.