You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kylin.apache.org by Xixin <zx...@163.com> on 2018/07/19 03:24:10 UTC
Configuration Overwrites the container memory for MR task not work
Hi Kylin users/developers
Kylin 2.3.0, hadoop 2.6.5 cdh 5.12
A error occurred when I built a cube at step 2 Redistribute Flat Hive Table.
I checked the MR log and the log shows that reducers containers is running
beyond physical memory limits
below is detail logs:
Diagnostic Messages for this Task:
Container [pid=50920,containerID=container_1518617687620_384770_01_000287]
is running beyond physical memory limits. Current usage: 1.0 GB of 1 GB
physical memory used; 2.6 GB of 2.1 GB virtual memory used. Killing
container.
then I try to overwrites the related config
kylin.engine.mr.config-override.mapreduce.reduce.memory.mb = 4096
kylin.engine.mr.config-override.mapreduce.reduce.java.opts = -Xmx3g
<http://apache-kylin.74782.x6.nabble.com/file/t746/%E5%B1%8F%E5%B9%95%E5%BF%AB%E7%85%A7_2018-07-19_11.png>
then built the cube again, the the same error came again.
I check the yarn config mapreduce.reduce.memory.mb = 2G. this is confused
me. where kylin get the 1GB physical memory config limit? why the overwrites
not work?
--
Sent from: http://apache-kylin.74782.x6.nabble.com/
Re:答复: Configuration Overwrites the container memory for MR task not work
Posted by 赵希欣 <zx...@163.com>.
Hi Silas
I did as you said, and it works, thanks a lot!
as for cube level I config at Web UI for Cube level, below shows the details:
kylin.source.hive.config-override.mapreduce.reduce.memory.mb = 1536
kylin.source.hive.config-override.mapreduce.reduce.java.opts = -Xmx1g
also thanks for Shaohui to make it clear
--
BR
Xixin
At 2018-07-19 13:44:39, "Ge Silas" <go...@live.cn> wrote:
Hi Xixin,
Can you try to configure these memory settings in conf/kylin_hive_conf.xml?
Thanks,
Silas
发件人: Xixin <zx...@163.com>
发送时间: 2018年7月19日 11:24
收件人:dev@kylin.apache.org
主题: Configuration Overwrites the container memory for MR task not work
Hi Kylin users/developers
Kylin 2.3.0, hadoop 2.6.5 cdh 5.12
A error occurred when I built a cube at step 2 Redistribute Flat Hive Table.
I checked the MR log and the log shows that reducers containers is running
beyond physical memory limits
below is detail logs:
Diagnostic Messages for this Task:
Container [pid=50920,containerID=container_1518617687620_384770_01_000287]
is running beyond physical memory limits. Current usage: 1.0 GB of 1 GB
physical memory used; 2.6 GB of 2.1 GB virtual memory used. Killing
container.
then I try to overwrites the related config
kylin.engine.mr.config-override.mapreduce.reduce.memory.mb = 4096
kylin.engine.mr.config-override.mapreduce.reduce.java.opts = -Xmx3g
<http://apache-kylin.74782.x6.nabble.com/file/t746/%E5%B1%8F%E5%B9%95%E5%BF%AB%E7%85%A7_2018-07-19_11.png>
then built the cube again, the the same error came again.
I check the yarn config mapreduce.reduce.memory.mb = 2G. this is confused
me. where kylin get the 1GB physical memory config limit? why the overwrites
not work?
--
Sent from: http://apache-kylin.74782.x6.nabble.com/
答复: Configuration Overwrites the container memory for MR task not work
Posted by Ge Silas <go...@live.cn>.
Hi Xixin,
Can you try to configure these memory settings in conf/kylin_hive_conf.xml?
Thanks,
Silas
________________________________
发件人: Xixin <zx...@163.com>
发送时间: 2018年7月19日 11:24
收件人: dev@kylin.apache.org
主题: Configuration Overwrites the container memory for MR task not work
Hi Kylin users/developers
Kylin 2.3.0, hadoop 2.6.5 cdh 5.12
A error occurred when I built a cube at step 2 Redistribute Flat Hive Table.
I checked the MR log and the log shows that reducers containers is running
beyond physical memory limits
below is detail logs:
Diagnostic Messages for this Task:
Container [pid=50920,containerID=container_1518617687620_384770_01_000287]
is running beyond physical memory limits. Current usage: 1.0 GB of 1 GB
physical memory used; 2.6 GB of 2.1 GB virtual memory used. Killing
container.
then I try to overwrites the related config
kylin.engine.mr.config-override.mapreduce.reduce.memory.mb = 4096
kylin.engine.mr.config-override.mapreduce.reduce.java.opts = -Xmx3g
<http://apache-kylin.74782.x6.nabble.com/file/t746/%E5%B1%8F%E5%B9%95%E5%BF%AB%E7%85%A7_2018-07-19_11.png>
then built the cube again, the the same error came again.
I check the yarn config mapreduce.reduce.memory.mb = 2G. this is confused
me. where kylin get the 1GB physical memory config limit? why the overwrites
not work?
--
Sent from: http://apache-kylin.74782.x6.nabble.com/