You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@bigtop.apache.org by Niels Basjes <Ni...@basjes.nl> on 2020/11/03 12:30:35 UTC

How to change the heap sizes in docker?

Hi,

I have the need to do some integration testing for a Hive UDF I have
written.
So I start a docker based  cluster with a centos 7 config that contains
   components: [hdfs, yarn, mapreduce, hive]

and create it with

   ./docker-hadoop.sh -c 3

The problem I have is that this UDF needs a few hundred MiB of ram and
Java heapsize of the processes is fixed at 1000m (I see -Xmx1000m for most
processes)

I have already increased the docker memory_limit but I have not been able
to figure out what the proper way is to increase this 1000m value to
something like 2000m

I searched the code and the documentation and could not find how to change
this.

I have tried several things (setting HADOOP_HEAPSIZE , YARN_HEAPSIZE, ...)
but none of those attempts had any impact. Also adding config lines like
this did not have any effect:

configurations::yarn-env::yarn_heapsize: 2222
yarn-env::yarn_heapsize: 2223
yarn_heapsize: 2224

So right now I hacked it by doing

./docker-hadoop.sh -e 1 sed -i 's@Xmx1000m@Xmx2000m@g'
/usr/lib/hadoop/libexec/hadoop-config.sh

and then forcing the various services to restart... which feels "so totally
wrong"

My question is simply what is the clean/correct way to specify this java
heapsize for these processes?

-- 
Best regards / Met vriendelijke groeten,

Niels Basjes