You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2021/12/09 08:32:51 UTC
[GitHub] [iceberg] moon-fall opened a new issue #3695: When flinkSink job runs on yarn ,the container is killed because physical memory use beyond physical memory limits
moon-fall opened a new issue #3695:
URL: https://github.com/apache/iceberg/issues/3695
flink version:1.12.1
iceberg version: 0.12.0
When flinkSink job runs on yarn serveral hours , the container is killed because physical memory use beyond physical memory limits and report errors like this:
> 2021-12-06 00:16:36,280 INFO org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - Worker container_e10_1634010428940_125776_01_000003 is terminated. Diagnostics: Container [pid=122806,containerID=container_e10_1634010428940_125776_01_000003] is running beyond physical memory limits. Current usage: 4.0 GB of 4 GB physical memory used; 6.2 GB of 8.4 GB virtual memory used. Killing container.
Dump of the process-tree for container_e10_1634010428940_125776_01_000003 :
|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
|- 122920 122806 122806 122806 (java) 1926809 110163 6567804928 1048262 /usr/local/jdk8/bin/java -Xmx2383706830 -Xms2383706830 -XX:MaxDirectMemorySize=493921243 -XX:MaxMetaspaceSize=268435456
I try to avoid some related flink problems by running flink job like this:
./bin/flink run -m yarn-cluster -p 1 -yjm 2048m -ytm 8000m -yD taskmanager.memory.jvm-overhead.min=2g -yD taskmanager.memory.jvm-overhead.max=2g -yD containerized.taskmanager.env.MALLOC_ARENA_MAX=1 -ys 1 -c icebergTest icebergTest.jar
parameter containerized.taskmanager.env.MALLOC_ARENA_MAX=1 is to solve the Arena "leak" in glibc like this https://publib.boulder.ibm.com/httpserv/cookbook/Operating_Systems-Linux.html?lang=en
parameter taskmanager.memory.jvm-overhead.min=2g and taskmanager.memory.jvm-overhead.max=2g is to increase the overhead memory and this makes taskmanager run longger
and use FsStateBackend as flink State Backends
but it still report errors every serveral hours
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org