You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Steve Howard <st...@gmail.com> on 2015/03/13 01:24:09 UTC
sqoop import to hive being killed by resource manager
Hi All,
We have not been able to get what is in the subject line to run. This is
on hive 0.14. While pulling a billion row table from Oracle using 12
splits on the primary key, each job continually runs out of memory such as
below...
15/03/13 00:22:23 INFO mapreduce.Job: Task Id :
attempt_1426097251374_0011_m_000011_0, Status : FAILED
Container [pid=27919,containerID=container_1426097251374_0011_01_000013] is
running beyond physical memory limits. Current usage: 513.5 MB of 512 MB
physical memory used; 879.3 MB of 1.0 GB virtual memory used. Killing
container.
Dump of the process-tree for container_1426097251374_0011_01_000013 :
|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS)
SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
|- 28078 27919 27919 27919 (java) 63513 834 912551936 131129
/usr/jdk64/jdk1.7.0_45/bin/java -server -XX:NewRatio=8
-Djava.net.preferIPv4Stack=true -Dhdp.version=2.2.0.0-2041 -Xmx410m
-Djava.io.tmpdir=/mnt/hdfs/hadoop/yarn/local/usercache/hdfs/appcache/application_1426097251374_0011/container_1426097251374_0011_01_000013/tmp
-Dlog4j.configuration=container-log4j.properties
-Dyarn.app.container.log.dir=/mnt/hdfs/hadoop/yarn/log/application_1426097251374_0011/container_1426097251374_0011_01_000013
-Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA
org.apache.hadoop.mapred.YarnChild 172.27.2.57 52335
attempt_1426097251374_0011_m_000011_0 13
|- 27919 27917 27919 27919 (bash) 1 2 9424896 317 /bin/bash -c
/usr/jdk64/jdk1.7.0_45/bin/java -server -XX:NewRatio=8
-Djava.net.preferIPv4Stack=true -Dhdp.version=2.2.0.0-2041 -Xmx410m
-Djava.io.tmpdir=/mnt/hdfs/hadoop/yarn/local/usercache/hdfs/appcache/application_1426097251374_0011/container_1426097251374_0011_01_000013/tmp
-Dlog4j.configuration=container-log4j.properties
-Dyarn.app.container.log.dir=/mnt/hdfs/hadoop/yarn/log/application_1426097251374_0011/container_1426097251374_0011_01_000013
-Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA
org.apache.hadoop.mapred.YarnChild 172.27.2.57 52335
attempt_1426097251374_0011_m_000011_0 13
1>/mnt/hdfs/hadoop/yarn/log/application_1426097251374_0011/container_1426097251374_0011_01_000013/stdout
2>/mnt/hdfs/hadoop/yarn/log/application_1426097251374_0011/container_1426097251374_0011_01_000013/stderr
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
We have tried several different sizes for various switches, but the job
always fails.
Is this simply a function of the data, or is there another issue?
Thanks,
Steve
RE: sqoop import to hive being killed by resource manager
Posted by "Martin, Nick" <Ni...@pssd.com>.
Adding @sqoop user ☺
From: Steve Howard [mailto:stevedhoward@gmail.com]
Sent: Thursday, March 12, 2015 8:24 PM
To: user@hive.apache.org
Subject: sqoop import to hive being killed by resource manager
Hi All,
We have not been able to get what is in the subject line to run. This is on hive 0.14. While pulling a billion row table from Oracle using 12 splits on the primary key, each job continually runs out of memory such as below...
15/03/13 00:22:23 INFO mapreduce.Job: Task Id : attempt_1426097251374_0011_m_000011_0, Status : FAILED
Container [pid=27919,containerID=container_1426097251374_0011_01_000013] is running beyond physical memory limits. Current usage: 513.5 MB of 512 MB physical memory used; 879.3 MB of 1.0 GB virtual memory used. Killing container.
Dump of the process-tree for container_1426097251374_0011_01_000013 :
|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
|- 28078 27919 27919 27919 (java) 63513 834 912551936 131129 /usr/jdk64/jdk1.7.0_45/bin/java -server -XX:NewRatio=8 -Djava.net.preferIPv4Stack=true -Dhdp.version=2.2.0.0-2041 -Xmx410m -Djava.io.tmpdir=/mnt/hdfs/hadoop/yarn/local/usercache/hdfs/appcache/application_1426097251374_0011/container_1426097251374_0011_01_000013/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/mnt/hdfs/hadoop/yarn/log/application_1426097251374_0011/container_1426097251374_0011_01_000013 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 172.27.2.57 52335 attempt_1426097251374_0011_m_000011_0 13
|- 27919 27917 27919 27919 (bash) 1 2 9424896 317 /bin/bash -c /usr/jdk64/jdk1.7.0_45/bin/java -server -XX:NewRatio=8 -Djava.net.preferIPv4Stack=true -Dhdp.version=2.2.0.0-2041 -Xmx410m -Djava.io.tmpdir=/mnt/hdfs/hadoop/yarn/local/usercache/hdfs/appcache/application_1426097251374_0011/container_1426097251374_0011_01_000013/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/mnt/hdfs/hadoop/yarn/log/application_1426097251374_0011/container_1426097251374_0011_01_000013 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 172.27.2.57 52335 attempt_1426097251374_0011_m_000011_0 13 1>/mnt/hdfs/hadoop/yarn/log/application_1426097251374_0011/container_1426097251374_0011_01_000013/stdout 2>/mnt/hdfs/hadoop/yarn/log/application_1426097251374_0011/container_1426097251374_0011_01_000013/stderr
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
We have tried several different sizes for various switches, but the job always fails.
Is this simply a function of the data, or is there another issue?
Thanks,
Steve