You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by shaik ahamed <sh...@gmail.com> on 2012/08/02 08:00:29 UTC
Error while inserting 100GB data in to hive external table
Hi Users,
After running for a long time i.e 5hrs i got the below
error
Below are the cmds as i used for inseting the data into hive table
CREATE EXTERNAL TABLE vender_100GB(vender string,supplier string,order_date
string,quantity int) row format delimited fields terminated by ' ' stored
as textfile
LOCATION '/user/hive/warehouse/vender';
CREATE EXTERNAL TABLE vender_100gb_part(supplier string,order_date
string,quantity int) PARTITIONED BY(vender string) row format delimited
fields terminated by ' '
stored as textfile LOCATION '/user/hive/warehouse/vender';
load data local inpath
'/usr/local/hadoop_dir/hadoop/big_data/vender_details.txt' overwrite into
table vender_100gb;
FROM vender_100GB v INSERT OVERWRITE TABLE vender_100gb_part
PARTITION(vender) SELECT v.vender, v.supplier,v.order_date,v.quantity
DISTRIBUTE BY vender;
*ERROR*:
2012-08-02 00:17:42,265 Stage-1 map = 100%, reduce = 99%, Cumulative CPU
19901.5 sec
2012-08-02 00:17:43,273 Stage-1 map = 100%, reduce = 99%, Cumulative CPU
19901.5 sec
2012-08-02 00:17:44,284 Stage-1 map = 100%, reduce = 100%, Cumulative CPU
19901.5 sec
MapReduce Total cumulative CPU time: 0 days 5 hours 31 minutes 41 seconds
500 msec
Ended Job = job_201208011427_0002 with errors
Error during job, obtaining debugging information...
Examining task ID: task_201208011427_0002_m_000414 (and more) from job
job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000005 (and more) from job
job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000009 (and more) from job
job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000012 (and more) from job
job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000251 (and more) from job
job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000255 (and more) from job
job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000028 (and more) from job
job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000034 (and more) from job
job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000269 (and more) from job
job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000044 (and more) from job
job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000050 (and more) from job
job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000283 (and more) from job
job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000060 (and more) from job
job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000067 (and more) from job
job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000073 (and more) from job
job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000303 (and more) from job
job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000077 (and more) from job
job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000317 (and more) from job
job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000084 (and more) from job
job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000331 (and more) from job
job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000089 (and more) from job
job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000345 (and more) from job
job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000352 (and more) from job
job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000359 (and more) from job
job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000102 (and more) from job
job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000373 (and more) from job
job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000107 (and more) from job
job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000387 (and more) from job
job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000395 (and more) from job
job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000401 (and more) from job
job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000409 (and more) from job
job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000127 (and more) from job
job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000136 (and more) from job
job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000141 (and more) from job
job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000152 (and more) from job
job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000162 (and more) from job
job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000172 (and more) from job
job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000173 (and more) from job
job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000185 (and more) from job
job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000195 (and more) from job
job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000211 (and more) from job
job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000222 (and more) from job
job_201208011427_0002
Examining task ID: task_201208011427_0002_r_000000 (and more) from job
job_201208011427_0002
Examining task ID: task_201208011427_0002_r_000009 (and more) from job
job_201208011427_0002
Examining task ID: task_201208011427_0002_r_000019 (and more) from job
job_201208011427_0002
Examining task ID: task_201208011427_0002_r_000029 (and more) from job
job_201208011427_0002
Examining task ID: task_201208011427_0002_r_000038 (and more) from job
job_201208011427_0002
Examining task ID: task_201208011427_0002_r_000050 (and more) from job
job_201208011427_0002
Examining task ID: task_201208011427_0002_r_000059 (and more) from job
job_201208011427_0002
Examining task ID: task_201208011427_0002_r_000070 (and more) from job
job_201208011427_0002
Examining task ID: task_201208011427_0002_r_000080 (and more) from job
job_201208011427_0002
Examining task ID: task_201208011427_0002_r_000090 (and more) from job
job_201208011427_0002
Examining task ID: task_201208011427_0002_r_000101 (and more) from job
job_201208011427_0002
Examining task ID: task_201208011427_0002_r_000110 (and more) from job
job_201208011427_0002
Exception in thread "Thread-22" java.lang.RuntimeException: Error while
reading from task log url
at
org.apache.hadoop.hive.ql.exec.errors.TaskLogProcessor.getStackTraces(TaskLogProcessor.java:240)
at
org.apache.hadoop.hive.ql.exec.JobDebugger.showJobFailDebugInfo(JobDebugger.java:227)
at
org.apache.hadoop.hive.ql.exec.JobDebugger.run(JobDebugger.java:92)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: Server returned HTTP response code: 400 for
URL:
http://md-trngpoc3:50060/tasklog?taskid=attempt_201208011427_0002_r_000078_1&start=-8193
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1436)
at java.net.URL.openStream(URL.java:1010)
at
org.apache.hadoop.hive.ql.exec.errors.TaskLogProcessor.getStackTraces(TaskLogProcessor.java:192)
... 3 more
FAILED: Execution Error, return code 2 from
org.apache.hadoop.hive.ql.exec.MapRedTask
MapReduce Jobs Launched:
Job 0: Map: 413 Reduce: 111 Cumulative CPU: 19901.5 sec HDFS Read:
110754163131 HDFS Write: 291604532 FAIL
Total MapReduce CPU Time Spent: 0 days 5 hours 31 minutes 41 seconds 500
msec
Please reply me the route cause for failing this.
Thanks in advance
shaik
Re: Error while inserting 100GB data in to hive external table
Posted by Bejoy Ks <be...@yahoo.com>.
Hi Shaik
What does the failed MR task logs say? Can you share the failed Task error log here?
If you are using hadoop 2.0/0.23 then there is a similar issue reported
https://issues.apache.org/jira/browse/HIVE-2804
Regards,
Bejoy KS
________________________________
From: shaik ahamed <sh...@gmail.com>
To: user@hive.apache.org
Sent: Thursday, August 2, 2012 11:30 AM
Subject: Error while inserting 100GB data in to hive external table
Hi Users,
After running for a long time i.e 5hrs i got the below error
Below are the cmds as i used for inseting the data into hive table
CREATE EXTERNAL TABLE vender_100GB(vender string,supplier string,order_date string,quantity int) row format delimited fields terminated by ' ' stored as textfile
LOCATION '/user/hive/warehouse/vender';
CREATE EXTERNAL TABLE vender_100gb_part(supplier string,order_date string,quantity int) PARTITIONED BY(vender string) row format delimited fields terminated by ' '
stored as textfile LOCATION '/user/hive/warehouse/vender';
load data local inpath '/usr/local/hadoop_dir/hadoop/big_data/vender_details.txt' overwrite into table vender_100gb;
FROM vender_100GB v INSERT OVERWRITE TABLE vender_100gb_part PARTITION(vender) SELECT v.vender, v.supplier,v.order_date,v.quantity DISTRIBUTE BY vender;
ERROR:
2012-08-02 00:17:42,265 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 19901.5 sec
2012-08-02 00:17:43,273 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 19901.5 sec
2012-08-02 00:17:44,284 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 19901.5 sec
MapReduce Total cumulative CPU time: 0 days 5 hours 31 minutes 41 seconds 500 msec
Ended Job = job_201208011427_0002 with errors
Error during job, obtaining debugging information...
Examining task ID: task_201208011427_0002_m_000414 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000005 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000009 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000012 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000251 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000255 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000028 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000034 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000269 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000044 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000050 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000283 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000060 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000067 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000073 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000303 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000077 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000317 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000084 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000331 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000089 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000345 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000352 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000359 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000102 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000373 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000107 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000387 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000395 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000401 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000409 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000127 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000136 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000141 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000152 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000162 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000172 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000173 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000185 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000195 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000211 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000222 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_r_000000 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_r_000009 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_r_000019 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_r_000029 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_r_000038 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_r_000050 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_r_000059 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_r_000070 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_r_000080 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_r_000090 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_r_000101 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_r_000110 (and more) from job job_201208011427_0002
Exception in thread "Thread-22" java.lang.RuntimeException: Error while reading from task log url
at org.apache.hadoop.hive.ql.exec.errors.TaskLogProcessor.getStackTraces(TaskLogProcessor.java:240)
at org.apache.hadoop.hive.ql.exec.JobDebugger.showJobFailDebugInfo(JobDebugger.java:227)
at org.apache.hadoop.hive.ql.exec.JobDebugger.run(JobDebugger.java:92)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: Server returned HTTP response code: 400 for URL: http://md-trngpoc3:50060/tasklog?taskid=attempt_201208011427_0002_r_000078_1&start=-8193
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1436)
at java.net.URL.openStream(URL.java:1010)
at org.apache.hadoop.hive.ql.exec.errors.TaskLogProcessor.getStackTraces(TaskLogProcessor.java:192)
... 3 more
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
MapReduce Jobs Launched:
Job 0: Map: 413 Reduce: 111 Cumulative CPU: 19901.5 sec HDFS Read: 110754163131 HDFS Write: 291604532 FAIL
Total MapReduce CPU Time Spent: 0 days 5 hours 31 minutes 41 seconds 500 msec
Please reply me the route cause for failing this.
Thanks in advance
shaik